Orionnnn/hub-stats huggingface dataset:
MoViE: Mobile Diffusion for Video Editing
Authors: name: Adil Karjauv, name: Noor Fathima, name: Ioannis Lelekas, name: Fatih Porikli, name: Amir Ghodrati, name: Amirhossein Habibian
Published: 2024-12-09 15:30:09

Recent progress in diffusion-based video editing has shown remarkable
potential for practical applications. However, these methods remain
prohibitively expensive and challenging to deploy on mobile devices. In this
study, we introduce a series of optimizations that render mobile video editing
feasible. Building upon the existing image editing model, we first optimize its
architecture and incorporate a lightweight autoencoder. Subsequently, we extend
classifier-free guidance distillation to multiple modalities, resulting in a
threefold on-device speedup. Finally, we reduce the number of sampling steps to
one by introducing a novel adversarial distillation scheme which preserves the
controllability of the editing process. Collectively, these optimizations
enable video editing at 12 frames per second on mobile devices, while
maintaining high quality. Our results are available at
https://qualcomm-ai-research.github.io/mobile-video-editing/
LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style
  Conditioned Image Generation
Authors: name: Donald Shenaj, name: Ondrej Bohdal, name: Mete Ozay, name: Pietro Zanuttigh, name: Umberto Michieli
Published: 2024-12-06 16:04:56

Recent advancements in image generation models have enabled personalized
image creation with both user-defined subjects (content) and styles. Prior
works achieved personalization by merging corresponding low-rank adaptation
parameters (LoRAs) through optimization-based methods, which are
computationally demanding and unsuitable for real-time use on
resource-constrained devices like smartphones. To address this, we introduce
LoRA.rar, a method that not only improves image quality but also achieves a
remarkable speedup of over 4000times in the merging process. LoRA.rar
pre-trains a hypernetwork on a diverse set of content-style LoRA pairs,
learning an efficient merging strategy that generalizes to new, unseen
content-style pairs, enabling fast, high-quality personalization. Moreover, we
identify limitations in existing evaluation metrics for content-style quality
and propose a new protocol using multimodal large language models (MLLM) for
more accurate assessment. Our method significantly outperforms the current
state of the art in both content and style fidelity, as validated by MLLM
assessments and human evaluations.
Contextualized Counterspeech: Strategies for Adaptation,
  Personalization, and Evaluation
Authors: name: Lorenzo Cima, name: Alessio Miaschi, name: Amaury Trujillo, name: Marco Avvenuti, name: Felice Dell'Orletta, name: Stefano Cresci
Published: 2024-12-10 09:29:52

AI-generated counterspeech offers a promising and scalable strategy to curb
online toxicity through direct replies that promote civil discourse. However,
current counterspeech is one-size-fits-all, lacking adaptation to the
moderation context and the users involved. We propose and evaluate multiple
strategies for generating tailored counterspeech that is adapted to the
moderation context and personalized for the moderated user. We instruct an
LLaMA2-13B model to generate counterspeech, experimenting with various
configurations based on different contextual information and fine-tuning
strategies. We identify the configurations that generate persuasive
counterspeech through a combination of quantitative indicators and human
evaluations collected via a pre-registered mixed-design crowdsourcing
experiment. Results show that contextualized counterspeech can significantly
outperform state-of-the-art generic counterspeech in adequacy and
persuasiveness, without compromising other characteristics. Our findings also
reveal a poor correlation between quantitative indicators and human
evaluations, suggesting that these methods assess different aspects and
highlighting the need for nuanced evaluation methodologies. The effectiveness
of contextualized AI-generated counterspeech and the divergence between human
and algorithmic evaluations underscore the importance of increased human-AI
collaboration in content moderation.
OmniDocBench: Benchmarking Diverse PDF Document Parsing with
  Comprehensive Annotations
Authors: name: Linke Ouyang, name: Yuan Qu, name: Hongbin Zhou, name: Jiawei Zhu, name: Rui Zhang, name: Qunshu Lin, name: Bin Wang, name: Zhiyuan Zhao, name: Man Jiang, name: Xiaomeng Zhao, name: Jin Shi, name: Fan Wu, name: Pei Chu, name: Minghao Liu, name: Zhenxiang Li, name: Chao Xu, name: Bo Zhang, name: Botian Shi, name: Zhongying Tu, name: Conghui He
Published: 2024-12-10 16:05:56

Document content extraction is crucial in computer vision, especially for
meeting the high-quality data needs of large language models (LLMs) and
retrieval-augmented generation (RAG) technologies. However, current document
parsing methods suffer from significant limitations in terms of diversity and
comprehensive evaluation. To address these challenges, we introduce
OmniDocBench, a novel multi-source benchmark designed to advance automated
document content extraction. OmniDocBench includes a meticulously curated and
annotated high-quality evaluation dataset comprising nine diverse document
types, such as academic papers, textbooks, slides, among others. Our benchmark
provides a flexible and comprehensive evaluation framework with 19 layout
category labels and 14 attribute labels, enabling multi-level assessments
across entire datasets, individual modules, or specific data types. Using
OmniDocBench, we perform an exhaustive comparative analysis of existing modular
pipelines and multimodal end-to-end methods, highlighting their limitations in
handling document diversity and ensuring fair evaluation. OmniDocBench
establishes a robust, diverse, and fair evaluation standard for the document
content extraction field, offering crucial insights for future advancements and
fostering the development of document parsing technologies. The codes and
dataset is available in https://github.com/opendatalab/OmniDocBench.
3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video
  Generation
Authors: name: Xiao Fu, name: Xian Liu, name: Xintao Wang, name: Sida Peng, name: Menghan Xia, name: Xiaoyu Shi, name: Ziyang Yuan, name: Pengfei Wan, name: Di Zhang, name: Dahua Lin
Published: 2024-12-10 18:55:13

This paper aims to manipulate multi-entity 3D motions in video generation.
Previous methods on controllable video generation primarily leverage 2D control
signals to manipulate object motions and have achieved remarkable synthesis
results. However, 2D control signals are inherently limited in expressing the
3D nature of object motions. To overcome this problem, we introduce
3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D
space, given user-desired 6DoF pose (location and rotation) sequences of
entities. At the core of our approach is a plug-and-play 3D-motion grounded
object injector that fuses multiple input entities with their respective 3D
trajectories through a gated self-attention mechanism. In addition, we exploit
an injector architecture to preserve the video diffusion prior, which is
crucial for generalization ability. To mitigate video quality degradation, we
introduce a domain adaptor during training and employ an annealed sampling
strategy during inference. To address the lack of suitable training data, we
construct a 360-Motion Dataset, which first correlates collected 3D human and
animal assets with GPT-generated trajectory and then captures their motion with
12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments
show that 3DTrajMaster sets a new state-of-the-art in both accuracy and
generalization for controlling multi-entity 3D motions. Project page:
http://fuxiao0719.github.io/projects/3dtrajmaster
Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards
  for Visuomotor Robot Policy Alignment
Authors: name: Ran Tian, name: Yilin Wu, name: Chenfeng Xu, name: Masayoshi Tomizuka, name: Jitendra Malik, name: Andrea Bajcsy
Published: 2024-12-06 08:04:02

Visuomotor robot policies, increasingly pre-trained on large-scale datasets,
promise significant advancements across robotics domains. However, aligning
these policies with end-user preferences remains a challenge, particularly when
the preferences are hard to specify. While reinforcement learning from human
feedback (RLHF) has become the predominant mechanism for alignment in
non-embodied domains like large language models, it has not seen the same
success in aligning visuomotor policies due to the prohibitive amount of human
feedback required to learn visual reward functions. To address this limitation,
we propose Representation-Aligned Preference-based Learning (RAPL), an
observation-only method for learning visual rewards from significantly less
human preference feedback. Unlike traditional RLHF, RAPL focuses human feedback
on fine-tuning pre-trained vision encoders to align with the end-user's visual
representation and then constructs a dense visual reward via feature matching
in this aligned representation space. We first validate RAPL through simulation
experiments in the X-Magical benchmark and Franka Panda robotic manipulation,
demonstrating that it can learn rewards aligned with human preferences, more
efficiently uses preference data, and generalizes across robot embodiments.
Finally, our hardware experiments align pre-trained Diffusion Policies for
three object manipulation tasks. We find that RAPL can fine-tune these policies
with 5x less real human preference data, taking the first step towards
minimizing human feedback while maximizing visuomotor robot policy alignment.
Chimera: Improving Generalist Model with Domain-Specific Experts
Authors: name: Tianshuo Peng, name: Mingsheng Li, name: Hongbin Zhou, name: Renqiu Xia, name: Renrui Zhang, name: Lei Bai, name: Song Mao, name: Bin Wang, name: Conghui He, name: Aojun Zhou, name: Botian Shi, name: Tao Chen, name: Bo Zhang, name: Xiangyu Yue
Published: 2024-12-08 16:10:42

Recent advancements in Large Multi-modal Models (LMMs) underscore the
importance of scaling by increasing image-text paired data, achieving
impressive performance on general tasks. Despite their effectiveness in broad
applications, generalist models are primarily trained on web-scale datasets
dominated by natural images, resulting in the sacrifice of specialized
capabilities for domain-specific tasks that require extensive domain prior
knowledge. Moreover, directly integrating expert models tailored for specific
domains is challenging due to the representational gap and imbalanced
optimization between the generalist model and experts. To address these
challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline
designed to boost the ability of existing LMMs with domain-specific experts.
Specifically, we design a progressive training strategy to integrate features
from expert models into the input of a generalist LMM. To address the
imbalanced optimization caused by the well-aligned general visual encoder, we
introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism.
This results in a versatile model that excels across the chart, table, math,
and document domains, achieving state-of-the-art performance on multi-modal
reasoning and visual content extraction tasks, both of which are challenging
tasks for assessing existing LMMs.
Frame Representation Hypothesis: Multi-Token LLM Interpretability and
  Concept-Guided Text Generation
Authors: name: Pedro H. V. Valois, name: Lincon S. Souza, name: Erica K. Shimomoto, name: Kazuhiro Fukui
Published: 2024-12-10 09:25:39

Interpretability is a key challenge in fostering trust for Large Language
Models (LLMs), which stems from the complexity of extracting reasoning from
model's parameters. We present the Frame Representation Hypothesis, a
theoretically robust framework grounded in the Linear Representation Hypothesis
(LRH) to interpret and control LLMs by modeling multi-token words. Prior
research explored LRH to connect LLM representations with linguistic concepts,
but was limited to single token analysis. As most words are composed of several
tokens, we extend LRH to multi-token words, thereby enabling usage on any
textual data with thousands of concepts. To this end, we propose words can be
interpreted as frames, ordered sequences of vectors that better capture
token-word relationships. Then, concepts can be represented as the average of
word frames sharing a common concept. We showcase these tools through Top-k
Concept-Guided Decoding, which can intuitively steer text generation using
concepts of choice. We verify said ideas on Llama 3.1, Gemma 2, and Phi 3
families, demonstrating gender and language biases, exposing harmful content,
but also potential to remediate them, leading to safer and more transparent
LLMs. Code is available at
https://github.com/phvv-me/frame-representation-hypothesis.git
Video Motion Transfer with Diffusion Transformers
Authors: name: Alexander Pondaven, name: Aliaksandr Siarohin, name: Sergey Tulyakov, name: Philip Torr, name: Fabio Pizzati
Published: 2024-12-10 18:59:58

We propose DiTFlow, a method for transferring the motion of a reference video
to a newly synthesized one, designed specifically for Diffusion Transformers
(DiT). We first process the reference video with a pre-trained DiT to analyze
cross-frame attention maps and extract a patch-wise motion signal called the
Attention Motion Flow (AMF). We guide the latent denoising process in an
optimization-based, training-free, manner by optimizing latents with our AMF
loss to generate videos reproducing the motion of the reference one. We also
apply our optimization strategy to transformer positional embeddings, granting
us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
against recently published methods, outperforming all across multiple metrics
and human evaluation.
FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion
  Models
Authors: name: Tong Wu, name: Yinghao Xu, name: Ryan Po, name: Mengchen Zhang, name: Guandao Yang, name: Jiaqi Wang, name: Ziwei Liu, name: Dahua Lin, name: Gordon Wetzstein
Published: 2024-12-10 17:02:58

Recent advances in text-to-image generation have enabled the creation of
high-quality images with diverse applications. However, accurately describing
desired visual attributes can be challenging, especially for non-experts in art
and photography. An intuitive solution involves adopting favorable attributes
from the source images. Current methods attempt to distill identity and style
from source images. However, "style" is a broad concept that includes texture,
color, and artistic elements, but does not cover other important attributes
such as lighting and dynamics. Additionally, a simplified "style" adaptation
prevents combining multiple attributes from different sources into one
generated image. In this work, we formulate a more effective approach to
decompose the aesthetics of a picture into specific visual attributes, allowing
users to apply characteristics such as lighting, texture, and dynamics from
different images. To achieve this goal, we constructed the first fine-grained
visual attributes dataset (FiVA) to the best of our knowledge. This FiVA
dataset features a well-organized taxonomy for visual attributes and includes
around 1 M high-quality generated images with visual attribute annotations.
Leveraging this dataset, we propose a fine-grained visual attribute adaptation
framework (FiVA-Adapter), which decouples and adapts visual attributes from one
or more source images into a generated one. This approach enhances
user-friendly customization, allowing users to selectively apply desired
attributes to create images that meet their unique preferences and specific
content requirements.
A New Federated Learning Framework Against Gradient Inversion Attacks
Authors: name: Pengxin Guo, name: Shuang Zeng, name: Wenhao Chen, name: Xiaodan Zhang, name: Weihong Ren, name: Yuyin Zhou, name: Liangqiong Qu
Published: 2024-12-10 04:53:42

Federated Learning (FL) aims to protect data privacy by enabling clients to
collectively train machine learning models without sharing their raw data.
However, recent studies demonstrate that information exchanged during FL is
subject to Gradient Inversion Attacks (GIA) and, consequently, a variety of
privacy-preserving methods have been integrated into FL to thwart such attacks,
such as Secure Multi-party Computing (SMC), Homomorphic Encryption (HE), and
Differential Privacy (DP). Despite their ability to protect data privacy, these
approaches inherently involve substantial privacy-utility trade-offs. By
revisiting the key to privacy exposure in FL under GIA, which lies in the
frequent sharing of model gradients that contain private data, we take a new
perspective by designing a novel privacy preserve FL framework that effectively
``breaks the direct connection'' between the shared parameters and the local
private data to defend against GIA. Specifically, we propose a Hypernetwork
Federated Learning (HyperFL) framework that utilizes hypernetworks to generate
the parameters of the local model and only the hypernetwork parameters are
uploaded to the server for aggregation. Theoretical analyses demonstrate the
convergence rate of the proposed HyperFL, while extensive experimental results
show the privacy-preserving capability and comparable performance of HyperFL.
Code is available at https://github.com/Pengxin-Guo/HyperFL.
STIV: Scalable Text and Image Conditioned Video Generation
Authors: name: Zongyu Lin, name: Wei Liu, name: Chen Chen, name: Jiasen Lu, name: Wenze Hu, name: Tsu-Jui Fu, name: Jesse Allardice, name: Zhengfeng Lai, name: Liangchen Song, name: Bowen Zhang, name: Cha Chen, name: Yiran Fei, name: Yifan Jiang, name: Lezhi Li, name: Yizhou Sun, name: Kai-Wei Chang, name: Yinfei Yang
Published: 2024-12-10 18:27:06

The field of video generation has made remarkable advancements, yet there
remains a pressing need for a clear, systematic recipe that can guide the
development of robust and scalable models. In this work, we present a
comprehensive study that systematically explores the interplay of model
architectures, training recipes, and data curation strategies, culminating in a
simple and scalable text-image-conditioned video generation method, named STIV.
Our framework integrates image condition into a Diffusion Transformer (DiT)
through frame replacement, while incorporating text conditioning via a joint
image-text conditional classifier-free guidance. This design enables STIV to
perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks
simultaneously. Additionally, STIV can be easily extended to various
applications, such as video prediction, frame interpolation, multi-view
generation, and long video generation, etc. With comprehensive ablation studies
on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple
design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V,
surpassing both leading open and closed-source models like CogVideoX-5B, Pika,
Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result
of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and
extensible recipe for building cutting-edge video generation models, we aim to
empower future research and accelerate progress toward more versatile and
reliable video generation solutions.
DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for
  Customized Manga Generation
Authors: name: Jianzong Wu, name: Chao Tang, name: Jingbo Wang, name: Yanhong Zeng, name: Xiangtai Li, name: Yunhai Tong
Published: 2024-12-10 15:24:12

Story visualization, the task of creating visual narratives from textual
descriptions, has seen progress with text-to-image generation models. However,
these models often lack effective control over character appearances and
interactions, particularly in multi-character scenes. To address these
limitations, we propose a new task: customized manga generation and
introduce DiffSensei, an innovative framework specifically designed
for generating manga with dynamic multi-character control. DiffSensei
integrates a diffusion-based image generator with a multimodal large language
model (MLLM) that acts as a text-compatible identity adapter. Our approach
employs masked cross-attention to seamlessly incorporate character features,
enabling precise layout control without direct pixel transfer. Additionally,
the MLLM-based adapter adjusts character features to align with panel-specific
text cues, allowing flexible adjustments in character expressions, poses, and
actions. We also introduce MangaZero, a large-scale dataset tailored
to this task, containing 43,264 manga pages and 427,147 annotated panels,
supporting the visualization of varied character interactions and movements
across sequential frames. Extensive experiments demonstrate that DiffSensei
outperforms existing models, marking a significant advancement in manga
generation by enabling text-adaptable character customization. The project page
is https://jianzongwu.github.io/projects/diffsensei/.
EMOv2: Pushing 5M Vision Model Frontier
Authors: name: Jiangning Zhang, name: Teng Hu, name: Haoyang He, name: Zhucun Xue, name: Yabiao Wang, name: Chengjie Wang, name: Yong Liu, name: Xiangtai Li, name: Dacheng Tao
Published: 2024-12-09 17:12:22

This work focuses on developing parameter-efficient and lightweight models
for dense predictions while trading off parameters, FLOPs, and performance. Our
goal is to set up the new frontier of the 5M magnitude lightweight model on
various downstream tasks. Inverted Residual Block (IRB) serves as the
infrastructure for lightweight CNNs, but no counterparts have been recognized
by attention-based design. Our work rethinks the lightweight infrastructure of
efficient IRB and practical components in Transformer from a unified
perspective, extending CNN-based IRB to attention-based models and abstracting
a one-residual Meta Mobile Block (MMBlock) for lightweight model design.
Following neat but effective design criterion, we deduce a modern Improved
Inverted Residual Mobile Block (i2RMB) and improve a hierarchical Efficient
MOdel (EMOv2) with no elaborate complex structures. Considering the
imperceptible latency for mobile users when downloading models under 4G/5G
bandwidth and ensuring model performance, we investigate the performance upper
limit of lightweight models with a magnitude of 5M. Extensive experiments on
various vision recognition, dense prediction, and image generation tasks
demonstrate the superiority of our EMOv2 over state-of-the-art methods, e.g.,
EMOv2-1M/2M/5M achieve 72.3, 75.8, and 79.4 Top-1 that surpass equal-order
CNN-/Attention-based models significantly. At the same time, EMOv2-5M equipped
RetinaNet achieves 41.5 mAP for object detection tasks that surpasses the
previous EMO-5M by +2.6. When employing the more robust training recipe, our
EMOv2-5M eventually achieves 82.9 Top-1 accuracy, which elevates the
performance of 5M magnitude models to a new level. Code is available at
https://github.com/zhangzjn/EMOv2.