Orionnnn/hub-stats huggingface dataset:
ReVideo: Remake a Video with Motion and Content Control
Authors: name: Chong Mou, name: Mingdeng Cao, name: Xintao Wang, name: Zhaoyang Zhang, name: Ying Shan, name: Jian Zhang
Published: 2024-05-22 17:46:08

Despite significant advancements in video generation and editing using
diffusion models, achieving accurate and localized video editing remains a
substantial challenge. Additionally, most existing video editing methods
primarily focus on altering visual content, with limited research dedicated to
motion editing. In this paper, we present a novel attempt to Remake a Video
(ReVideo) which stands out from existing methods by allowing precise video
editing in specific areas through the specification of both content and motion.
Content editing is facilitated by modifying the first frame, while the
trajectory-based motion control offers an intuitive user interaction
experience. ReVideo addresses a new task involving the coupling and training
imbalance between content and motion control. To tackle this, we develop a
three-stage training strategy that progressively decouples these two aspects
from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion
module to integrate content and motion control across various sampling steps
and spatial locations. Extensive experiments demonstrate that our ReVideo has
promising performance on several accurate video editing applications, i.e., (1)
locally changing video content while keeping the motion constant, (2) keeping
content unchanged and customizing new motion trajectories, (3) modifying both
content and motion trajectories. Our method can also seamlessly extend these
applications to multi-area editing without specific training, demonstrating its
flexibility and robustness.
CamViG: Camera Aware Image-to-Video Generation with Multimodal
  Transformers
Authors: name: Andrew Marmon, name: Grant Schindler, name: José Lezama, name: Dan Kondratyuk, name: Bryan Seybold, name: Irfan Essa
Published: 2024-05-21 20:54:27

We extend multimodal transformers to include 3D camera motion as a
conditioning signal for the task of video generation. Generative video models
are becoming increasingly powerful, thus focusing research efforts on methods
of controlling the output of such models. We propose to add virtual 3D camera
controls to generative video methods by conditioning generated video on an
encoding of three-dimensional camera movement over the course of the generated
video. Results demonstrate that we are (1) able to successfully control the
camera during video generation, starting from a single frame and a camera
signal, and (2) we demonstrate the accuracy of the generated 3D camera paths
using traditional computer vision methods.
OmniGlue: Generalizable Feature Matching with Foundation Model Guidance
Authors: name: Hanwen Jiang, name: Arjun Karpur, name: Bingyi Cao, name: Qixing Huang, name: Andre Araujo
Published: 2024-05-21 17:59:22

The image matching field has been witnessing a continuous emergence of novel
learnable feature matching techniques, with ever-improving performance on
conventional benchmarks. However, our investigation shows that despite these
gains, their potential for real-world applications is restricted by their
limited generalization capabilities to novel image domains. In this paper, we
introduce OmniGlue, the first learnable image matcher that is designed with
generalization as a core principle. OmniGlue leverages broad knowledge from a
vision foundation model to guide the feature matching process, boosting
generalization to domains not seen at training time. Additionally, we propose a
novel keypoint position-guided attention mechanism which disentangles spatial
and appearance information, leading to enhanced matching descriptors. We
perform comprehensive experiments on a suite of 7 datasets with varied image
domains, including scene-level, object-centric and aerial images. OmniGlue's
novel components lead to relative gains on unseen domains of 20.9% with
respect to a directly comparable reference model, while also outperforming the
recent LightGlue method by 9.5% relatively.Code and model can be found at
https://hwjiang1510.github.io/OmniGlue
Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and
  Attribute Control
Authors: name: Yue Han, name: Junwei Zhu, name: Keke He, name: Xu Chen, name: Yanhao Ge, name: Wei Li, name: Xiangtai Li, name: Jiangning Zhang, name: Chengjie Wang, name: Yong Liu
Published: 2024-05-21 17:50:12

Current face reenactment and swapping methods mainly rely on GAN frameworks,
but recent focus has shifted to pre-trained diffusion models for their superior
generation capabilities. However, training these models is resource-intensive,
and the results have not yet achieved satisfactory performance levels. To
address this issue, we introduce Face-Adapter, an efficient and effective
adapter designed for high-precision and high-fidelity face editing for
pre-trained diffusion models. We observe that both face reenactment/swapping
tasks essentially involve combinations of target structure, ID and attribute.
We aim to sufficiently decouple the control of these factors to achieve both
tasks in one model. Specifically, our method contains: 1) A Spatial Condition
Generator that provides precise landmarks and background; 2) A Plug-and-play
Identity Encoder that transfers face embeddings to the text space by a
transformer decoder. 3) An Attribute Controller that integrates spatial
conditions and detailed attributes. Face-Adapter achieves comparable or even
superior performance in terms of motion control precision, ID retention
capability, and generation quality compared to fully fine-tuned face
reenactment/swapping models. Additionally, Face-Adapter seamlessly integrates
with various StableDiffusion models.
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
Authors: name: William Brandon, name: Mayank Mishra, name: Aniruddha Nrusimha, name: Rameswar Panda, name: Jonathan Ragan Kelly
Published: 2024-05-21 17:59:29

Key-value (KV) caching plays an essential role in accelerating decoding for
transformer-based autoregressive large language models (LLMs). However, the
amount of memory required to store the KV cache can become prohibitive at long
sequence lengths and large batch sizes. Since the invention of the transformer,
two of the most effective interventions discovered for reducing the size of the
KV cache have been Multi-Query Attention (MQA) and its generalization,
Grouped-Query Attention (GQA). MQA and GQA both modify the design of the
attention block so that multiple query heads can share a single key/value head,
reducing the number of distinct key/value heads by a large factor while only
minimally degrading accuracy. In this paper, we show that it is possible to
take Multi-Query Attention a step further by also sharing key and value heads
between adjacent layers, yielding a new attention design we call Cross-Layer
Attention (CLA). With CLA, we find that it is possible to reduce the size of
the KV cache by another 2x while maintaining nearly the same accuracy as
unmodified MQA. In experiments training 1B- and 3B-parameter models from
scratch, we demonstrate that CLA provides a Pareto improvement over the
memory/accuracy tradeoffs which are possible with traditional MQA, enabling
inference with longer sequence lengths and larger batch sizes than would
otherwise be possible
Your Transformer is Secretly Linear
Authors: name: Anton Razzhigaev, name: Matvey Mikhalchuk, name: Elizaveta Goncharova, name: Nikolai Gerasimenko, name: Ivan Oseledets, name: Denis Dimitrov, name: Andrey Kuznetsov
Published: 2024-05-19 22:44:00

This paper reveals a novel linear characteristic exclusive to transformer
decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We
analyze embedding transformations between sequential layers, uncovering a
near-perfect linear relationship (Procrustes similarity score of 0.99).
However, linearity decreases when the residual component is removed due to a
consistently low output norm of the transformer layer. Our experiments show
that removing or linearly approximating some of the most linear blocks of
transformers does not affect significantly the loss or model performance.
Moreover, in our pretraining experiments on smaller models we introduce a
cosine-similarity-based regularization, aimed at reducing layer linearity. This
regularization improves performance metrics on benchmarks like Tiny Stories and
SuperGLUE and as well successfully decreases the linearity of the models. This
study challenges the existing understanding of transformer architectures,
suggesting that their operation may be more linear than previously assumed.
Personalized Residuals for Concept-Driven Text-to-Image Generation
Authors: name: Cusuh Ham, name: Matthew Fisher, name: James Hays, name: Nicholas Kolkin, name: Yuchen Liu, name: Richard Zhang, name: Tobias Hinz
Published: 2024-05-21 17:59:01

We present personalized residuals and localized attention-guided sampling for
efficient concept-driven generation using text-to-image diffusion models. Our
method first represents concepts by freezing the weights of a pretrained
text-conditioned diffusion model and learning low-rank residuals for a small
subset of the model's layers. The residual-based approach then directly enables
application of our proposed sampling technique, which applies the learned
residuals only in areas where the concept is localized via cross-attention and
applies the original diffusion weights in all other regions. Localized sampling
therefore combines the learned identity of the concept with the existing
generative prior of the underlying diffusion model. We show that personalized
residuals effectively capture the identity of a concept in ~3 minutes on a
single GPU without the use of regularization images and with fewer parameters
than previous models, and localized sampling allows using the original model as
strong prior for large parts of the image.
Diffusion for World Modeling: Visual Details Matter in Atari
Authors: name: Eloi Alonso, name: Adam Jelley, name: Vincent Micheli, name: Anssi Kanervisto, name: Amos Storkey, name: Tim Pearce, name: François Fleuret
Published: 2024-05-20 22:51:05

World models constitute a promising approach for training reinforcement
learning agents in a safe and sample-efficient manner. Recent world models
predominantly operate on sequences of discrete latent variables to model
environment dynamics. However, this compression into a compact discrete
representation may ignore visual details that are important for reinforcement
learning. Concurrently, diffusion models have become a dominant approach for
image generation, challenging well-established methods modeling discrete
latents. Motivated by this paradigm shift, we introduce DIAMOND (DIffusion As a
Model Of eNvironment Dreams), a reinforcement learning agent trained in a
diffusion world model. We analyze the key design choices that are required to
make diffusion suitable for world modeling, and demonstrate how improved visual
details can lead to improved agent performance. DIAMOND achieves a mean human
normalized score of 1.46 on the competitive Atari 100k benchmark; a new best
for agents trained entirely within a world model. To foster future research on
diffusion for world modeling, we release our code, agents and playable world
models at https://github.com/eloialonso/diamond.
Octo: An Open-Source Generalist Robot Policy
Authors: name: Octo Model Team, name: Dibya Ghosh, name: Homer Walke, name: Karl Pertsch, name: Kevin Black, name: Oier Mees, name: Sudeep Dasari, name: Joey Hejna, name: Tobias Kreiman, name: Charles Xu, name: Jianlan Luo, name: You Liang Tan, name: Pannag Sanketi, name: Quan Vuong, name: Ted Xiao, name: Dorsa Sadigh, name: Chelsea Finn, name: Sergey Levine
Published: 2024-05-20 17:57:01

Large policies pretrained on diverse robot datasets have the potential to
transform robotic learning: instead of training new policies from scratch, such
generalist robot policies may be finetuned with only a little in-domain data,
yet generalize broadly. However, to be widely applicable across a range of
robotic learning scenarios, environments, and tasks, such policies need to
handle diverse sensors and action spaces, accommodate a variety of commonly
used robotic platforms, and finetune readily and efficiently to new domains. In
this work, we aim to lay the groundwork for developing open-source, widely
applicable, generalist policies for robotic manipulation. As a first step, we
introduce Octo, a large transformer-based policy trained on 800k trajectories
from the Open X-Embodiment dataset, the largest robot manipulation dataset to
date. It can be instructed via language commands or goal images and can be
effectively finetuned to robot setups with new sensory inputs and action spaces
within a few hours on standard consumer GPUs. In experiments across 9 robotic
platforms, we demonstrate that Octo serves as a versatile policy initialization
that can be effectively finetuned to new observation and action spaces. We also
perform detailed ablations of design decisions for the Octo model, from
architecture to training data, to guide future research on building generalist
robot models.
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Authors: name: Ting Jiang, name: Shaohan Huang, name: Shengyue Luo, name: Zihan Zhang, name: Haizhen Huang, name: Furu Wei, name: Weiwei Deng, name: Feng Sun, name: Qi Zhang, name: Deqing Wang, name: Fuzhen Zhuang
Published: 2024-05-20 15:48:32

Low-rank adaptation is a popular parameter-efficient fine-tuning method for
large language models. In this paper, we analyze the impact of low-rank
updating, as implemented in LoRA. Our findings suggest that the low-rank
updating mechanism may limit the ability of LLMs to effectively learn and
memorize new knowledge. Inspired by this observation, we propose a new method
called MoRA, which employs a square matrix to achieve high-rank updating while
maintaining the same number of trainable parameters. To achieve it, we
introduce the corresponding non-parameter operators to reduce the input
dimension and increase the output dimension for the square matrix. Furthermore,
these operators ensure that the weight can be merged back into LLMs, which
makes our method can be deployed like LoRA. We perform a comprehensive
evaluation of our method across five tasks: instruction tuning, mathematical
reasoning, continual pretraining, memory and pretraining. Our method
outperforms LoRA on memory-intensive tasks and achieves comparable performance
on other tasks.
Imp: Highly Capable Large Multimodal Models for Mobile Devices
Authors: name: Zhenwei Shao, name: Zhou Yu, name: Jun Yu, name: Xuecheng Ouyang, name: Lihao Zheng, name: Zhenbiao Gai, name: Mingyang Wang, name: Jiajun Ding
Published: 2024-05-20 15:23:19

By harnessing the capabilities of large language models (LLMs), recent large
multimodal models (LMMs) have shown remarkable versatility in open-world
multimodal understanding. Nevertheless, they are usually parameter-heavy and
computation-intensive, thus hindering their applicability in
resource-constrained scenarios. To this end, several lightweight LMMs have been
proposed successively to maximize the capabilities under constrained scale
(e.g., 3B). Despite the encouraging results achieved by these methods, most of
them only focus on one or two aspects of the design space, and the key design
choices that influence model capability have not yet been thoroughly
investigated. In this paper, we conduct a systematic study for lightweight LMMs
from the aspects of model architecture, training strategy, and training data.
Based on our findings, we obtain Imp -- a family of highly capable LMMs at the
2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing
lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs
at the 13B scale. With low-bit quantization and resolution reduction
techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile
chip with a high inference speed of about 13 tokens/s.
SLAB: Efficient Transformers with Simplified Linear Attention and
  Progressive Re-parameterized Batch Normalization
Authors: name: Jialong Guo, name: Xinghao Chen, name: Yehui Tang, name: Yunhe Wang
Published: 2024-05-19 15:22:25

Transformers have become foundational architectures for both natural language
and computer vision tasks. However, the high computational cost makes it quite
challenging to deploy on resource-constraint devices. This paper investigates
the computational bottleneck modules of efficient transformer, i.e.,
normalization layers and attention modules. LayerNorm is commonly used in
transformer architectures but is not computational friendly due to statistic
calculation during inference. However, replacing LayerNorm with more efficient
BatchNorm in transformer often leads to inferior performance and collapse in
training. To address this problem, we propose a novel method named PRepBN to
progressively replace LayerNorm with re-parameterized BatchNorm in training.
Moreover, we propose a simplified linear attention (SLA) module that is simple
yet effective to achieve strong performance. Extensive experiments on image
classification as well as object detection demonstrate the effectiveness of our
proposed method. For example, our SLAB-Swin obtains 83.6% top-1 accuracy on
ImageNet-1K with 16.2ms latency, which is 2.4ms less than that of
Flatten-Swin with 0.1% higher accuracy. We also evaluated our method for
language modeling task and obtain comparable performance and lower
latency.Codes are publicly available at https://github.com/xinghaochen/SLAB and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SLAB.
Towards Modular LLMs by Building and Reusing a Library of LoRAs
Authors: name: Oleksiy Ostapenko, name: Zhan Su, name: Edoardo Maria Ponti, name: Laurent Charlin, name: Nicolas Le Roux, name: Matheus Pereira, name: Lucas Caccia, name: Alessandro Sordoni
Published: 2024-05-18 03:02:23

The growing number of parameter-efficient adaptations of a base large
language model (LLM) calls for studying whether we can reuse such trained
adapters to improve performance for new tasks. We study how to best build a
library of adapters given multi-task data and devise techniques for both
zero-shot and supervised task generalization through routing in such library.
We benchmark existing approaches to build this library and introduce
model-based clustering, MBC, a method that groups tasks based on the similarity
of their adapter parameters, indirectly optimizing for transfer across the
multi-task dataset. To re-use the library, we present a novel zero-shot routing
mechanism, Arrow, which enables dynamic selection of the most relevant adapters
for new inputs without the need for retraining. We experiment with several
LLMs, such as Phi-2 and Mistral, on a wide array of held-out tasks, verifying
that MBC-based adapters and Arrow routing lead to superior generalization to
new tasks. We make steps towards creating modular, adaptable LLMs that can
match or outperform traditional joint training.
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
Authors: name: Jian Hu, name: Xibin Wu, name: Weixun Wang, name: Xianyu, name: Dehao Zhang, name: Yu Cao
Published: 2024-05-20 01:04:40

Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human
Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
significantly improve the alignment of human-AI values and further raise the
upper bound of AI capabilities, particularly in reasoning-intensive,
long-context Chain-of-Thought (long-CoT) tasks. However, existing RLHF (or
RLVR) frameworks commonly face challenges such as inference bottlenecks and
complexity barriers, restricting their accessibility for newcomers. To bridge
this gap, we introduce OpenRLHF, a user-friendly, scalable, and easy-to-learn
open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace
Transformers, featuring a simplified design, clear code structure, and
comprehensive documentation to facilitate entry for researchers and
practitioners. Experimental results show that OpenRLHF achieves superior
training efficiency with speedups ranging from 1.22x to 1.68x across different
model sizes compared to state-of-the-art frameworks, while requiring
significantly fewer lines of code for implementation. OpenRLHF is publicly
available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted
by leading institutions to accelerate RLHF research and learning.
Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory
  Score Matching
Authors: name: Xingyu Miao, name: Haoran Duan, name: Varun Ojha, name: Jun Song, name: Tejal Shah, name: Yang Long, name: Rajiv Ranjan
Published: 2024-05-18 10:41:57

In this work, we propose a novel Trajectory Score Matching (TSM) method that
aims to solve the pseudo ground truth inconsistency problem caused by the
accumulated error in Interval Score Matching (ISM) when using the Denoising
Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the
inversion process of DDIM to calculate on a single path, our TSM method
leverages the inversion process of DDIM to generate two paths from the same
starting point for calculation. Since both paths start from the same starting
point, TSM can reduce the accumulated error compared to ISM, thus alleviating
the problem of pseudo ground truth inconsistency. TSM enhances the stability
and consistency of the model's generated paths during the distillation process.
We demonstrate this experimentally and further show that ISM is a special case
of TSM. Furthermore, to optimize the current multi-stage optimization process
from high-resolution text to 3D generation, we adopt Stable Diffusion XL for
guidance. In response to the issues of abnormal replication and splitting
caused by unstable gradients during the 3D Gaussian splatting process when
using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping
method. Extensive experiments show that our model significantly surpasses the
state-of-the-art models in terms of visual quality and performance. Code:
https://github.com/xingy038/Dreamer-XL.
FIFO-Diffusion: Generating Infinite Videos from Text without Training
Authors: name: Jihwan Kim, name: Junoh Kang, name: Jinyoung Choi, name: Bohyung Han
Published: 2024-05-19 07:48:41

We propose a novel inference technique based on a pretrained diffusion model
for text-conditional video generation. Our approach, called FIFO-Diffusion, is
conceptually capable of generating infinitely long videos without training.
This is achieved by iteratively performing diagonal denoising, which
concurrently processes a series of consecutive frames with increasing noise
levels in a queue; our method dequeues a fully denoised frame at the head while
enqueuing a new random noise frame at the tail. However, diagonal denoising is
a double-edged sword as the frames near the tail can take advantage of cleaner
ones by forward reference but such a strategy induces the discrepancy between
training and inference. Hence, we introduce latent partitioning to reduce the
training-inference gap and lookahead denoising to leverage the benefit of
forward referencing. We have demonstrated the promising results and
effectiveness of the proposed methods on existing text-to-video generation
baselines.