Orionnnn/hub-stats huggingface dataset:
XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
Authors: name: João Monteiro, name: Étienne Marcotte, name: Pierre-André Noël, name: Valentina Zantedeschi, name: David Vázquez, name: Nicolas Chapados, name: Christopher Pal, name: Perouz Taslakian
Published: 2024-04-23 18:10:42

In-context learning (ICL) approaches typically leverage prompting to
condition decoder-only language model generation on reference information.
Just-in-time processing of a context is inefficient due to the quadratic cost
of self-attention operations, and caching is desirable. However, caching
transformer states can easily require almost as much space as the model
parameters. When the right context isn't known in advance, caching ICL can be
challenging. This work addresses these limitations by introducing models that,
inspired by the encoder-decoder architecture, use cross-attention to condition
generation on reference text without the prompt. More precisely, we leverage
pre-trained decoder-only models and only train a small number of added layers.
We use Question-Answering (QA) as a testbed to evaluate the ability of our
models to perform conditional generation and observe that they outperform ICL,
are comparable to fine-tuned prompted LLMs, and drastically reduce the space
footprint relative to standard KV caching by two orders of magnitude.
Editable Image Elements for Controllable Synthesis
Authors: name: Jiteng Mu, name: Michaël Gharbi, name: Richard Zhang, name: Eli Shechtman, name: Nuno Vasconcelos, name: Xiaolong Wang, name: Taesung Park
Published: 2024-04-24 17:59:11

Diffusion models have made significant advances in text-guided synthesis
tasks. However, editing user-provided images remains challenging, as the high
dimensional noise input space of diffusion models is not naturally suited for
image inversion or spatial editing. In this work, we propose an image
representation that promotes spatial editing of input images using a diffusion
model. Concretely, we learn to encode an input into "image elements" that can
faithfully reconstruct an input image. These elements can be intuitively edited
by a user, and are decoded by a diffusion model into realistic images. We show
the effectiveness of our representation on various image editing tasks, such as
object resizing, rearrangement, dragging, de-occlusion, removal, variation, and
image composition. Project page:
https://jitengmu.github.io/Editable_Image_Elements/
BASS: Batched Attention-optimized Speculative Sampling
Authors: name: Haifeng Qian, name: Sujan Kumar Gonugondla, name: Sungsoo Ha, name: Mingyue Shang, name: Sanjay Krishna Gouda, name: Ramesh Nallapati, name: Sudipta Sengupta, name: Xiaofei Ma, name: Anoop Deoras
Published: 2024-04-24 09:57:11

Speculative decoding has emerged as a powerful method to improve latency and
throughput in hosting large language models. However, most existing
implementations focus on generating a single sequence. Real-world generative AI
applications often require multiple responses and how to perform speculative
decoding in a batched setting while preserving its latency benefits poses
non-trivial challenges. This paper describes a system of batched speculative
decoding that sets a new state of the art in multi-sequence generation latency
and that demonstrates superior GPU utilization as well as quality of
generations within a time budget. For example, for a 7.8B-size model on a
single A100 GPU and with a batch size of 8, each sequence is generated at an
average speed of 5.8ms per token, the overall throughput being 1.1K tokens per
second. These results represent state-of-the-art latency and a 2.15X speed-up
over optimized regular decoding. Within a time budget that regular decoding
does not finish, our system is able to generate sequences with HumanEval
Pass@First of 43% and Pass@All of 61%, far exceeding what's feasible with
single-sequence speculative decoding. Our peak GPU utilization during decoding
reaches as high as 15.8%, more than 3X the highest of that of regular decoding
and around 10X of single-sequence speculative decoding.
MotionMaster: Training-free Camera Motion Transfer For Video Generation
Authors: name: Teng Hu, name: Jiangning Zhang, name: Ran Yi, name: Yating Wang, name: Hongrui Huang, name: Jieyu Weng, name: Yabiao Wang, name: Lizhuang Ma
Published: 2024-04-24 10:28:54

The emergence of diffusion models has greatly propelled the progress in image
and video generation. Recently, some efforts have been made in controllable
video generation, including text-to-video generation and video motion control,
among which camera motion control is an important topic. However, existing
camera motion control methods rely on training a temporal camera module, and
necessitate substantial computation resources due to the large amount of
parameters in video generation models. Moreover, existing methods pre-define
camera motion types during training, which limits their flexibility in camera
control. Therefore, to reduce training costs and achieve flexible camera
control, we propose COMD, a novel training-free video motion transfer model,
which disentangles camera motions and object motions in source videos and
transfers the extracted camera motions to new videos. We first propose a
one-shot camera motion disentanglement method to extract camera motion from a
single source video, which separates the moving objects from the background and
estimates the camera motion in the moving objects region based on the motion in
the background by solving a Poisson equation. Furthermore, we propose a
few-shot camera motion disentanglement method to extract the common camera
motion from multiple videos with similar camera motions, which employs a
window-based clustering technique to extract the common features in temporal
attention maps of multiple videos. Finally, we propose a motion combination
method to combine different types of camera motions together, enabling our
model a more controllable and flexible camera control. Extensive experiments
demonstrate that our training-free approach can effectively decouple
camera-object motion and apply the decoupled camera motion to a wide range of
controllable video generation tasks, achieving flexible and diverse camera
motion control.
PuLID: Pure and Lightning ID Customization via Contrastive Alignment
Authors: name: Zinan Guo, name: Yanze Wu, name: Zhuowei Chen, name: Lang Chen, name: Qian He
Published: 2024-04-24 17:55:33

We propose Pure and Lightning ID customization (PuLID), a novel tuning-free
ID customization method for text-to-image generation. By incorporating a
Lightning T2I branch with a standard diffusion one, PuLID introduces both
contrastive alignment loss and accurate ID loss, minimizing disruption to the
original model and ensuring high ID fidelity. Experiments show that PuLID
achieves superior performance in both ID fidelity and editability. Another
attractive property of PuLID is that the image elements (e.g., background,
lighting, composition, and style) before and after the ID insertion are kept as
consistent as possible. Codes and models will be available at
https://github.com/ToTheBeginning/PuLID
ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with
  Reward Feedback Learning
Authors: name: Weifeng Chen, name: Jiacheng Zhang, name: Jie Wu, name: Hefeng Wu, name: Xuefeng Xiao, name: Liang Lin
Published: 2024-04-23 18:41:56

The rapid development of diffusion models has triggered diverse applications.
Identity-preserving text-to-image generation (ID-T2I) particularly has received
significant attention due to its wide range of application scenarios like AI
portrait and advertising. While existing ID-T2I methods have demonstrated
impressive results, several key challenges remain: (1) It is hard to maintain
the identity characteristics of reference portraits accurately, (2) The
generated images lack aesthetic appeal especially while enforcing identity
retention, and (3) There is a limitation that cannot be compatible with
LoRA-based and Adapter-based methods simultaneously. To address these issues,
we present ID-Aligner, a general feedback learning framework to
enhance ID-T2I performance. To resolve identity features lost, we introduce
identity consistency reward fine-tuning to utilize the feedback from face
detection and recognition models to improve generated identity preservation.
Furthermore, we propose identity aesthetic reward fine-tuning leveraging
rewards from human-annotated preference data and automatically constructed
feedback on character structure generation to provide aesthetic tuning signals.
Thanks to its universal feedback fine-tuning framework, our method can be
readily applied to both LoRA and Adapter models, achieving consistent
performance gains. Extensive experiments on SD1.5 and SDXL diffusion models
validate the effectiveness of our approach. Project Page:
\url{https://idaligner.github.io/}
Transformers Can Represent n-gram Language Models
Authors: name: Anej Svete, name: Ryan Cotterell
Published: 2024-04-23 12:51:37

Plenty of existing work has analyzed the abilities of the transformer
architecture by describing its representational capacity with formal models of
computation. However, the focus so far has been on analyzing the architecture
in terms of language acceptance. We contend that this is an ill-suited
problem in the study of language models (LMs), which are definitionally
probability distributions over strings. In this paper, we focus on the
relationship between transformer LMs and n-gram LMs, a simple and
historically relevant class of language models. We show that transformer LMs
using the hard or sparse attention mechanisms can exactly represent any
n-gram LM, giving us a concrete lower bound on their probabilistic
representational capacity. This provides a first step towards understanding the
mechanisms that transformer LMs can use to represent probability distributions
over strings.
FlashSpeech: Efficient Zero-Shot Speech Synthesis
Authors: name: Zhen Ye, name: Zeqian Ju, name: Haohe Liu, name: Xu Tan, name: Jianyi Chen, name: Yiwen Lu, name: Peiwen Sun, name: Jiahao Pan, name: Weizhen Bian, name: Shulin He, name: Qifeng Liu, name: Yike Guo, name: Wei Xue
Published: 2024-04-23 02:57:46

Recent progress in large-scale zero-shot speech synthesis has been
significantly advanced by language models and diffusion models. However, the
generation process of both methods is slow and computationally intensive.
Efficient speech synthesis using a lower computing budget to achieve quality on
par with previous work remains a significant challenge. In this paper, we
present FlashSpeech, a large-scale zero-shot speech synthesis system with
approximately 5\% of the inference time compared with previous work.
FlashSpeech is built on the latent consistency model and applies a novel
adversarial consistency training approach that can train from scratch without
the need for a pre-trained diffusion model as the teacher. Furthermore, a new
prosody generator module enhances the diversity of prosody, making the rhythm
of the speech sound more natural. The generation processes of FlashSpeech can
be achieved efficiently with one or two sampling steps while maintaining high
audio quality and high similarity to the audio prompt for zero-shot speech
generation. Our experimental results demonstrate the superior performance of
FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other
zero-shot speech synthesis systems while maintaining comparable performance in
terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates
its versatility by efficiently performing tasks like voice conversion, speech
editing, and diverse speech sampling. Audio samples can be found in
https://flashspeech.github.io/.
Pegasus-v1 Technical Report
Authors: name: Raehyuk Jung, name: Hyojun Go, name: Jaehyuk Yi, name: Jiho Jang, name: Daniel Kim, name: Jay Suh, name: Aiden Lee, name: Cooper Han, name: Jae Lee, name: Jeff Kim, name: Jin-Young Kim, name: Junwan Kim, name: Kyle Park, name: Lucas Lee, name: Mars Ha, name: Minjoon Seo, name: Abraham Jo, name: Ed Park, name: Hassan Kianinejad, name: SJ Kim, name: Tony Moon, name: Wade Jeong, name: Andrei Popescu, name: Esther Kim, name: EK Yoon, name: Genie Heo, name: Henry Choi, name: Jenna Kang, name: Kevin Han, name: Noah Seo, name: Sunny Nguyen, name: Ryan Won, name: Yeonhoo Park, name: Anthony Giuliani, name: Dave Chung, name: Hans Yoon, name: James Le, name: Jenny Ahn, name: June Lee, name: Maninder Saini, name: Meredith Sanders, name: Soyoung Lee, name: Sue Kim, name: Travis Couture
Published: 2024-04-23 02:32:57

This technical report introduces Pegasus-1, a multimodal language model
specialized in video content understanding and interaction through natural
language. Pegasus-1 is designed to address the unique challenges posed by video
data, such as interpreting spatiotemporal information, to offer nuanced video
content comprehension across various lengths. This technical report overviews
Pegasus-1's architecture, training strategies, and its performance in
benchmarks on video conversation, zero-shot video question answering, and video
summarization. We also explore qualitative characteristics of Pegasus-1 ,
demonstrating its capabilities as well as its limitations, in order to provide
readers a balanced view of its current state and its future direction.
Multi-Head Mixture-of-Experts
Authors: name: Xun Wu, name: Shaohan Huang, name: Wenhui Wang, name: Furu Wei
Published: 2024-04-23 13:47:09

Sparse Mixtures of Experts (SMoE) scales model capacity without significant
increases in training and inference costs, but exhibits the following two
issues: (1) Low expert activation, where only a small subset of experts are
activated for optimization. (2) Lacking fine-grained analytical capabilities
for multiple semantic concepts within individual tokens. We propose Multi-Head
Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each
token into multiple sub-tokens. These sub-tokens are then assigned to and
processed by a diverse set of experts in parallel, and seamlessly reintegrated
into the original token form. The multi-head mechanism enables the model to
collectively attend to information from various representation spaces within
different experts, while significantly enhances expert activation, thus deepens
context understanding and alleviate overfitting. Moreover, our MH-MoE is
straightforward to implement and decouples from other SMoE optimization
methods, making it easy to integrate with other SMoE models for enhanced
performance. Extensive experimental results across three tasks: English-focused
language modeling, Multi-lingual language modeling and Masked multi-modality
modeling tasks, demonstrate the effectiveness of MH-MoE.
OpenELM: An Efficient Language Model Family with Open-source Training
  and Inference Framework
Authors: name: Sachin Mehta, name: Mohammad Hossein Sekhavat, name: Qingqing Cao, name: Maxwell Horton, name: Yanzi Jin, name: Chenfan Sun, name: Iman Mirzadeh, name: Mahyar Najibi, name: Dmitry Belenko, name: Peter Zatloukal, name: Mohammad Rastegari
Published: 2024-04-22 23:12:03

The reproducibility and transparency of large language models are crucial for
advancing open research, ensuring the trustworthiness of results, and enabling
investigations into data and model biases, as well as potential risks. To this
end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a
layer-wise scaling strategy to efficiently allocate parameters within each
layer of the transformer model, leading to enhanced accuracy. For example, with
a parameter budget of approximately one billion parameters, OpenELM exhibits a
2.36% improvement in accuracy compared to OLMo while requiring 2times fewer
pre-training tokens.
  Diverging from prior practices that only provide model weights and inference
code, and pre-train on private datasets, our release includes the complete
framework for training and evaluation of the language model on publicly
available datasets, including training logs, multiple checkpoints, and
pre-training configurations. We also release code to convert models to MLX
library for inference and fine-tuning on Apple devices. This comprehensive
release aims to empower and strengthen the open research community, paving the
way for future open research endeavors.
  Our source code along with pre-trained model weights and training recipes is
available at https://github.com/apple/corenet. Additionally, \model
models can be found on HuggingFace at:
https://huggingface.co/apple/OpenELM.
Align Your Steps: Optimizing Sampling Schedules in Diffusion Models
Authors: name: Amirmojtaba Sabour, name: Sanja Fidler, name: Karsten Kreis
Published: 2024-04-22 18:18:41

Diffusion models (DMs) have established themselves as the state-of-the-art
generative modeling approach in the visual domain and beyond. A crucial
drawback of DMs is their slow sampling speed, relying on many sequential
function evaluations through large neural networks. Sampling from DMs can be
seen as solving a differential equation through a discretized set of noise
levels known as the sampling schedule. While past works primarily focused on
deriving efficient solvers, little attention has been given to finding optimal
sampling schedules, and the entire literature relies on hand-crafted
heuristics. In this work, for the first time, we propose a general and
principled approach to optimizing the sampling schedules of DMs for
high-quality outputs, called Align Your Steps. We leverage methods
from stochastic calculus and find optimal schedules specific to different
solvers, trained DMs and datasets. We evaluate our novel approach on several
image, video as well as 2D toy data synthesis benchmarks, using a variety of
different samplers, and observe that our optimized schedules outperform
previous hand-crafted schedules in almost all experiments. Our method
demonstrates the untapped potential of sampling schedule optimization,
especially in the few-step synthesis regime.
SnapKV: LLM Knows What You are Looking for Before Generation
Authors: name: Yuhong Li, name: Yingbing Huang, name: Bowen Yang, name: Bharat Venkitesh, name: Acyr Locatelli, name: Hanchen Ye, name: Tianle Cai, name: Patrick Lewis, name: Deming Chen
Published: 2024-04-22 17:42:58

Large Language Models (LLMs) have made remarkable progress in processing
extensive contexts, with the Key-Value (KV) cache playing a vital role in
enhancing their performance. However, the growth of the KV cache in response to
increasing input length poses challenges to memory and time efficiency. To
address this problem, this paper introduces SnapKV, an innovative and
fine-tuning-free approach that efficiently minimizes KV cache size while still
delivering comparable performance in real-world applications.
  We discover that each attention head in the model consistently focuses on
specific prompt attention features during generation. Meanwhile, this robust
pattern can be obtained from an `observation' window located at the end of the
prompts. Drawing on this insight, SnapKV automatically compresses KV caches by
selecting clustered important KV positions for each attention head. Our
approach significantly reduces the growing computational overhead and memory
footprint when processing long input sequences. Specifically, SnapKV achieves a
consistent decoding speed with a 3.6x increase in generation speed and an 8.2x
enhancement in memory efficiency compared to baseline when processing inputs of
16K tokens. At the same time, it maintains comparable performance to baseline
models across 16 long sequence datasets. Moreover, SnapKV can process up to
380K context tokens on a single A100-80GB GPU using HuggingFace implementation
with minor changes, exhibiting only a negligible accuracy drop in the
Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's
potential for practical applications.
Learning H-Infinity Locomotion Control
Authors: name: Junfeng Long, name: Wenye Yu, name: Quanyi Li, name: Zirui Wang, name: Dahua Lin, name: Jiangmiao Pang
Published: 2024-04-22 17:59:07

Stable locomotion in precipitous environments is an essential capability of
quadruped robots, demanding the ability to resist various external
disturbances. However, recent learning-based policies only use basic domain
randomization to improve the robustness of learned policies, which cannot
guarantee that the robot has adequate disturbance resistance capabilities. In
this paper, we propose to model the learning process as an adversarial
interaction between the actor and a newly introduced disturber and ensure their
optimization with H_{infty} constraint. In contrast to the actor that
maximizes the discounted overall reward, the disturber is responsible for
generating effective external forces and is optimized by maximizing the error
between the task reward and its oracle, i.e., "cost" in each iteration. To keep
joint optimization between the actor and the disturber stable, our H_{infty}
constraint mandates the bound of ratio between the cost to the intensity of the
external forces. Through reciprocal interaction throughout the training phase,
the actor can acquire the capability to navigate increasingly complex physical
disturbances. We verify the robustness of our approach on quadrupedal
locomotion tasks with Unitree Aliengo robot, and also a more challenging task
with Unitree A1 robot, where the quadruped is expected to perform locomotion
merely on its hind legs as if it is a bipedal robot. The simulated quantitative
results show improvement against baselines, demonstrating the effectiveness of
the method and each design choice. On the other hand, real-robot experiments
qualitatively exhibit how robust the policy is when interfering with various
disturbances on various terrains, including stairs, high platforms, slopes, and
slippery terrains. All code, checkpoints, and real-world deployment guidance
will be made public.
Scene Coordinate Reconstruction: Posing of Image Collections via
  Incremental Learning of a Relocalizer
Authors: name: Eric Brachmann, name: Jamie Wynn, name: Shuai Chen, name: Tommaso Cavallari, name: Áron Monszpart, name: Daniyar Turmukhambetov, name: Victor Adrian Prisacariu
Published: 2024-04-22 17:02:33

We address the task of estimating camera parameters from a set of images
depicting a scene. Popular feature-based structure-from-motion (SfM) tools
solve this task by incremental reconstruction: they repeat triangulation of
sparse 3D points and registration of more camera views to the sparse point
cloud. We re-interpret incremental structure-from-motion as an iterated
application and refinement of a visual relocalizer, that is, of a method that
registers new views to the current state of the reconstruction. This
perspective allows us to investigate alternative visual relocalizers that are
not rooted in local feature matching. We show that scene coordinate regression,
a learning-based relocalization approach, allows us to build implicit, neural
scene representations from unposed images. Different from other learning-based
reconstruction methods, we do not require pose priors nor sequential inputs,
and we optimize efficiently over thousands of images. Our method, ACE0 (ACE
Zero), estimates camera poses to an accuracy comparable to feature-based SfM,
as demonstrated by novel view synthesis. Project page:
https://nianticlabs.github.io/acezero/