Orionnnn/hub-stats huggingface dataset:
DeAL: Decoding-time Alignment for Large Language Models
Authors: name: James Y. Huang, name: Sailik Sengupta, name: Daniele Bonadiman, name: Yi-an Lai, name: Arshit Gupta, name: Nikolaos Pappas, name: Saab Mansour, name: Katrin Kirchoff, name: Dan Roth
Published: 2024-02-05 06:12:29

Large Language Models (LLMs) are nowadays expected to generate content
aligned with human preferences. Current work focuses on alignment at model
training time, through techniques such as Reinforcement Learning with Human
Feedback (RLHF). However, it is unclear if such methods are an effective choice
to teach alignment objectives to the model. First, the inability to incorporate
multiple, custom rewards and reliance on a model developer's view of universal
and static principles are key limitations. Second, the residual gaps in model
training and the reliability of such approaches are also questionable (e.g.
susceptibility to jail-breaking even after safety training). To address these,
we propose DeAL, a framework that allows the user to customize reward functions
and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view
decoding as a heuristic-guided search process and facilitate the use of a wide
variety of alignment objectives. Our experiments with programmatic constraints
such as keyword and length constraints (studied widely in the pre-LLM era) and
abstract objectives such as harmlessness and helpfulness (proposed in the
post-LLM era) show that we can DeAL with fine-grained trade-offs, improve
adherence to alignment objectives, and address residual gaps in LLMs. Lastly,
while DeAL can be effectively paired with RLHF and prompting techniques, its
generality makes decoding slower, an optimization we leave for future work.
Model Editing with Canonical Examples
Authors: name: John Hewitt, name: Sarah Chen, name: Lanruo Lora Xie, name: Edward Adams, name: Percy Liang, name: Christopher D. Manning
Published: 2024-02-09 03:08:12

We introduce model editing with canonical examples, a setting in which (1) a
single learning example is provided per desired behavior, (2) evaluation is
performed exclusively out-of-distribution, and (3) deviation from an initial
model is strictly limited. A canonical example is a simple instance of good
behavior, e.g., The capital of Mauritius is Port Louis) or bad behavior, e.g.,
An aspect of researchers is coldhearted). The evaluation set contains more
complex examples of each behavior (like a paragraph in which the capital of
Mauritius is called for.) We create three datasets and modify three more for
model editing with canonical examples, covering knowledge-intensive
improvements, social bias mitigation, and syntactic edge cases. In our
experiments on Pythia language models, we find that LoRA outperforms full
finetuning and MEMIT. We then turn to the Backpack language model architecture
because it is intended to enable targeted improvement. The Backpack defines a
large bank of sense vectors--a decomposition of the different uses of each
word--which are weighted and summed to form the output logits of the model. We
propose sense finetuning, which selects and finetunes a few (approx 10)
sense vectors for each canonical example, and find that it outperforms other
finetuning methods, e.g., 4.8% improvement vs 0.3%. Finally, we improve
GPT-J-6B by an inference-time ensemble with just the changes from sense
finetuning of a 35x smaller Backpack, in one setting outperforming editing
GPT-J itself (4.1% vs 1.0%).
Aya Dataset: An Open-Access Collection for Multilingual Instruction
  Tuning
Authors: name: Shivalika Singh, name: Freddie Vargus, name: Daniel Dsouza, name: Börje F. Karlsson, name: Abinaya Mahendiran, name: Wei-Yin Ko, name: Herumb Shandilya, name: Jay Patel, name: Deividas Mataciunas, name: Laura OMahony, name: Mike Zhang, name: Ramith Hettiarachchi, name: Joseph Wilson, name: Marina Machado, name: Luisa Souza Moura, name: Dominik Krzemiński, name: Hakimeh Fadaei, name: Irem Ergün, name: Ifeoma Okoh, name: Aisha Alaagib, name: Oshan Mudannayake, name: Zaid Alyafeai, name: Vu Minh Chien, name: Sebastian Ruder, name: Surya Guthikonda, name: Emad A. Alghamdi, name: Sebastian Gehrmann, name: Niklas Muennighoff, name: Max Bartolo, name: Julia Kreutzer, name: Ahmet Üstün, name: Marzieh Fadaee, name: Sara Hooker
Published: 2024-02-09 18:51:49

Datasets are foundational to many breakthroughs in modern artificial
intelligence. Many recent achievements in the space of natural language
processing (NLP) can be attributed to the finetuning of pre-trained models on a
diverse set of tasks that enables a large language model (LLM) to respond to
instructions. Instruction fine-tuning (IFT) requires specifically constructed
and annotated datasets. However, existing datasets are almost all in the
English language. In this work, our primary goal is to bridge the language gap
by building a human-curated instruction-following dataset spanning 65
languages. We worked with fluent speakers of languages from around the world to
collect natural instances of instructions and completions. Furthermore, we
create the most extensive multilingual collection to date, comprising 513
million instances through templating and translating existing datasets across
114 languages. In total, we contribute four key resources: we develop and
open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection,
and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case
study in participatory research, involving collaborators from 119 countries. We
see this as a valuable framework for future research collaborations that aim to
bridge gaps in resources.
MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models
Authors: name: Yixiao Zhang, name: Yukara Ikemiya, name: Gus Xia, name: Naoki Murata, name: Marco Martínez, name: Wei-Hsiang Liao, name: Yuki Mitsufuji, name: Simon Dixon
Published: 2024-02-09 04:34:08

Recent advances in text-to-music generation models have opened new avenues in
musical creativity. However, music generation usually involves iterative
refinements, and how to edit the generated music remains a significant
challenge. This paper introduces a novel approach to the editing of music
generated by such models, enabling the modification of specific attributes,
such as genre, mood and instrument, while maintaining other aspects unchanged.
Our method transforms text editing to latent space manipulation while
adding an extra constraint to enforce consistency. It seamlessly integrates
with existing pretrained text-to-music diffusion models without requiring
additional training. Experimental results demonstrate superior performance over
both zero-shot and certain supervised baselines in style and timbre transfer
evaluations. Additionally, we showcase the practical applicability of our
approach in real-world music editing scenarios.
HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting
Authors: name: Zhenglin Zhou, name: Fan Ma, name: Hehe Fan, name: Yi Yang
Published: 2024-02-09 02:58:37

Creating digital avatars from textual prompts has long been a desirable yet
challenging task. Despite the promising outcomes obtained through 2D diffusion
priors in recent works, current methods face challenges in achieving
high-quality and animated avatars effectively. In this paper, we present
HeadStudio, a novel framework that utilizes 3D Gaussian splatting to
generate realistic and animated avatars from text prompts. Our method drives 3D
Gaussians semantically to create a flexible and achievable appearance through
the intermediate FLAME representation. Specifically, we incorporate the FLAME
into both 3D representation and score distillation: 1) FLAME-based 3D Gaussian
splatting, driving 3D Gaussian points by rigging each point to a FLAME mesh. 2)
FLAME-based score distillation sampling, utilizing FLAME-based fine-grained
control signal to guide score distillation from the text prompt. Extensive
experiments demonstrate the efficacy of HeadStudio in generating animatable
avatars from textual prompts, exhibiting visually appealing appearances. The
avatars are capable of rendering high-quality real-time (geq 40 fps) novel
views at a resolution of 1024. They can be smoothly controlled by real-world
speech and video. We hope that HeadStudio can advance digital avatar creation
and that the present method can widely be applied across various domains.
Animated Stickers: Bringing Stickers to Life with Video Diffusion
Authors: name: David Yan, name: Winnie Zhang, name: Luxin Zhang, name: Anmol Kalia, name: Dingkang Wang, name: Ankit Ramchandani, name: Miao Liu, name: Albert Pumarola, name: Edgar Schoenfeld, name: Elliot Blanchard, name: Krishna Narni, name: Yaqiao Luo, name: Lawrence Chen, name: Guan Pang, name: Ali Thabet, name: Peter Vajda, name: Amy Bearman, name: Licheng Yu
Published: 2024-02-08 22:49:32

We introduce animated stickers, a video diffusion model which generates an
animation conditioned on a text prompt and static sticker image. Our model is
built on top of the state-of-the-art Emu text-to-image model, with the addition
of temporal layers to model motion. Due to the domain gap, i.e. differences in
visual and motion style, a model which performed well on generating natural
videos can no longer generate vivid videos when applied to stickers. To bridge
this gap, we employ a two-stage finetuning pipeline: first with weakly
in-domain data, followed by human-in-the-loop (HITL) strategy which we term
ensemble-of-teachers. It distills the best qualities of multiple teachers into
a smaller student model. We show that this strategy allows us to specifically
target improvements to motion quality while maintaining the style from the
static image. With inference optimizations, our model is able to generate an
eight-frame video with high-quality, interesting, and relevant motion in under
one second.
Real-World Fluid Directed Rigid Body Control via Deep Reinforcement
  Learning
Authors: name: Mohak Bhardwaj, name: Thomas Lampe, name: Michael Neunert, name: Francesco Romano, name: Abbas Abdolmaleki, name: Arunkumar Byravan, name: Markus Wulfmeier, name: Martin Riedmiller, name: Jonas Buchli
Published: 2024-02-08 23:35:03

Recent advances in real-world applications of reinforcement learning (RL)
have relied on the ability to accurately simulate systems at scale. However,
domains such as fluid dynamical systems exhibit complex dynamic phenomena that
are hard to simulate at high integration rates, limiting the direct application
of modern deep RL algorithms to often expensive or safety critical hardware. In
this work, we introduce "Box o Flows", a novel benchtop experimental control
system for systematically evaluating RL algorithms in dynamic real-world
scenarios. We describe the key components of the Box o Flows, and through a
series of experiments demonstrate how state-of-the-art model-free RL algorithms
can synthesize a variety of complex behaviors via simple reward specifications.
Furthermore, we explore the role of offline RL in data-efficient hypothesis
testing by reusing past experiences. We believe that the insights gained from
this preliminary study and the availability of systems like the Box o Flows
support the way forward for developing systematic RL algorithms that can be
generally applied to complex, dynamical systems. Supplementary material and
videos of experiments are available at
https://sites.google.com/view/box-o-flows/home.
Premier-TACO: Pretraining Multitask Representation via Temporal
  Action-Driven Contrastive Loss
Authors: name: Ruijie Zheng, name: Yongyuan Liang, name: Xiyao Wang, name: Shuang Ma, name: Hal Daumé III, name: Huazhe Xu, name: John Langford, name: Praveen Palanisamy, name: Kalyan Shankar Basu, name: Furong Huang
Published: 2024-02-09 05:04:40

We present Premier-TACO, a multitask feature representation learning approach
designed to improve few-shot policy learning efficiency in sequential
decision-making tasks. Premier-TACO leverages a subset of multitask offline
datasets for pretraining a general feature representation, which captures
critical environmental dynamics and is fine-tuned using minimal expert
demonstrations. It advances the temporal action contrastive learning (TACO)
objective, known for state-of-the-art results in visual control tasks, by
incorporating a novel negative example sampling strategy. This strategy is
crucial in significantly boosting TACO's computational efficiency, making
large-scale multitask offline pretraining feasible. Our extensive empirical
evaluation in a diverse set of continuous control benchmarks including Deepmind
Control Suite, MetaWorld, and LIBERO demonstrate Premier-TACO's effectiveness
in pretraining visual representations, significantly enhancing few-shot
imitation learning of novel tasks. Our code, pretraining data, as well as
pretrained model checkpoints will be released at
https://github.com/PremierTACO/premier-taco.
Multilingual E5 Text Embeddings: A Technical Report
Authors: name: Liang Wang, name: Nan Yang, name: Xiaolong Huang, name: Linjun Yang, name: Rangan Majumder, name: Furu Wei
Published: 2024-02-08 13:47:50

This technical report presents the training methodology and evaluation
results of the open-source multilingual E5 text embedding models, released in
mid-2023. Three embedding models of different sizes (small / base / large) are
provided, offering a balance between the inference efficiency and embedding
quality. The training procedure adheres to the English E5 model recipe,
involving contrastive pre-training on 1 billion multilingual text pairs,
followed by fine-tuning on a combination of labeled datasets. Additionally, we
introduce a new instruction-tuned embedding model, whose performance is on par
with state-of-the-art, English-only models of similar sizes. Information
regarding the model release can be found at
https://github.com/microsoft/unilm/tree/master/e5 .
Memory Consolidation Enables Long-Context Video Understanding
Authors: name: Ivana Balažević, name: Yuge Shi, name: Pinelopi Papalampidi, name: Rahma Chaabouni, name: Skanda Koppula, name: Olivier J. Hénaff
Published: 2024-02-08 17:50:22

Most transformer-based video encoders are limited to short temporal contexts
due to their quadratic complexity. While various attempts have been made to
extend this context, this has often come at the cost of both conceptual and
computational complexity. We propose to instead re-purpose existing pre-trained
video transformers by simply fine-tuning them to attend to memories derived
non-parametrically from past activations. By leveraging redundancy reduction,
our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
context far into the past and exhibits excellent scaling behavior when learning
from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
long-context video understanding on EgoSchema, Perception Test, and Diving48,
outperforming methods that benefit from orders of magnitude more parameters.
Offline Actor-Critic Reinforcement Learning Scales to Large Models
Authors: name: Jost Tobias Springenberg, name: Abbas Abdolmaleki, name: Jingwei Zhang, name: Oliver Groth, name: Michael Bloesch, name: Thomas Lampe, name: Philemon Brakel, name: Sarah Bechtle, name: Steven Kapturowski, name: Roland Hafner, name: Nicolas Heess, name: Martin Riedmiller
Published: 2024-02-08 10:29:46

We show that offline actor-critic reinforcement learning can scale to large
models - such as transformers - and follows similar scaling laws as supervised
learning. We find that offline actor-critic algorithms can outperform strong,
supervised, behavioral cloning baselines for multi-task training on a large
dataset containing both sub-optimal and expert behavior on 132 continuous
control tasks. We introduce a Perceiver-based actor-critic model and elucidate
the key model features needed to make offline RL work with self- and
cross-attention modules. Overall, we find that: i) simple offline actor critic
algorithms are a natural choice for gradually moving away from the currently
predominant paradigm of behavioral cloning, and ii) via offline RL it is
possible to learn multi-task policies that master many domains simultaneously,
including real robotics tasks, from sub-optimal demonstrations or
self-generated data.
InstaGen: Enhancing Object Detection by Training on Synthetic Dataset
Authors: name: Chengjian Feng, name: Yujie Zhong, name: Zequn Jie, name: Weidi Xie, name: Lin Ma
Published: 2024-02-08 18:59:53

In this paper, we introduce a novel paradigm to enhance the ability of object
detector, e.g., expanding categories or improving detection performance, by
training on synthetic dataset generated from diffusion models. Specifically, we
integrate an instance-level grounding head into a pre-trained, generative
diffusion model, to augment it with the ability of localising arbitrary
instances in the generated images. The grounding head is trained to align the
text embedding of category names with the regional visual feature of the
diffusion model, using supervision from an off-the-shelf object detector, and a
novel self-training scheme on (novel) categories not covered by the detector.
This enhanced version of diffusion model, termed as InstaGen, can serve as a
data synthesizer for object detection. We conduct thorough experiments to show
that, object detector can be enhanced while training on the synthetic dataset
from InstaGen, demonstrating superior performance over existing
state-of-the-art methods in open-vocabulary (+4.5 AP) and data-sparse (+1.2 to
5.2 AP) scenarios.
Implicit Diffusion: Efficient Optimization through Stochastic Sampling
Authors: name: Pierre Marion, name: Anna Korba, name: Peter Bartlett, name: Mathieu Blondel, name: Valentin De Bortoli, name: Arnaud Doucet, name: Felipe Llinares-López, name: Courtney Paquette, name: Quentin Berthet
Published: 2024-02-08 08:00:11

We present a new algorithm to optimize distributions defined implicitly by
parameterized stochastic diffusions. Doing so allows us to modify the outcome
distribution of sampling processes by optimizing over their parameters. We
introduce a general framework for first-order optimization of these processes,
that performs jointly, in a single loop, optimization and sampling steps. This
approach is inspired by recent advances in bilevel optimization and automatic
implicit differentiation, leveraging the point of view of sampling as
optimization over the space of probability distributions. We provide
theoretical guarantees on the performance of our method, as well as
experimental results demonstrating its effectiveness in real-world settings.
λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion
  Models by Leveraging CLIP Latent Space
Authors: name: Maitreya Patel, name: Sangmin Jung, name: Chitta Baral, name: Yezhou Yang
Published: 2024-02-07 19:07:10

Despite the recent advances in personalized text-to-image (P-T2I) generative
models, subject-driven T2I remains challenging. The primary bottlenecks include
1) Intensive training resource requirements, 2) Hyper-parameter sensitivity
leading to inconsistent outputs, and 3) Balancing the intricacies of novel
visual concept and composition alignment. We start by re-iterating the core
philosophy of T2I diffusion models to address the above limitations.
Predominantly, contemporary subject-driven T2I approaches hinge on Latent
Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention
layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the
latent space of these diffusion models significantly escalates resource
demands, leading to inconsistent results and necessitating numerous iterations
for a single desired image. Recently, ECLIPSE has demonstrated a more
resource-efficient pathway for training UnCLIP-based T2I models, circumventing
the need for diffusion text-to-image priors. Building on this, we introduce
lambda-ECLIPSE. Our method illustrates that effective P-T2I does not
necessarily depend on the latent space of diffusion models. lambda-ECLIPSE
achieves single, multi-subject, and edge-guided T2I personalization with just
34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text
interleaved data. Through extensive experiments, we also establish that
lambda-ECLIPSE surpasses existing baselines in composition alignment while
preserving concept alignment performance, even with significantly lower
resource utilization.
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
  Language Models
Authors: name: Peng Gao, name: Renrui Zhang, name: Chris Liu, name: Longtian Qiu, name: Siyuan Huang, name: Weifeng Lin, name: Shitian Zhao, name: Shijie Geng, name: Ziyi Lin, name: Peng Jin, name: Kaipeng Zhang, name: Wenqi Shao, name: Chao Xu, name: Conghui He, name: Junjun He, name: Hao Shao, name: Pan Lu, name: Hongsheng Li, name: Yu Qiao
Published: 2024-02-08 18:59:48

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
series developed upon SPHINX. To improve the architecture and training
efficiency, we modify the SPHINX framework by removing redundant visual
encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
dataset covering publicly available resources in language, vision, and
vision-language tasks. We further enrich this collection with our curated OCR
intensive and Set-of-Mark datasets, extending the diversity and generality. By
training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
parameter size and multilingual capabilities. Comprehensive benchmarking
reveals a strong correlation between the multi-modal performance with the data
and parameter scales. Code and models are released at
https://github.com/Alpha-VLLM/LLaMA2-Accessory
Driving Everywhere with Large Language Model Policy Adaptation
Authors: name: Boyi Li, name: Yue Wang, name: Jiageng Mao, name: Boris Ivanovic, name: Sushant Veer, name: Karen Leung, name: Marco Pavone
Published: 2024-02-08 18:59:03

Adapting driving behavior to new environments, customs, and laws is a
long-standing problem in autonomous driving, precluding the widespread
deployment of autonomous vehicles (AVs). In this paper, we present LLaDA, a
simple yet powerful tool that enables human drivers and autonomous vehicles
alike to drive everywhere by adapting their tasks and motion plans to traffic
rules in new locations. LLaDA achieves this by leveraging the impressive
zero-shot generalizability of large language models (LLMs) in interpreting the
traffic rules in the local driver handbook. Through an extensive user study, we
show that LLaDA's instructions are useful in disambiguating in-the-wild
unexpected situations. We also demonstrate LLaDA's ability to adapt AV motion
planning policies in real-world datasets; LLaDA outperforms baseline planning
approaches on all our metrics. Please check our website for more details:
https://boyiliee.github.io/llada.