Orionnnn/hub-stats huggingface dataset:
V-IRL: Grounding Virtual Intelligence in Real Life
Authors: name: Jihan Yang, name: Runyu Ding, name: Ellis Brown, name: Xiaojuan Qi, name: Saining Xie
Published: 2024-02-05 18:59:36

There is a sensory gulf between the Earth that humans inhabit and the digital
realms in which modern AI agents are created. To develop AI agents that can
sense, think, and act as flexibly as humans in real-world settings, it is
imperative to bridge the realism gap between the digital and physical worlds.
How can we embody agents in an environment as rich and diverse as the one we
inhabit, without the constraints imposed by real hardware and control? Towards
this end, we introduce V-IRL: a platform that enables agents to scalably
interact with the real world in a virtual yet realistic environment. Our
platform serves as a playground for developing agents that can accomplish
various practical tasks and as a vast testbed for measuring progress in
capabilities spanning perception, decision-making, and interaction with
real-world data across the entire globe.
Video-LaVIT: Unified Video-Language Pre-training with Decoupled
  Visual-Motional Tokenization
Authors: name: Yang Jin, name: Zhicheng Sun, name: Kun Xu, name: Kun Xu, name: Liwei Chen, name: Hao Jiang, name: Quzhe Huang, name: Chengru Song, name: Yuliang Liu, name: Di Zhang, name: Yang Song, name: Kun Gai, name: Yadong Mu
Published: 2024-02-05 16:30:49

In light of recent advances in multimodal Large Language Models (LLMs), there
is increasing attention to scaling them from image-text data to more
informative real-world videos. Compared to static images, video poses unique
challenges for effective large-scale pre-training due to the modeling of its
spatiotemporal dynamics. In this paper, we address such limitations in
video-language pre-training with an efficient video decomposition that
represents each video as keyframes and temporal motions. These are then adapted
to an LLM using well-designed tokenizers that discretize visual and temporal
information as a few tokens, thus enabling unified generative pre-training of
videos, images, and text. At inference, the generated tokens from the LLM are
carefully recovered to the original continuous pixel space to create various
video content. Our proposed framework is both capable of comprehending and
generating image and video content, as demonstrated by its competitive
performance across 13 multimodal benchmarks in image and video understanding
and generation. Our code and models will be available at
https://video-lavit.github.io.
Shortened LLaMA: A Simple Depth Pruning for Large Language Models
Authors: name: Bo-Kyeong Kim, name: Geonmin Kim, name: Tae-Ho Kim, name: Thibault Castells, name: Shinkook Choi, name: Junho Shin, name: Hyoung-Kyu Song
Published: 2024-02-05 09:44:49

Structured pruning of modern large language models (LLMs) has emerged as a
way of decreasing their high computational needs. Width pruning reduces the
size of projection weight matrices (e.g., by removing attention heads) while
maintaining the number of layers. Depth pruning, in contrast, removes entire
layers or blocks, while keeping the size of the remaining weights unchanged.
Most current research focuses on either width-only or a blend of width and
depth pruning, with little comparative analysis between the two units (width
vs. depth) concerning their impact on LLM inference efficiency. In this work,
we show that a simple depth pruning approach can compete with recent width
pruning methods in terms of zero-shot task performance. Our pruning method
boosts inference speeds, especially under memory-constrained conditions that
require limited batch sizes for running LLMs, where width pruning is
ineffective. We hope this work can help deploy LLMs on local and edge devices.
Rethinking Optimization and Architecture for Tiny Language Models
Authors: name: Yehui Tang, name: Fangcheng Liu, name: Yunsheng Ni, name: Yuchuan Tian, name: Zheyuan Bai, name: Yi-Qi Hu, name: Sichao Liu, name: Shangling Jui, name: Kai Han, name: Yunhe Wang
Published: 2024-02-05 07:59:38

The power of large language models (LLMs) has been demonstrated through
numerous data and computing resources. However, the application of language
models on mobile devices is facing huge challenge on the computation and memory
costs, that is, tiny language models with high performance are urgently
required. Limited by the highly complex training process, there are many
details for optimizing language models that are seldom studied carefully. In
this study, based on a tiny language model with 1B parameters, we carefully
design a series of empirical study to analyze the effect of each component.
Three perspectives are mainly discussed, i.e., neural architecture, parameter
initialization, and optimization strategy. Several design formulas are
empirically proved especially effective for tiny language models, including
tokenizer compression, architecture tweaking, parameter inheritance and
multiple-round training. Then we train PanGu-pi-1B Pro and PanGu-pi-1.5B
Pro on 1.6T multilingual corpora, following the established formulas.
Experimental results demonstrate the improved optimization and architecture
yield a notable average improvement of 8.87 on benchmark evaluation sets for
PanGu-pi-1B Pro. Besides, PanGu-pi-1.5B Pro surpasses a range of SOTA
models with larger model sizes, validating its superior performance. The code
will be released soon (https://github.com/YuchuanTian/RethinkTinyLM).
Code Representation Learning At Scale
Authors: name: Dejiao Zhang, name: Wasi Ahmad, name: Ming Tan, name: Hantian Ding, name: Ramesh Nallapati, name: Dan Roth, name: Xiaofei Ma, name: Bing Xiang
Published: 2024-02-02 22:19:15

Recent studies have shown that code language models at scale demonstrate
significant performance gains on downstream tasks, i.e., code generation.
However, most of the existing works on code representation learning train
models at a hundred million parameter scale using very limited pretraining
corpora. In this work, we fuel code representation learning with a vast amount
of code data via a two-stage pretraining scheme. We first train the encoders
via a mix that leverages both randomness in masking language modeling and the
structure aspect of programming language. We then enhance the representations
via contrastive learning with hard negative and hard positive constructed in an
unsupervised manner. We establish an off-the-shelf encoder model that
persistently outperforms the existing models on a wide variety of downstream
tasks by large margins. To comprehend the factors contributing to successful
code representation learning, we conduct detailed ablations and share our
findings on (i) a customized and effective token-level denoising scheme for
source code; (ii) the importance of hard negatives and hard positives; (iii)
how the proposed bimodal contrastive learning boost the cross-lingual semantic
search performance; and (iv) how the pretraining schemes decide the downstream
task performance scales with the model size.
LiPO: Listwise Preference Optimization through Learning-to-Rank
Authors: name: Tianqi Liu, name: Zhen Qin, name: Junru Wu, name: Jiaming Shen, name: Misha Khalman, name: Rishabh Joshi, name: Yao Zhao, name: Mohammad Saleh, name: Simon Baumgartner, name: Jialu Liu, name: Peter J. Liu, name: Xuanhui Wang
Published: 2024-02-02 20:08:10

Aligning language models (LMs) with curated human feedback is critical to
control their behaviors in real-world applications. Several recent policy
optimization methods, such as DPO and SLiC, serve as promising alternatives to
the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In
practice, human feedback often comes in a format of a ranked list over multiple
responses to amortize the cost of reading prompt. Multiple responses can also
be ranked by reward models or AI feedback. There lacks such a study on directly
fitting upon a list of responses. In this work, we formulate the LM alignment
as a listwise ranking problem and describe the Listwise Preference Optimization
(LiPO) framework, where the policy can potentially learn more effectively from
a ranked list of plausible responses given the prompt. This view draws an
explicit connection to Learning-to-Rank (LTR), where most existing preference
optimization work can be mapped to existing ranking objectives, especially
pairwise ones. Following this connection, we provide an examination of ranking
objectives that are not well studied for LM alignment withDPO and SLiC as
special cases when list size is two. In particular, we highlight a specific
method, LiPO-{\lambda}, which leverages a state-of-the-art listwise ranking
objective and weights each preference pair in a more advanced manner. We show
that LiPO-{\lambda} can outperform DPO and SLiC by a clear margin on two
preference alignment tasks.
Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and
  Dialogue Abilities
Authors: name: Zhifeng Kong, name: Arushi Goel, name: Rohan Badlani, name: Wei Ping, name: Rafael Valle, name: Bryan Catanzaro
Published: 2024-02-02 18:58:34

Augmenting large language models (LLMs) to understand audio -- including
non-speech sounds and non-verbal speech -- is critically important for diverse
real-world applications of LLMs. In this paper, we propose Audio Flamingo, a
novel audio language model with 1) strong audio understanding abilities, 2) the
ability to quickly adapt to unseen tasks via in-context learning and retrieval,
and 3) strong multi-turn dialogue abilities. We introduce a series of training
techniques, architecture design, and data strategies to enhance our model with
these abilities. Extensive evaluations across various audio understanding tasks
confirm the efficacy of our method, setting new state-of-the-art benchmarks.
BlackMamba: Mixture of Experts for State-Space Models
Authors: name: Quentin Anthony, name: Yury Tokpanov, name: Paolo Glorioso, name: Beren Millidge
Published: 2024-02-01 07:15:58

State-space models (SSMs) have recently demonstrated competitive performance
to transformers at large-scale language modeling benchmarks while achieving
linear time and memory complexity as a function of sequence length. Mamba, a
recently released SSM model, shows impressive performance in both language
modeling and long sequence processing tasks. Simultaneously, mixture-of-expert
(MoE) models have shown remarkable performance while significantly reducing the
compute and latency costs of inference at the expense of a larger memory
footprint. In this paper, we present BlackMamba, a novel architecture that
combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate
that BlackMamba performs competitively against both Mamba and transformer
baselines, and outperforms in inference and training FLOPs. We fully train and
open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a
custom dataset. We show that BlackMamba inherits and combines both of the
benefits of SSM and MoE architectures, combining linear-complexity generation
from SSM with cheap and fast inference from MoE. We release all weights,
checkpoints, and inference code open-source. Inference code at:
https://github.com/Zyphra/BlackMamba
Rethinking Interpretability in the Era of Large Language Models
Authors: name: Chandan Singh, name: Jeevana Priya Inala, name: Michel Galley, name: Rich Caruana, name: Jianfeng Gao
Published: 2024-01-30 17:38:54

Interpretable machine learning has exploded as an area of interest over the
last decade, sparked by the rise of increasingly large datasets and deep neural
networks. Simultaneously, large language models (LLMs) have demonstrated
remarkable capabilities across a wide array of tasks, offering a chance to
rethink opportunities in interpretable machine learning. Notably, the
capability to explain in natural language allows LLMs to expand the scale and
complexity of patterns that can be given to a human. However, these new
capabilities raise new challenges, such as hallucinated explanations and
immense computational costs.
  In this position paper, we start by reviewing existing methods to evaluate
the emerging field of LLM interpretation (both interpreting LLMs and using LLMs
for explanation). We contend that, despite their limitations, LLMs hold the
opportunity to redefine interpretability with a more ambitious scope across
many applications, including in auditing LLMs themselves. We highlight two
emerging research priorities for LLM interpretation: using LLMs to directly
analyze new datasets and to generate interactive explanations.
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Authors: name: Fuzhao Xue, name: Zian Zheng, name: Yao Fu, name: Jinjie Ni, name: Zangwei Zheng, name: Wangchunshu Zhou, name: Yang You
Published: 2024-01-29 12:05:02

To help the open-source community have a better understanding of
Mixture-of-Experts (MoE) based large language models (LLMs), we train and
release OpenMoE, a series of fully open-sourced and reproducible decoder-only
MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T
tokens. Our investigation confirms that MoE-based LLMs can offer a more
favorable cost-effectiveness trade-off than dense LLMs, highlighting the
potential effectiveness for future LLM development.
  One more important contribution of this study is an in-depth analysis of the
routing mechanisms within our OpenMoE models, leading to three significant
findings: Context-Independent Specialization, Early Routing Learning, and
Drop-towards-the-End. We discovered that routing decisions in MoE models are
predominantly based on token IDs, with minimal context relevance. The
token-to-expert assignments are determined early in the pre-training phase and
remain largely unchanged. This imperfect routing can result in performance
degradation, particularly in sequential tasks like multi-turn conversations,
where tokens appearing later in a sequence are more likely to be dropped.
Finally, we rethink our design based on the above-mentioned observations and
analysis. To facilitate future MoE LLM development, we propose potential
strategies for mitigating the issues we found and further improving
off-the-shelf MoE LLM designs.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open
  Language Models
Authors: name: Zhihong Shao, name: Peiyi Wang, name: Qihao Zhu, name: Runxin Xu, name: Junxiao Song, name: Mingchuan Zhang, name: Y. K. Li, name: Y. Wu, name: Daya Guo
Published: 2024-02-05 18:55:32

Mathematical reasoning poses a significant challenge for language models due
to its complex and structured nature. In this paper, we introduce DeepSeekMath
7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B
math-related tokens sourced from Common Crawl, together with natural language
and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the
competition-level MATH benchmark without relying on external toolkits and
voting techniques, approaching the performance level of Gemini-Ultra and GPT-4.
Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.
The mathematical reasoning capability of DeepSeekMath is attributed to two key
factors: First, we harness the significant potential of publicly available web
data through a meticulously engineered data selection pipeline. Second, we
introduce Group Relative Policy Optimization (GRPO), a variant of Proximal
Policy Optimization (PPO), that enhances mathematical reasoning abilities while
concurrently optimizing the memory usage of PPO.
Training-Free Consistent Text-to-Image Generation
Authors: name: Yoad Tewel, name: Omri Kaduri, name: Rinon Gal, name: Yoni Kasten, name: Lior Wolf, name: Gal Chechik, name: Yuval Atzmon
Published: 2024-02-05 18:42:34

Text-to-image models offer a new level of creative flexibility by allowing
users to guide the image generation process through natural language. However,
using these models to consistently portray the same subject across diverse
prompts remains challenging. Existing approaches fine-tune the model to teach
it new words that describe specific user-provided subjects or add image
conditioning to the model. These methods require lengthy per-subject
optimization or large-scale pre-training. Moreover, they struggle to align
generated images with text prompts and face difficulties in portraying multiple
subjects. Here, we present ConsiStory, a training-free approach that enables
consistent subject generation by sharing the internal activations of the
pretrained model. We introduce a subject-driven shared attention block and
correspondence-based feature injection to promote subject consistency between
images. Additionally, we develop strategies to encourage layout diversity while
maintaining subject consistency. We compare ConsiStory to a range of baselines,
and demonstrate state-of-the-art performance on subject consistency and text
alignment, without requiring a single optimization step. Finally, ConsiStory
can naturally extend to multi-subject scenarios, and even enable training-free
personalization for common objects.
DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image
  Editing
Authors: name: Chong Mou, name: Xintao Wang, name: Jiechong Song, name: Ying Shan, name: Jian Zhang
Published: 2024-02-04 18:50:29

Large-scale Text-to-Image (T2I) diffusion models have revolutionized image
generation over the last few years. Although owning diverse and high-quality
generation capabilities, translating these abilities to fine-grained image
editing remains challenging. In this paper, we propose DiffEditor to rectify
two weaknesses in existing diffusion-based image editing: (1) in complex
scenarios, editing results often lack editing accuracy and exhibit unexpected
artifacts; (2) lack of flexibility to harmonize editing operations, e.g.,
imagine new content. In our solution, we introduce image prompts in
fine-grained image editing, cooperating with the text prompt to better describe
the editing content. To increase the flexibility while maintaining content
consistency, we locally combine stochastic differential equation (SDE) into the
ordinary differential equation (ODE) sampling. In addition, we incorporate
regional score-based gradient guidance and a time travel strategy into the
diffusion sampling, further improving the editing quality. Extensive
experiments demonstrate that our method can efficiently achieve
state-of-the-art performance on various fine-grained image editing tasks,
including editing within a single image (e.g., object moving, resizing, and
content dragging) and across images (e.g., appearance replacing and object
pasting). Our source code is released at
https://github.com/MC-E/DragonDiffusion.
Direct-a-Video: Customized Video Generation with User-Directed Camera
  Movement and Object Motion
Authors: name: Shiyuan Yang, name: Liang Hou, name: Haibin Huang, name: Chongyang Ma, name: Pengfei Wan, name: Di Zhang, name: Xiaodong Chen, name: Jing Liao
Published: 2024-02-05 16:30:57

Recent text-to-video diffusion models have achieved impressive progress. In
practice, users often desire the ability to control object motion and camera
movement independently for customized video creation. However, current methods
lack the focus on separately controlling object motion and camera movement in a
decoupled manner, which limits the controllability and flexibility of
text-to-video models. In this paper, we introduce Direct-a-Video, a system that
allows users to independently specify motions for one or multiple objects
and/or camera movements, as if directing a video. We propose a simple yet
effective strategy for the decoupled control of object motion and camera
movement. Object motion is controlled through spatial cross-attention
modulation using the model's inherent priors, requiring no additional
optimization. For camera movement, we introduce new temporal cross-attention
layers to interpret quantitative camera movement parameters. We further employ
an augmentation-based approach to train these layers in a self-supervised
manner on a small-scale dataset, eliminating the need for explicit motion
annotation. Both components operate independently, allowing individual or
combined control, and can generalize to open-domain scenarios. Extensive
experiments demonstrate the superiority and effectiveness of our method.
Project page: https://direct-a-video.github.io/.
InteractiveVideo: User-Centric Controllable Video Generation with
  Synergistic Multimodal Instructions
Authors: name: Yiyuan Zhang, name: Yuhao Kang, name: Zhixin Zhang, name: Xiaohan Ding, name: Sanyuan Zhao, name: Xiangyu Yue
Published: 2024-02-05 14:24:46

We introduce InteractiveVideo, a user-centric framework for video
generation. Different from traditional generative approaches that operate based
on user-provided images or text, our framework is designed for dynamic
interaction, allowing users to instruct the generative model through various
intuitive mechanisms during the whole generation process, e.g. text and image
prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal
Instruction mechanism, designed to seamlessly integrate users' multimodal
instructions into generative models, thus facilitating a cooperative and
responsive interaction between user inputs and the generative process. This
approach enables iterative and fine-grained refinement of the generation result
through precise and effective user instructions. With
InteractiveVideo, users are given the flexibility to meticulously
tailor key aspects of a video. They can paint the reference image, edit
semantics, and adjust video motions until their requirements are fully met.
Code, models, and demo are available at
https://github.com/invictus717/InteractiveVideo
Nomic Embed: Training a Reproducible Long Context Text Embedder
Authors: name: Zach Nussbaum, name: John X. Morris, name: Brandon Duderstadt, name: Andriy Mulyar
Published: 2024-02-02 18:23:18

This technical report describes the training of nomic-embed-text-v1, the
first fully reproducible, open-source, open-weights, open-data, 8192 context
length English text embedding model that outperforms both OpenAI Ada-002 and
OpenAI text-embedding-3-small on short and long-context tasks. We release the
training code and model weights under an Apache 2 license. In contrast with
other open-source models, we release a training data loader with 235 million
curated text pairs that allows for the full replication of nomic-embed-text-v1.
You can find code and data to replicate the model at
https://github.com/nomic-ai/contrastors
Boximator: Generating Rich and Controllable Motions for Video Synthesis
Authors: name: Jiawei Wang, name: Yuchen Zhang, name: Jiaxin Zou, name: Yan Zeng, name: Guoqiang Wei, name: Liping Yuan, name: Hang Li
Published: 2024-02-02 16:59:48

Generating rich and controllable motion is a pivotal challenge in video
synthesis. We propose Boximator, a new approach for fine-grained motion
control. Boximator introduces two constraint types: hard box and soft box.
Users select objects in the conditional frame using hard boxes and then use
either type of boxes to roughly or rigorously define the object's position,
shape, or motion path in future frames. Boximator functions as a plug-in for
existing video diffusion models. Its training process preserves the base
model's knowledge by freezing the original weights and training only the
control module. To address training challenges, we introduce a novel
self-tracking technique that greatly simplifies the learning of box-object
correlations. Empirically, Boximator achieves state-of-the-art video quality
(FVD) scores, improving on two base models, and further enhanced after
incorporating box constraints. Its robust motion controllability is validated
by drastic increases in the bounding box alignment metric. Human evaluation
also shows that users favor Boximator generation results over the base model.