Orionnnn/hub-stats huggingface dataset:
Collaborative Video Diffusion: Consistent Multi-video Generation with
  Camera Control
Authors: name: Zhengfei Kuang, name: Shengqu Cai, name: Hao He, name: Yinghao Xu, name: Hongsheng Li, name: Leonidas Guibas, name: Gordon Wetzstein
Published: 2024-05-27 17:58:01

Research on video generation has recently made tremendous progress, enabling
high-quality videos to be generated from text prompts or images. Adding control
to the video generation process is an important goal moving forward and recent
approaches that condition video generation models on camera trajectories make
strides towards it. Yet, it remains challenging to generate a video of the same
scene from multiple different camera trajectories. Solutions to this
multi-video generation problem could enable large-scale 3D scene generation
with editable camera trajectories, among other applications. We introduce
collaborative video diffusion (CVD) as an important step towards this vision.
The CVD framework includes a novel cross-video synchronization module that
promotes consistency between corresponding frames of the same video rendered
from different camera poses using an epipolar attention mechanism. Trained on
top of a state-of-the-art camera-control module for video generation, CVD
generates multiple videos rendered from different camera trajectories with
significantly better consistency than baselines, as shown in extensive
experiments. Project page: https://collaborativevideodiffusion.github.io/.
Matryoshka Multimodal Models
Authors: name: Mu Cai, name: Jianwei Yang, name: Jianfeng Gao, name: Yong Jae Lee
Published: 2024-05-27 17:59:56

Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in
visual-linguistic reasoning. These models first embed images into a fixed large
number of visual tokens and then feed them into a Large Language Model (LLM).
However, this design causes an excessive number of tokens for dense visual
scenarios such as high-resolution images and videos, leading to great
inefficiency. While token pruning/merging methods do exist, they produce a
single length output for each image and do not afford flexibility in trading
off information density v.s. efficiency. Inspired by the concept of Matryoshka
Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent
visual content as nested sets of visual tokens that capture information across
multiple coarse-to-fine granularities. Our approach offers several unique
benefits for LMMs: (1) One can explicitly control the visual granularity per
test instance during inference, e.g. , adjusting the number of tokens used to
represent an image based on the anticipated complexity or simplicity of the
content; (2) M3 provides a framework for analyzing the granularity needed for
existing datasets, where we find that COCO-style benchmarks only need around ~9
visual tokens to obtain accuracy similar to that of using all 576 tokens; (3)
Our approach provides a foundation to explore the best trade-off between
performance and visual token length at sample level, where our investigation
reveals that a large gap exists between the oracle upper bound and current
fixed-scale representations.
The Road Less Scheduled
Authors: name: Aaron Defazio, name: Xingyu, name: Yang, name: Harsh Mehta, name: Konstantin Mishchenko, name: Ahmed Khaled, name: Ashok Cutkosky
Published: 2024-05-24 16:20:46

Existing learning rate schedules that do not require specification of the
optimization stopping step T are greatly out-performed by learning rate
schedules that depend on T. We propose an approach that avoids the need for
this stopping time by eschewing the use of schedules entirely, while exhibiting
state-of-the-art performance compared to schedules across a wide family of
problems ranging from convex problems to large-scale deep learning problems.
Our Schedule-Free approach introduces no additional hyper-parameters over
standard optimizers with momentum. Our method is a direct consequence of a new
theory we develop that unifies scheduling and iterate averaging. An open source
implementation of our method is available
(https://github.com/facebookresearch/schedule_free).
ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal
  Models
Authors: name: Chunjiang Ge, name: Sijie Cheng, name: Ziming Wang, name: Jiale Yuan, name: Yuan Gao, name: Jun Song, name: Shiji Song, name: Gao Huang, name: Bo Zheng
Published: 2024-05-24 17:34:15

High-resolution Large Multimodal Models (LMMs) encounter the challenges of
excessive visual tokens and quadratic visual complexity. Current
high-resolution LMMs address the quadratic complexity while still generating
excessive visual tokens. However, the redundancy in visual tokens is the key
problem as it leads to more substantial compute. To mitigate this issue, we
propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the
visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses
high-resolution images into information-rich visual features, effectively
preventing the generation of excessive visual tokens. To enhance the
capabilities of ConvLLaVA, we propose two critical optimizations. Since the
low-resolution pretrained ConvNeXt underperforms when directly applied on high
resolution, we update it to bridge the gap. Moreover, since ConvNeXt's original
compression ratio is inadequate for much higher resolution inputs, we train a
successive stage to further compress the visual tokens, thereby reducing
redundancy. These optimizations enable ConvLLaVA to support inputs of 1536x1536
resolution generating only 576 visual tokens, capable of handling images of
arbitrary aspect ratios. Experimental results demonstrate that our method
achieves competitive performance with state-of-the-art models on mainstream
benchmarks. The ConvLLaVA model series are publicly available at
https://github.com/alibaba/conv-llava.
Stacking Your Transformers: A Closer Look at Model Growth for Efficient
  LLM Pre-Training
Authors: name: Wenyu Du, name: Tongxu Luo, name: Zihan Qiu, name: Zeyu Huang, name: Yikang Shen, name: Reynold Cheng, name: Yike Guo, name: Jie Fu
Published: 2024-05-24 08:00:00

LLMs are computationally expensive to pre-train due to their large scale.
Model growth emerges as a promising approach by leveraging smaller models to
accelerate the training of larger ones. However, the viability of these model
growth methods in efficient LLM pre-training remains underexplored. This work
identifies three critical textit{O}bstacles: (O1)
lack of comprehensive evaluation, (O2) untested viability for
scaling, and (O3) lack of empirical guidelines. To tackle
O1, we summarize existing approaches into four atomic growth
operators and systematically evaluate them in a standardized LLM pre-training
setting. Our findings reveal that a depthwise stacking operator, called
G_{stack}, exhibits remarkable acceleration in training, leading to
decreased loss and improved overall performance on eight standard NLP
benchmarks compared to strong baselines. Motivated by these promising results,
we conduct extensive experiments to delve deeper into G_{stack} to
address O2 and O3. For O2 (untested
scalability), our study shows that G_{stack} is scalable and
consistently performs well, with experiments up to 7B LLMs after growth and
pre-training LLMs with 750B tokens. For example, compared to a conventionally
trained 7B model using 300B tokens, our G_{stack} model converges to
the same loss with 194B tokens, resulting in a 54.6\% speedup. We further
address O3 (lack of empirical guidelines) by formalizing guidelines
to determine growth timing and growth factor for G_{stack}, making it
practical in general LLM pre-training. We also provide in-depth discussions and
comprehensive ablation studies of G_{stack}. Our code and pre-trained
model are available at
https://llm-stacking.github.io/{https://llm-stacking.github.io/}.
Automatic Data Curation for Self-Supervised Learning: A Clustering-Based
  Approach
Authors: name: Huy V. Vo, name: Vasil Khalidov, name: Timothée Darcet, name: Théo Moutakanni, name: Nikita Smetanin, name: Marc Szafraniec, name: Hugo Touvron, name: Camille Couprie, name: Maxime Oquab, name: Armand Joulin, name: Hervé Jégou, name: Patrick Labatut, name: Piotr Bojanowski
Published: 2024-05-24 14:58:51

Self-supervised features are the cornerstone of modern machine learning
systems. They are typically pre-trained on data collections whose construction
and curation typically require extensive human effort. This manual process has
some limitations similar to those encountered in supervised learning, e.g., the
crowd-sourced selection of data is costly and time-consuming, preventing
scaling the dataset size. In this work, we consider the problem of automatic
curation of high-quality datasets for self-supervised pre-training. We posit
that such datasets should be large, diverse and balanced, and propose a
clustering-based approach for building ones satisfying all these criteria. Our
method involves successive and hierarchical applications of k-means on a
large and diverse data repository to obtain clusters that distribute uniformly
among data concepts, followed by a hierarchical, balanced sampling step from
these clusters. Extensive experiments on three different data domains including
web-based images, satellite images and text show that features trained on our
automatically curated datasets outperform those trained on uncurated data while
being on par or better than ones trained on manually curated data.
iVideoGPT: Interactive VideoGPTs are Scalable World Models
Authors: name: Jialong Wu, name: Shaofeng Yin, name: Ningya Feng, name: Xu He, name: Dong Li, name: Jianye Hao, name: Mingsheng Long
Published: 2024-05-24 05:29:12

World models empower model-based agents to interactively explore, reason, and
plan within imagined environments for real-world decision-making. However, the
high demand for interactivity poses challenges in harnessing recent
advancements in video generative models for developing world models at scale.
This work introduces Interactive VideoGPT (iVideoGPT), a scalable
autoregressive transformer framework that integrates multimodal signals--visual
observations, actions, and rewards--into a sequence of tokens, facilitating an
interactive experience of agents via next-token prediction. iVideoGPT features
a novel compressive tokenization technique that efficiently discretizes
high-dimensional visual observations. Leveraging its scalable architecture, we
are able to pre-train iVideoGPT on millions of human and robotic manipulation
trajectories, establishing a versatile foundation that is adaptable to serve as
interactive world models for a wide range of downstream tasks. These include
action-conditioned video prediction, visual planning, and model-based
reinforcement learning, where iVideoGPT achieves competitive performance
compared with state-of-the-art methods. Our work advances the development of
interactive general world models, bridging the gap between generative video
models and practical model-based reinforcement learning applications.
HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed
  via Gaussian Splatting
Authors: name: Yuanhao Cai, name: Zihao Xiao, name: Yixun Liang, name: Yulun Zhang, name: Xiaokang Yang, name: Yaoyao Liu, name: Alan Yuille
Published: 2024-05-24 00:46:58

High dynamic range (HDR) novel view synthesis (NVS) aims to create
photorealistic images from novel viewpoints using HDR imaging techniques. The
rendered HDR images capture a wider range of brightness levels containing more
details of the scene than normal low dynamic range (LDR) images. Existing HDR
NVS methods are mainly based on NeRF. They suffer from long training time and
slow inference speed. In this paper, we propose a new framework, High Dynamic
Range Gaussian Splatting (HDR-GS), which can efficiently render novel HDR views
and reconstruct LDR images with a user input exposure time. Specifically, we
design a Dual Dynamic Range (DDR) Gaussian point cloud model that uses
spherical harmonics to fit HDR color and employs an MLP-based tone-mapper to
render LDR color. The HDR and LDR colors are then fed into two Parallel
Differentiable Rasterization (PDR) processes to reconstruct HDR and LDR views.
To establish the data foundation for the research of 3D Gaussian
splatting-based methods in HDR NVS, we recalibrate the camera parameters and
compute the initial positions for Gaussian point clouds. Experiments
demonstrate that our HDR-GS surpasses the state-of-the-art NeRF-based method by
3.84 and 1.91 dB on LDR and HDR NVS while enjoying 1000x inference speed and
only requiring 6.3% training time.
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision
  Models
Authors: name: Byung-Kwan Lee, name: Chae Won Kim, name: Beomchan Park, name: Yong Man Ro
Published: 2024-05-24 14:04:03

The rapid development of large language and vision models (LLVMs) has been
driven by advances in visual instruction tuning. Recently, open-source LLVMs
have curated high-quality visual instruction tuning datasets and utilized
additional vision encoders or multiple computer vision models in order to
narrow the performance gap with powerful closed-source LLVMs. These
advancements are attributed to multifaceted information required for diverse
capabilities, including fundamental image understanding, real-world knowledge
about common-sense and non-object concepts (e.g., charts, diagrams, symbols,
signs, and math problems), and step-by-step procedures for solving complex
questions. Drawing from the multifaceted information, we present a new
efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages
multifaceted rationale to enhance understanding and answering capabilities. To
embed lengthy rationales containing abundant information, we employ the Mamba
architecture, capable of processing sequential data with linear time
complexity. We introduce a new concept of traversal of rationale that
facilitates efficient embedding of rationale. Subsequently, the backbone
multimodal language model (MLM) is trained to generate answers with the aid of
rationale. Through these steps, Meteor achieves significant improvements in
vision language performances across multiple evaluation benchmarks requiring
diverse capabilities, without scaling up the model size or employing additional
vision encoders and computer vision models.
Denoising LM: Pushing the Limits of Error Correction Models for Speech
  Recognition
Authors: name: Zijin Gu, name: Tatiana Likhomanenko, name: He Bai, name: Erik McDermott, name: Ronan Collobert, name: Navdeep Jaitly
Published: 2024-05-24 05:05:12

Language models (LMs) have long been used to improve results of automatic
speech recognition (ASR) systems, but they are unaware of the errors that ASR
systems make. Error correction models are designed to fix ASR errors, however,
they showed little improvement over traditional LMs mainly due to the lack of
supervised training data. In this paper, we present Denoising LM (DLM), which
is a scaled error correction model trained with vast amounts of
synthetic data, significantly exceeding prior attempts meanwhile achieving new
state-of-the-art ASR performance. We use text-to-speech (TTS) systems to
synthesize audio, which is fed into an ASR system to produce noisy hypotheses,
which are then paired with the original texts to train the DLM. DLM has several
key ingredients: (i) up-scaled model and data; (ii) usage of
multi-speaker TTS systems; (iii) combination of multiple noise augmentation
strategies; and (iv) new decoding techniques. With a Transformer-CTC ASR, DLM
achieves 1.5% word error rate (WER) on test-clean and 3.3% WER on
test-other on Librispeech, which to our knowledge are the best
reported numbers in the setting where no external audio data are used and even
match self-supervised methods which use external audio data. Furthermore, a
single DLM is applicable to different ASRs, and greatly surpassing the
performance of conventional LM based beam-search rescoring. These results
indicate that properly investigated error correction models have the potential
to replace conventional LMs, holding the key to a new level of accuracy in ASR
systems.
Aya 23: Open Weight Releases to Further Multilingual Progress
Authors: name: Viraat Aryabumi, name: John Dang, name: Dwarak Talupuru, name: Saurabh Dash, name: David Cairuz, name: Hangyu Lin, name: Bharat Venkitesh, name: Madeline Smith, name: Kelly Marchisio, name: Sebastian Ruder, name: Acyr Locatelli, name: Julia Kreutzer, name: Nick Frosst, name: Phil Blunsom, name: Marzieh Fadaee, name: Ahmet Üstün, name: Sara Hooker
Published: 2024-05-23 20:10:38

This technical report introduces Aya 23, a family of multilingual language
models. Aya 23 builds on the recent release of the Aya model (\"Ust\"un et al.,
2024), focusing on pairing a highly performant pre-trained model with the
recently released Aya collection (Singh et al., 2024). The result is a powerful
multilingual large language model serving 23 languages, expanding state-of-art
language modeling capabilities to approximately half of the world's population.
The Aya model covered 101 languages whereas Aya 23 is an experiment in depth vs
breadth, exploring the impact of allocating more capacity to fewer languages
that are included during pre-training. Aya 23 outperforms both previous
massively multilingual models like Aya 101 for the languages it covers, as well
as widely used models like Gemma, Mistral and Mixtral on an extensive range of
discriminative and generative tasks. We release the open weights for both the
8B and 35B models as part of our continued commitment for expanding access to
multilingual progress.
Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model
  Pretraining
Authors: name: Ce Ge, name: Zhijian Ma, name: Daoyuan Chen, name: Yaliang Li, name: Bolin Ding
Published: 2024-05-23 09:44:02

Large language models exhibit exceptional generalization capabilities,
primarily attributed to the utilization of diversely sourced data. However,
conventional practices in integrating this diverse data heavily rely on
heuristic schemes, lacking theoretical guidance. This research tackles these
limitations by investigating strategies based on low-cost proxies for data
mixtures, with the aim of streamlining data curation to enhance training
efficiency. Specifically, we propose a unified scaling law, termed BiMix, which
accurately models the bivariate scaling behaviors of both data quantity and
mixing proportions. We conduct systematic experiments and provide empirical
evidence for the predictive power and fundamental principles of BiMix. Notably,
our findings reveal that entropy-driven training-free data mixtures can achieve
comparable or even better performance than more resource-intensive methods. We
hope that our quantitative insights can shed light on further judicious
research and development in cost-effective language modeling.
AutoCoder: Enhancing Code Large Language Model with
  AIEV-Instruct
Authors: name: Bin Lei, name: Yuchen Li, name: Qiuwu Chen
Published: 2024-05-23 02:53:25

We introduce AutoCoder, the first Large Language Model to surpass GPT-4 Turbo
(April 2024) and GPT-4o in pass@1 on the Human Eval benchmark test
(90.9% vs. 90.2%). In addition, AutoCoder offers a more
versatile code interpreter compared to GPT-4 Turbo and GPT-4o. It's code
interpreter can install external packages instead of limiting to built-in
packages. AutoCoder's training data is a multi-turn dialogue dataset created by
a system combining agent interaction and external code execution verification,
a method we term \textsc{AIEV-Instruct} (Instruction Tuning with
Agent-Interaction and Execution-Verified). Compared to previous large-scale
code dataset generation methods, AIEV-Instruct reduces dependence on
proprietary large models and provides execution-validated code dataset. The
code and the demo video is available in
https://github.com/bin123apple/AutoCoder.
Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to
  the Edge of Generalization
Authors: name: Boshi Wang, name: Xiang Yue, name: Yu Su, name: Huan Sun
Published: 2024-05-23 21:42:19

We study whether transformers can learn to implicitly reason over parametric
knowledge, a skill that even the most capable language models struggle with.
Focusing on two representative reasoning types, composition and comparison, we
consistently find that transformers can learn implicit reasoning, but only
through grokking, i.e., extended training far beyond overfitting. The levels of
generalization also vary across reasoning types: when faced with
out-of-distribution examples, transformers fail to systematically generalize
for composition but succeed for comparison. We delve into the model's internals
throughout training, conducting analytical experiments that reveal: 1) the
mechanism behind grokking, such as the formation of the generalizing circuit
and its relation to the relative efficiency of generalizing and memorizing
circuits, and 2) the connection between systematicity and the configuration of
the generalizing circuit. Our findings guide data and training setup to better
induce implicit reasoning and suggest potential improvements to the transformer
architecture, such as encouraging cross-layer knowledge sharing. Furthermore,
we demonstrate that for a challenging reasoning task with a large search space,
GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly
regardless of prompting styles or retrieval augmentation, while a fully grokked
transformer can achieve near-perfect accuracy, showcasing the power of
parametric memory for complex reasoning.
CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and
  Interactive Geometry Refiner
Authors: name: Weiyu Li, name: Jiarui Liu, name: Rui Chen, name: Yixun Liang, name: Xuelin Chen, name: Ping Tan, name: Xiaoxiao Long
Published: 2024-05-23 18:30:12

We present a novel generative 3D modeling system, coined CraftsMan, which can
generate high-fidelity 3D geometries with highly varied shapes, regular mesh
topologies, and detailed surfaces, and, notably, allows for refining the
geometry in an interactive manner. Despite the significant advancements in 3D
generation, existing methods still struggle with lengthy optimization
processes, irregular mesh topologies, noisy surfaces, and difficulties in
accommodating user edits, consequently impeding their widespread adoption and
implementation in 3D modeling software. Our work is inspired by the craftsman,
who usually roughs out the holistic figure of the work first and elaborates the
surface details subsequently. Specifically, we employ a 3D native diffusion
model, which operates on latent space learned from latent set-based 3D
representations, to generate coarse geometries with regular mesh topology in
seconds. In particular, this process takes as input a text prompt or a
reference image and leverages a powerful multi-view (MV) diffusion model to
generate multiple views of the coarse geometry, which are fed into our
MV-conditioned 3D diffusion model for generating the 3D geometry, significantly
improving robustness and generalizability. Following that, a normal-based
geometry refiner is used to significantly enhance the surface details. This
refinement can be performed automatically, or interactively with user-supplied
edits. Extensive experiments demonstrate that our method achieves high efficacy
in producing superior-quality 3D assets compared to existing methods. HomePage:
https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan