Orionnnn/hub-stats huggingface dataset:
Multi-Draft Speculative Sampling: Canonical Architectures and
  Theoretical Limits
Authors: name: Ashish Khisti, name: M. Reza Ebrahimi, name: Hassan Dbouk, name: Arash Behboodi, name: Roland Memisevic, name: Christos Louizos
Published: 2024-10-23 19:28:34

We consider multi-draft speculative sampling, where the proposal sequences
are sampled independently from different draft models. At each step, a
token-level draft selection scheme takes a list of valid tokens as input and
produces an output token whose distribution matches that of the target model.
Previous works have demonstrated that the optimal scheme (which maximizes the
probability of accepting one of the input tokens) can be cast as a solution to
a linear program. In this work we show that the optimal scheme can be
decomposed into a two-step solution: in the first step an importance sampling
(IS) type scheme is used to select one intermediate token; in the second step
(single-draft) speculative sampling is applied to generate the output token.
For the case of two identical draft models we further 1) establish a necessary
and sufficient condition on the distributions of the target and draft models
for the acceptance probability to equal one and 2) provide an explicit
expression for the optimal acceptance probability. Our theoretical analysis
also motives a new class of token-level selection scheme based on weighted
importance sampling. Our experimental results demonstrate consistent
improvements in the achievable block efficiency and token rates over baseline
schemes in a number of scenarios.
Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis
  from Scratch
Authors: name: Yuyang Ding, name: Xinyu Shi, name: Xiaobo Liang, name: Juntao Li, name: Qiaoming Zhu, name: Min Zhang
Published: 2024-10-24 12:42:04

The availability of high-quality data is one of the most important factors in
improving the reasoning capability of LLMs. Existing works have demonstrated
the effectiveness of creating more instruction data from seed questions or
knowledge bases. Recent research indicates that continually scaling up data
synthesis from strong models (e.g., GPT-4) can further elicit reasoning
performance. Though promising, the open-sourced community still lacks
high-quality data at scale and scalable data synthesis methods with affordable
costs. To address this, we introduce ScaleQuest, a scalable and novel data
synthesis method that utilizes "small-size" (e.g., 7B) open-source models to
generate questions from scratch without the need for seed data with complex
augmentation constraints. With the efficient ScaleQuest, we automatically
constructed a mathematical reasoning dataset consisting of 1 million
problem-solution pairs, which are more effective than existing open-sourced
datasets. It can universally increase the performance of mainstream open-source
models (i.e., Mistral, Llama3, DeepSeekMath, and Qwen2-Math) by achieving 29.2%
to 46.4% gains on MATH. Notably, simply fine-tuning the Qwen2-Math-7B-Base
model with our dataset can even surpass Qwen2-Math-7B-Instruct, a strong and
well-aligned model on closed-source data, and proprietary models such as
GPT-4-Turbo and Claude-3.5 Sonnet.
LOGO -- Long cOntext aliGnment via efficient preference Optimization
Authors: name: Zecheng Tang, name: Zechen Sun, name: Juntao Li, name: Qiaoming Zhu, name: Min Zhang
Published: 2024-10-24 08:27:26

Long-context models(LCMs) have shown great potential in processing long input
sequences(even more than 100M tokens) conveniently and effectively. With
significant progress, recent research has pointed out that LCMs can accurately
locate token-level salient information within the context. Yet, the generation
performance of these LCMs is far from satisfactory and might result in
misaligned responses, such as hallucinations. To enhance the generation
capability of LCMs, existing works have investigated the effects of data size
and quality for both pre-training and instruction tuning. Though achieving
meaningful improvement, previous methods fall short in either effectiveness or
efficiency. In this paper, we introduce LOGO(Long cOntext aliGnment via
efficient preference Optimization), a training strategy that first introduces
preference optimization for long-context alignment. To overcome the GPU
memory-bound issue caused by the long sequence, LOGO employs a reference-free
preference optimization strategy and adopts a position synthesis method to
construct the training data. By training with only 0.3B data on a single
8timesA800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K
model to achieve comparable performance with GPT-4 in real-world long-context
tasks while preserving the model's original capabilities on other tasks, e.g.,
language modeling and MMLU. Moreover, LOGO can extend the model's context
window size while enhancing its generation performance.
Breaking the Memory Barrier: Near Infinite Batch Size Scaling for
  Contrastive Loss
Authors: name: Zesen Cheng, name: Hang Zhang, name: Kehan Li, name: Sicong Leng, name: Zhiqiang Hu, name: Fei Wu, name: Deli Zhao, name: Xin Li, name: Lidong Bing
Published: 2024-10-22 17:59:30

Contrastive loss is a powerful approach for representation learning, where
larger batch sizes enhance performance by providing more negative samples to
better distinguish between similar and dissimilar data. However, scaling batch
sizes is constrained by the quadratic growth in GPU memory consumption,
primarily due to the full instantiation of the similarity matrix. To address
this, we propose a tile-based computation strategy that partitions the
contrastive loss calculation into arbitrary small blocks, avoiding full
materialization of the similarity matrix. Furthermore, we introduce a
multi-level tiling strategy to leverage the hierarchical structure of
distributed systems, employing ring-based communication at the GPU level to
optimize synchronization and fused kernels at the CUDA core level to reduce I/O
overhead. Experimental results show that the proposed method scales batch sizes
to unprecedented levels. For instance, it enables contrastive training of a
CLIP-ViT-L/14 model with a batch size of 4M or 12M using 8 or 32 A800 80GB
without sacrificing any accuracy. Compared to SOTA memory-efficient solutions,
it achieves a two-order-of-magnitude reduction in memory while maintaining
comparable speed. The code will be made publicly available.
Why Does the Effective Context Length of LLMs Fall Short?
Authors: name: Chenxin An, name: Jun Zhang, name: Ming Zhong, name: Lei Li, name: Shansan Gong, name: Yao Luo, name: Jingjing Xu, name: Lingpeng Kong
Published: 2024-10-24 13:51:50

Advancements in distributed training and efficient attention mechanisms have
significantly expanded the context window sizes of large language models
(LLMs). However, recent work reveals that the effective context lengths of
open-source LLMs often fall short, typically not exceeding half of their
training lengths. In this work, we attribute this limitation to the left-skewed
frequency distribution of relative positions formed in LLMs pretraining and
post-training stages, which impedes their ability to effectively gather distant
information. To address this challenge, we introduce ShifTed Rotray position
embeddING (STRING). STRING shifts well-trained positions to overwrite the
original ineffective positions during inference, enhancing performance within
their existing training lengths. Experimental results show that without
additional training, STRING dramatically improves the performance of the latest
large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on
popular long-context benchmarks RULER and InfiniteBench, establishing new
state-of-the-art results for open-source LLMs. Compared to commercial models,
Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and
clearly surpasses Claude 2 and Kimi-chat.
Framer: Interactive Frame Interpolation
Authors: name: Wen Wang, name: Qiuyu Wang, name: Kecheng Zheng, name: Hao Ouyang, name: Zhekai Chen, name: Biao Gong, name: Hao Chen, name: Yujun Shen, name: Chunhua Shen
Published: 2024-10-24 17:59:51

We propose Framer for interactive frame interpolation, which targets
producing smoothly transitioning frames between two images as per user
creativity. Concretely, besides taking the start and end frames as inputs, our
approach supports customizing the transition process by tailoring the
trajectory of some selected keypoints. Such a design enjoys two clear benefits.
First, incorporating human interaction mitigates the issue arising from
numerous possibilities of transforming one image to another, and in turn
enables finer control of local motions. Second, as the most basic form of
interaction, keypoints help establish the correspondence across frames,
enhancing the model to handle challenging cases (e.g., objects on the start and
end frames are of different shapes and styles). It is noteworthy that our
system also offers an "autopilot" mode, where we introduce a module to estimate
the keypoints and refine the trajectory automatically, to simplify the usage in
practice. Extensive experimental results demonstrate the appealing performance
of Framer on various applications, such as image morphing, time-lapse video
generation, cartoon interpolation, etc. The code, the model, and the interface
will be released to facilitate further research.
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
Authors: name: Chris Yuhao Liu, name: Liang Zeng, name: Jiacai Liu, name: Rui Yan, name: Jujie He, name: Chaojie Wang, name: Shuicheng Yan, name: Yang Liu, name: Yahui Zhou
Published: 2024-10-24 06:06:26

In this report, we introduce a collection of methods to enhance reward
modeling for LLMs, focusing specifically on data-centric techniques. We propose
effective data selection and filtering strategies for curating high-quality
open-source preference datasets, culminating in the Skywork-Reward data
collection, which contains only 80K preference pairs -- significantly smaller
than existing datasets. Using this curated dataset, we developed the
Skywork-Reward model series -- Skywork-Reward-Gemma-27B and
Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top
position on the RewardBench leaderboard. Notably, our techniques and datasets
have directly enhanced the performance of many top-ranked models on
RewardBench, highlighting the practical impact of our contributions in
real-world preference learning applications.
CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for
  pre-training large language models
Authors: name: Liangdong Wang, name: Bo-Wen Zhang, name: Chengwei Wu, name: Hanyu Zhao, name: Xiaofeng Shi, name: Shuhao Gu, name: Jijie Li, name: Quanyue Ma, name: TengFei Pan, name: Guang Liu
Published: 2024-10-24 07:50:07

We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a
high-quality 500GB subset of the Chinese Corpora Internet 3.0
(CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a
novel two-stage hybrid filtering pipeline that significantly enhances data
quality. To evaluate its effectiveness, we trained a 0.5B parameter model from
scratch on 100B tokens across various datasets, achieving superior performance
on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and
WanjuanV1. The high-quality filtering process effectively distills the
capabilities of the Qwen2-72B-instruct model into a compact 0.5B model,
attaining optimal F1 scores for Chinese web data classification. We believe
this open-access dataset will facilitate broader access to high-quality
language models.
Should We Really Edit Language Models? On the Evaluation of Edited
  Language Models
Authors: name: Qi Li, name: Xiang Liu, name: Zhenheng Tang, name: Peijie Dong, name: Zeyu Li, name: Xinglin Pan, name: Xiaowen Chu
Published: 2024-10-24 14:36:48

Model editing has become an increasingly popular alternative for efficiently
updating knowledge within language models. Current methods mainly focus on
reliability, generalization, and locality, with many methods excelling across
these criteria. Some recent works disclose the pitfalls of these editing
methods such as knowledge distortion or conflict. However, the general
abilities of post-edited language models remain unexplored. In this paper, we
perform a comprehensive evaluation on various editing methods and different
language models, and have following findings. (1) Existing editing methods lead
to inevitable performance deterioration on general benchmarks, indicating that
existing editing methods maintain the general abilities of the model within
only a few dozen edits. When the number of edits is slightly large, the
intrinsic knowledge structure of the model is disrupted or even completely
damaged. (2) Instruction-tuned models are more robust to editing, showing less
performance drop on general knowledge after editing. (3) Language model with
large scale is more resistant to editing compared to small model. (4) The
safety of the edited model, is significantly weakened, even for those
safety-aligned models. Our findings indicate that current editing methods are
only suitable for small-scale knowledge updates within language models, which
motivates further research on more practical and reliable editing methods. The
details of code and reproduction can be found in
https://github.com/lqinfdim/EditingEvaluation.
Stable Consistency Tuning: Understanding and Improving Consistency
  Models
Authors: name: Fu-Yun Wang, name: Zhengyang Geng, name: Hongsheng Li
Published: 2024-10-24 17:55:52

Diffusion models achieve superior generation quality but suffer from slow
generation speed due to the iterative nature of denoising. In contrast,
consistency models, a new generative family, achieve competitive performance
with significantly faster sampling. These models are trained either through
consistency distillation, which leverages pretrained diffusion models, or
consistency training/tuning directly from raw data. In this work, we propose a
novel framework for understanding consistency models by modeling the denoising
process of the diffusion model as a Markov Decision Process (MDP) and framing
consistency model training as the value estimation through Temporal
Difference~(TD) Learning. More importantly, this framework allows us to analyze
the limitations of current consistency training/tuning strategies. Built upon
Easy Consistency Tuning (ECT), we propose Stable Consistency Tuning (SCT),
which incorporates variance-reduced learning using the score identity. SCT
leads to significant performance improvements on benchmarks such as CIFAR-10
and ImageNet-64. On ImageNet-64, SCT achieves 1-step FID 2.42 and 2-step FID
1.55, a new SoTA for consistency models.
Unbounded: A Generative Infinite Game of Character Life Simulation
Authors: name: Jialu Li, name: Yuanzhen Li, name: Neal Wadhwa, name: Yael Pritch, name: David E. Jacobs, name: Michael Rubinstein, name: Mohit Bansal, name: Nataniel Ruiz
Published: 2024-10-24 17:59:31

We introduce the concept of a generative infinite game, a video game that
transcends the traditional boundaries of finite, hard-coded systems by using
generative models. Inspired by James P. Carse's distinction between finite and
infinite games, we leverage recent advances in generative AI to create
Unbounded: a game of character life simulation that is fully encapsulated in
generative models. Specifically, Unbounded draws inspiration from sandbox life
simulations and allows you to interact with your autonomous virtual character
in a virtual world by feeding, playing with and guiding it - with open-ended
mechanics generated by an LLM, some of which can be emergent. In order to
develop Unbounded, we propose technical innovations in both the LLM and visual
generation domains. Specifically, we present: (1) a specialized, distilled
large language model (LLM) that dynamically generates game mechanics,
narratives, and character interactions in real-time, and (2) a new dynamic
regional image prompt Adapter (IP-Adapter) for vision models that ensures
consistent yet flexible visual generation of a character across multiple
environments. We evaluate our system through both qualitative and quantitative
analysis, showing significant improvements in character life simulation, user
instruction following, narrative coherence, and visual consistency for both
characters and the environments compared to traditional related approaches.
Steering Your Generalists: Improving Robotic Foundation Models via Value
  Guidance
Authors: name: Mitsuhiko Nakamoto, name: Oier Mees, name: Aviral Kumar, name: Sergey Levine
Published: 2024-10-17 17:46:26

Large, general-purpose robotic policies trained on diverse demonstration
datasets have been shown to be remarkably effective both for controlling a
variety of robots in a range of different scenes, and for acquiring broad
repertoires of manipulation skills. However, the data that such policies are
trained on is generally of mixed quality -- not only are human-collected
demonstrations unlikely to perform the task perfectly, but the larger the
dataset is, the harder it is to curate only the highest quality examples. It
also remains unclear how optimal data from one embodiment is for training on
another embodiment. In this paper, we present a general and broadly applicable
approach that enhances the performance of such generalist robot policies at
deployment time by re-ranking their actions according to a value function
learned via offline RL. This approach, which we call Value-Guided Policy
Steering (V-GPS), is compatible with a wide range of different generalist
policies, without needing to fine-tune or even access the weights of the
policy. We show that the same value function can improve the performance of
five different state-of-the-art policies with different architectures, even
though they were trained on distinct datasets, attaining consistent performance
improvement on multiple robotic platforms across a total of 12 tasks. Code and
videos can be found at: https://nakamotoo.github.io/V-GPS
LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
Authors: name: Haian Jin, name: Hanwen Jiang, name: Hao Tan, name: Kai Zhang, name: Sai Bi, name: Tianyuan Zhang, name: Fujun Luan, name: Noah Snavely, name: Zexiang Xu
Published: 2024-10-22 17:58:28

We propose the Large View Synthesis Model (LVSM), a novel transformer-based
approach for scalable and generalizable novel view synthesis from sparse-view
inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which
encodes input image tokens into a fixed number of 1D latent tokens, functioning
as a fully learned scene representation, and decodes novel-view images from
them; and (2) a decoder-only LVSM, which directly maps input images to
novel-view outputs, completely eliminating intermediate scene representations.
Both models bypass the 3D inductive biases used in previous methods -- from 3D
representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar
projections, plane sweeps) -- addressing novel view synthesis with a fully
data-driven approach. While the encoder-decoder model offers faster inference
due to its independent latent representation, the decoder-only LVSM achieves
superior quality, scalability, and zero-shot generalization, outperforming
previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive
evaluations across multiple datasets demonstrate that both LVSM variants
achieve state-of-the-art novel view synthesis quality. Notably, our models
surpass all previous methods even with reduced computational resources (1-2
GPUs). Please see our website for more details:
https://haian-jin.github.io/projects/LVSM/ .
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
  Understanding
Authors: name: Xiaoqian Shen, name: Yunyang Xiong, name: Changsheng Zhao, name: Lemeng Wu, name: Jun Chen, name: Chenchen Zhu, name: Zechun Liu, name: Fanyi Xiao, name: Balakrishnan Varadarajan, name: Florian Bordes, name: Zhuang Liu, name: Hu Xu, name: Hyunwoo J. Kim, name: Bilge Soran, name: Raghuraman Krishnamoorthi, name: Mohamed Elhoseiny, name: Vikas Chandra
Published: 2024-10-22 21:21:37

Multimodal Large Language Models (MLLMs) have shown promising progress in
understanding and analyzing video content. However, processing long videos
remains a significant challenge constrained by LLM's context size. To address
this limitation, we propose LongVU, a spatiotemporal adaptive compression
mechanism thats reduces the number of video tokens while preserving visual
details of long videos. Our idea is based on leveraging cross-modal query and
inter-frame dependencies to adaptively reduce temporal and spatial redundancy
in videos. Specifically, we leverage DINOv2 features to remove redundant frames
that exhibit high similarity. Then we utilize text-guided cross-modal query for
selective frame feature reduction. Further, we perform spatial token reduction
across frames based on their temporal dependencies. Our adaptive compression
strategy effectively processes a large number of frames with little visual
information loss within given context length. Our LongVU consistently surpass
existing methods across a variety of video understanding benchmarks, especially
on hour-long video understanding tasks such as VideoMME and MLVU. Given a
light-weight LLM, our LongVU also scales effectively into a smaller size with
state-of-the-art video understanding performance.
DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
Authors: name: Hengwei Bian, name: Lingdong Kong, name: Haozhe Xie, name: Liang Pan, name: Yu Qiao, name: Ziwei Liu
Published: 2024-10-23 17:59:58

LiDAR scene generation has been developing rapidly recently. However,
existing methods primarily focus on generating static and single-frame scenes,
overlooking the inherently dynamic nature of real-world driving environments.
In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework
capable of generating large-scale, high-quality LiDAR scenes that capture the
temporal evolution of dynamic environments. DynamicCity mainly consists of two
key models. 1) A VAE model for learning HexPlane as the compact 4D
representation. Instead of using naive averaging operations, DynamicCity
employs a novel Projection Module to effectively compress 4D LiDAR features
into six 2D feature maps for HexPlane construction, which significantly
enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we
utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in
parallel, which improves both network training efficiency and reconstruction
accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x
training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model
for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded
Rollout Operation is proposed to reorganize all six feature planes of the
HexPlane as a squared 2D feature map. In particular, various conditions could
be introduced in the diffusion or sampling process, supporting versatile 4D
generation applications, such as trajectory- and command-driven generation,
inpainting, and layout-conditioned generation. Extensive experiments on the
CarlaSC and Waymo datasets demonstrate that DynamicCity significantly
outperforms existing state-of-the-art 4D LiDAR generation methods across
multiple metrics. The code will be released to facilitate future research.