Orionnnn/hub-stats huggingface dataset:
DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning
  Trajectories Search
Authors: name: Murong Yue, name: Wenlin Yao, name: Haitao Mi, name: Dian Yu, name: Ziyu Yao, name: Dong Yu
Published: 2024-10-04 18:58:09

Enhancing the capability of large language models (LLMs) in reasoning has
gained significant attention in recent years. Previous studies have
demonstrated the effectiveness of various prompting strategies in aiding LLMs
in reasoning (called "reasoning actions"), such as step-by-step thinking,
reflecting before answering, solving with programs, and their combinations.
However, these approaches often applied static, predefined reasoning actions
uniformly to all questions, without considering the specific characteristics of
each question or the capability of the task-solving LLM. In this paper, we
propose DOTS, an approach enabling LLMs to reason dynamically via optimal
reasoning trajectory search, tailored to the specific characteristics of each
question and the inherent capability of the task-solving LLM. Our approach
involves three key steps: i) defining atomic reasoning action modules that can
be composed into various reasoning action trajectories; ii) searching for the
optimal action trajectory for each training question through iterative
exploration and evaluation for the specific task-solving LLM; and iii) using
the collected optimal trajectories to train an LLM to plan for the reasoning
trajectories of unseen questions. In particular, we propose two learning
paradigms, i.e., fine-tuning an external LLM as a planner to guide the
task-solving LLM, or directly fine-tuning the task-solving LLM with an
internalized capability for reasoning actions planning. Our experiments across
eight reasoning tasks show that our method consistently outperforms static
reasoning techniques and the vanilla instruction tuning approach. Further
analysis reveals that our method enables LLMs to adjust their computation based
on problem complexity, allocating deeper thinking and reasoning to harder
problems.
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent
  Sparse Attention
Authors: name: Lijie Yang, name: Zhihao Zhang, name: Zhuofu Chen, name: Zikun Li, name: Zhihao Jia
Published: 2024-10-07 14:30:27

Large language models (LLMs) have driven significant advancements across
diverse NLP tasks, with long-context models gaining prominence for handling
extended inputs. However, the expanding key-value (KV) cache size required by
Transformer architectures intensifies the memory constraints, particularly
during the decoding phase, creating a significant bottleneck. Existing sparse
attention mechanisms designed to address this bottleneck have two limitations:
(1) they often fail to reliably identify the most relevant tokens for
attention, and (2) they overlook the spatial coherence of token selection
across consecutive Transformer layers, which can lead to performance
degradation and substantial overhead in token selection. This paper introduces
TidalDecode, a simple yet effective algorithm and system for fast and accurate
LLM decoding through position persistent sparse attention. TidalDecode
leverages the spatial coherence of tokens selected by existing sparse attention
methods and introduces a few token selection layers that perform full attention
to identify the tokens with the highest attention scores, while all other
layers perform sparse attention with the pre-selected tokens. This design
enables TidalDecode to substantially reduce the overhead of token selection for
sparse attention without sacrificing the quality of the generated results.
Evaluation on a diverse set of LLMs and tasks shows that TidalDecode closely
matches the generative performance of full attention methods while reducing the
LLM decoding latency by up to 2.1x.
EBES: Easy Benchmarking for Event Sequences
Authors: name: Dmitry Osin, name: Igor Udovichenko, name: Viktor Moskvoretskii, name: Egor Shvetsov, name: Evgeny Burnaev
Published: 2024-10-04 13:03:43

Event sequences, characterized by irregular sampling intervals and a mix of
categorical and numerical features, are common data structures in various
real-world domains such as healthcare, finance, and user interaction logs.
Despite advances in temporal data modeling techniques, there is no standardized
benchmarks for evaluating their performance on event sequences. This
complicates result comparison across different papers due to varying evaluation
protocols, potentially misleading progress in this field. We introduce EBES, a
comprehensive benchmarking tool with standardized evaluation scenarios and
protocols, focusing on regression and classification problems with
sequence-level targets. Our library simplifies benchmarking, dataset addition,
and method integration through a unified interface. It includes a novel
synthetic dataset and provides preprocessed real-world datasets, including the
largest publicly available banking dataset. Our results provide an in-depth
analysis of datasets, identifying some as unsuitable for model comparison. We
investigate the importance of modeling temporal and sequential components, as
well as the robustness and scaling properties of the models. These findings
highlight potential directions for future research. Our benchmark aim is to
facilitate reproducible research, expediting progress and increasing real-world
impacts.
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video
  Large Language Models
Authors: name: Haibo Wang, name: Zhiyang Xu, name: Yu Cheng, name: Shizhe Diao, name: Yufan Zhou, name: Yixin Cao, name: Qifan Wang, name: Weifeng Ge, name: Lifu Huang
Published: 2024-10-04 10:04:37

Video Large Language Models (Video-LLMs) have demonstrated remarkable
capabilities in coarse-grained video understanding, however, they struggle with
fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM,
a novel Video-LLM adept at perceiving and reasoning over specific video moments
in a fine-grained manner. We identify that current Video-LLMs have limitations
for fine-grained video understanding since they lack effective temporal
modeling and timestamp representation. In light of this, we sharpen our model
by incorporating (1) an additional temporal stream to encode the relationships
between frames and (2) discrete temporal tokens enriched with specific time
knowledge to represent timestamps. To optimize the training of
Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with
simple video-captioning tasks and progressively introducing video temporal
grounding tasks of increasing complexity. To further enhance
Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded
VideoQA dataset by an automatic annotation pipeline. Extensive experiments
demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding
tasks such as temporal sentence grounding, dense video captioning, and grounded
VideoQA, but also shows great potential as a versatile video assistant for
general video understanding.
A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive
  Transformer for Efficient Finegrained Image Generation
Authors: name: Liang Chen, name: Sinan Tan, name: Zefan Cai, name: Weichu Xie, name: Haozhe Zhao, name: Yichi Zhang, name: Junyang Lin, name: Jinze Bai, name: Tianyu Liu, name: Baobao Chang
Published: 2024-10-02 18:10:05

This work tackles the information loss bottleneck of vector-quantization (VQ)
autoregressive image generation by introducing a novel model architecture
called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer
predicts more codes for an image by introducing a new autoregression direction,
model depth, along with the sequence length direction. Compared to
traditional 1D autoregression and previous work utilizing similar 2D image
decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end
model that can generate higher quality images with the same backbone model size
and sequence length, opening a new optimization perspective for autoregressive
image generation. Furthermore, our experiments reveal that the
DnD-Transformer's potential extends beyond generating natural images. It can
even generate images with rich text and graphical elements in a self-supervised
manner, demonstrating an understanding of these combined modalities. This has
not been previously demonstrated for popular vision generative models such as
diffusion models, showing a spark of vision-language intelligence when trained
solely on images. Code, datasets and models are open at
https://github.com/chenllliang/DnD-Transformer.
ControlAR: Controllable Image Generation with Autoregressive Models
Authors: name: Zongming Li, name: Tianheng Cheng, name: Shoufa Chen, name: Peize Sun, name: Haocheng Shen, name: Longjin Ran, name: Xiaoxin Chen, name: Wenyu Liu, name: Xinggang Wang
Published: 2024-10-03 17:28:07

Autoregressive (AR) models have reformulated image generation as next-token
prediction, demonstrating remarkable potential and emerging as strong
competitors to diffusion models. However, control-to-image generation, akin to
ControlNet, remains largely unexplored within AR models. Although a natural
approach, inspired by advancements in Large Language Models, is to tokenize
control images into tokens and prefill them into the autoregressive model
before decoding image tokens, it still falls short in generation quality
compared to ControlNet and suffers from inefficiency. To this end, we introduce
ControlAR, an efficient and effective framework for integrating spatial
controls into autoregressive image generation models. Firstly, we explore
control encoding for AR models and propose a lightweight control encoder to
transform spatial inputs (e.g., canny edges or depth maps) into control tokens.
Then ControlAR exploits the conditional decoding method to generate the next
image token conditioned on the per-token fusion between control and image
tokens, similar to positional encodings. Compared to prefilling tokens, using
conditional decoding significantly strengthens the control capability of AR
models but also maintains the model's efficiency. Furthermore, the proposed
ControlAR surprisingly empowers AR models with arbitrary-resolution image
generation via conditional decoding and specific controls. Extensive
experiments can demonstrate the controllability of the proposed ControlAR for
the autoregressive control-to-image generation across diverse inputs, including
edges, depths, and segmentation masks. Furthermore, both quantitative and
qualitative results indicate that ControlAR surpasses previous state-of-the-art
controllable diffusion models, e.g., ControlNet++. Code, models, and demo will
soon be available at https://github.com/hustvl/ControlAR.
Hyper-multi-step: The Truth Behind Difficult Long-context Tasks
Authors: name: Yijiong Yu
Published: 2024-10-06 09:29:19

Long-context language models (LCLM), characterized by their extensive context
window, is becoming increasingly popular. Meanwhile, many long-context
benchmarks present challenging tasks that even the most advanced LCLMs struggle
to complete. However, the underlying sources of various challenging
long-context tasks have seldom been studied. To bridge this gap, we conduct
experiments to indicate their difficulty stems primarily from two basic issues:
"multi-matching retrieval," which requires the simultaneous retrieval of
multiple items, and "logic-based retrieval," which necessitates logical
judgment within retrieval criteria. These two problems, while seemingly
straightforward, actually exceed the capabilities of LCLMs because they are
proven to be hyper-multi-step (demanding numerous steps to solve) in nature.
This finding could explain why LLMs struggle with more advanced long-context
tasks, providing a more accurate perspective for rethinking solutions for them.
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Authors: name: Yekun Chai, name: Haoran Sun, name: Huang Fang, name: Shuohuan Wang, name: Yu Sun, name: Hua Wu
Published: 2024-10-03 17:55:13

Reinforcement learning from human feedback (RLHF) has demonstrated
effectiveness in aligning large language models (LLMs) with human preferences.
However, token-level RLHF suffers from the credit assignment problem over long
sequences, where delayed rewards make it challenging for the model to discern
which actions contributed to successful outcomes. This hinders learning
efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple
yet effective RLHF framework that incorporates macro actions -- sequences of
tokens or higher-level language constructs -- into the learning process. By
operating at this higher level of abstraction, our approach reduces the
temporal distance between actions and rewards, facilitating faster and more
accurate credit assignment. This results in more stable policy gradient
estimates and enhances learning efficiency within each episode, all without
increasing computational complexity during training or inference. We validate
our approach through extensive experiments across various model sizes and
tasks, including text summarization, dialogue generation, question answering,
and program synthesis. Our method achieves substantial performance improvements
over standard RLHF, with performance gains of up to 30% in text summarization
and code generation, 18% in dialogue, and 8% in question answering tasks.
Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in
terms of training time and continues to outperform it with further training. We
will make our code and data publicly available at
https://github.com/ernie-research/MA-RLHF .
LongGenBench: Long-context Generation Benchmark
Authors: name: Xiang Liu, name: Peijie Dong, name: Xuming Hu, name: Xiaowen Chu
Published: 2024-10-05 15:33:25

Current long-context benchmarks primarily focus on retrieval-based tests,
requiring Large Language Models (LLMs) to locate specific information within
extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
Long-context generation refers to the ability of a language model to generate
coherent and contextually accurate text that spans across lengthy passages or
documents. While recent studies show strong performance on NIAH and other
retrieval-based long-context benchmarks, there is a significant lack of
benchmarks for evaluating long-context generation capabilities. To bridge this
gap and offer a comprehensive assessment, we introduce a synthetic benchmark,
LongGenBench, which allows for flexible configurations of customized generation
context lengths. LongGenBench advances beyond traditional benchmarks by
redesigning the format of questions and necessitating that LLMs respond with a
single, cohesive long-context answer. Upon extensive evaluation using
LongGenBench, we observe that: (1) both API accessed and open source models
exhibit performance degradation in long-context generation scenarios, ranging
from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of
performance degradation, with the Gemini-1.5-Flash model showing the least
degradation among API accessed models, and the Qwen2 series exhibiting the
least degradation in LongGenBench among open source models.
Only-IF:Revealing the Decisive Effect of Instruction
  Diversity on Generalization
Authors: name: Dylan Zhang, name: Justin Wang, name: Francois Charton
Published: 2024-10-07 03:15:11

Understanding and accurately following instructions is critical for large
language models (LLMs) to be effective across diverse tasks. In this work, we
rigorously examine the key factors that enable models to generalize to unseen
instructions, providing insights to guide the collection of data for
instruction-tuning. Through controlled experiments, inspired by the
Turing-complete Markov algorithm, we demonstrate that such generalization
only emerges when training data is diversified enough across
semantic domains. Our findings also reveal that merely diversifying within
limited domains fails to ensure robust generalization. In contrast,
cross-domain data diversification, even under constrained data budgets,
significantly enhances a model's adaptability. We further extend our analysis
to real-world scenarios, including fine-tuning of
$textbf{specialist} and textbf{generalist}$ models.
In both cases, we demonstrate that 1) better performance can be achieved by
increasing the diversity of an established dataset while keeping the data size
constant, and 2) when scaling up the data, diversifying the semantics of
instructions is more effective than simply increasing the quantity of similar
data. Our research provides important insights for dataset collation,
particularly when optimizing model performance by expanding training data for
both specialist and generalist scenarios. We show that careful consideration of
data diversification is key: training specialist models with data extending
beyond their core domain leads to significant performance improvements, while
generalist models benefit from diverse data mixtures that enhance their overall
instruction-following capabilities across a wide range of applications. Our
results highlight the critical role of strategic diversification and offer
clear guidelines for improving data quality.
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
Authors: name: Qiyuan Zhang, name: Yufei Wang, name: Tiezheng YU, name: Yuxin Jiang, name: Chuhan Wu, name: Liangyou Li, name: Yasheng Wang, name: Xin Jiang, name: Lifeng Shang, name: Ruiming Tang, name: Fuyuan Lyu, name: Chen Ma
Published: 2024-10-07 16:50:47

With significant efforts in recent studies, LLM-as-a-Judge has become a
cost-effective alternative to human evaluation for assessing the text
generation quality in a wide range of tasks. However, there still remains a
reliability gap between LLM-as-a-Judge and human evaluation. One important
reason is the lack of guided oracles in the evaluation process. Motivated by
the role of reference pervasively used in classic text evaluation, we introduce
RevisEval, a novel text generation evaluation paradigm via the response-adapted
references. RevisEval is driven by the key observation that an ideal reference
should maintain the necessary relevance to the response to be evaluated.
Specifically, RevisEval leverages the text revision capabilities of large
language models (LLMs) to adaptively revise the response, then treat the
revised text as the reference (response-adapted reference) for the subsequent
evaluation. Extensive experiments demonstrate that RevisEval outperforms
traditional reference-free and reference-based evaluation paradigms that use
LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.
More importantly, our response-adapted references can further boost the
classical text metrics, e.g., BLEU and BERTScore, compared to traditional
references and even rival the LLM-as-a-Judge. A detailed analysis is also
conducted to confirm RevisEval's effectiveness in bias reduction, the impact of
inference cost, and reference relevance.
Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep
  Approach
Authors: name: Yaofang Liu, name: Yumeng Ren, name: Xiaodong Cun, name: Aitor Artola, name: Yang Liu, name: Tieyong Zeng, name: Raymond H. Chan, name: Jean-michel Morel
Published: 2024-10-04 05:47:39

Diffusion models have revolutionized image generation, and their extension to
video generation has shown promise. However, current video diffusion
models~(VDMs) rely on a scalar timestep variable applied at the clip level,
which limits their ability to model complex temporal dependencies needed for
various tasks like image-to-video generation. To address this limitation, we
propose a frame-aware video diffusion model~(FVDM), which introduces a novel
vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach
allows each frame to follow an independent noise schedule, enhancing the
model's capacity to capture fine-grained temporal dependencies. FVDM's
flexibility is demonstrated across multiple tasks, including standard video
generation, image-to-video generation, video interpolation, and long video
synthesis. Through a diverse set of VTV configurations, we achieve superior
quality in generated videos, overcoming challenges such as catastrophic
forgetting during fine-tuning and limited generalizability in zero-shot
methods.Our empirical evaluations show that FVDM outperforms state-of-the-art
methods in video generation quality, while also excelling in extended tasks. By
addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm
in video synthesis, offering a robust framework with significant implications
for generative modeling and multimedia applications.
SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving
  Model Transformation
Authors: name: Aurick Qiao, name: Zhewei Yao, name: Samyam Rajbhandari, name: Yuxiong He
Published: 2024-10-04 22:45:26

LLM inference for popular enterprise use cases, such as summarization, RAG,
and code-generation, typically observes orders of magnitude longer prompt
lengths than generation lengths. This characteristic leads to high cost of
prefill and increased response latency. In this paper, we present SwiftKV, a
novel model transformation and distillation procedure specifically designed to
reduce the time and cost of processing prompt tokens while preserving high
quality of generated tokens. SwiftKV combines three key mechanisms: i)
SingleInputKV, which prefills later layers' KV cache using a much earlier
layer's output, allowing prompt tokens to skip much of the model computation,
ii) AcrossKV, which merges the KV caches of neighboring layers to reduce the
memory footprint and support larger batch size for higher throughput, and iii)
a knowledge-preserving distillation procedure that can adapt existing LLMs for
SwiftKV with minimal accuracy impact and low compute and data requirement. For
Llama-3.1-8B and 70B, SwiftKV reduces the compute requirement of prefill by 50%
and the memory requirement of the KV cache by 62.5% while incurring minimum
quality degradation across a wide range of tasks. In the end-to-end inference
serving using an optimized vLLM implementation, SwiftKV realizes up to 2x
higher aggregate throughput and 60% lower time per output token. It can achieve
a staggering 560 TFlops/GPU of normalized inference throughput, which
translates to 16K tokens/s for Llama-3.1-70B in 16-bit precision on 4x H100
GPUs.