Orionnnn/hub-stats huggingface dataset:
Exploring the Limit of Outcome Reward for Learning Mathematical
  Reasoning
Authors: name: Chengqi Lyu, name: Songyang Gao, name: Yuzhe Gu, name: Wenwei Zhang, name: Jianfei Gao, name: Kuikun Liu, name: Ziyi Wang, name: Shuaibin Li, name: Qian Zhao, name: Haian Huang, name: Weihan Cao, name: Jiangning Liu, name: Hongwei Liu, name: Junnan Liu, name: Songyang Zhang, name: Dahua Lin, name: Kai Chen
Published: 2025-02-10 18:57:29

Reasoning abilities, especially those for solving complex math problems, are
crucial components of general intelligence. Recent advances by proprietary
companies, such as o-series models of OpenAI, have made remarkable progress on
reasoning tasks. However, the complete technical details remain unrevealed, and
the techniques that are believed certainly to be adopted are only reinforcement
learning (RL) and the long chain of thoughts. This paper proposes a new RL
framework, termed OREAL, to pursue the performance limit that can be achieved
through Outcome REwArd-based reinforcement
Learning for mathematical reasoning tasks, where only binary outcome
rewards are easily accessible. We theoretically prove that behavior cloning on
positive trajectories from best-of-N (BoN) sampling is sufficient to learn the
KL-regularized optimal policy in binary feedback environments. This formulation
further implies that the rewards of negative samples should be reshaped to
ensure the gradient consistency between positive and negative samples. To
alleviate the long-existing difficulties brought by sparse rewards in RL, which
are even exacerbated by the partial correctness of the long chain of thought
for reasoning tasks, we further apply a token-level reward model to sample
important tokens in reasoning trajectories for learning. With OREAL, for the
first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL,
being on par with 32B models. OREAL-32B also surpasses previous 32B models
trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our
investigation also indicates the importance of initial policy models and
training queries for RL. Code, models, and data will be released to benefit
future researchhttps://github.com/InternLM/OREAL.
Lossless Acceleration of Large Language Models with Hierarchical
  Drafting based on Temporal Locality in Speculative Decoding
Authors: name: Sukmin Cho, name: Sangjin Choi, name: Taeho Hwang, name: Jeongyeon Seo, name: Soyeong Jeong, name: Huije Lee, name: Hoyun Song, name: Jong C. Park, name: Youngjin Kwon
Published: 2025-02-08 15:32:53

Accelerating inference in Large Language Models (LLMs) is critical for
real-time interactions, as they have been widely incorporated into real-world
services. Speculative decoding, a fully algorithmic solution, has gained
attention for improving inference speed by drafting and verifying tokens,
thereby generating multiple tokens in a single forward pass. However, current
drafting strategies usually require significant fine-tuning or have
inconsistent performance across tasks. To address these challenges, we propose
Hierarchy Drafting (HD), a novel lossless drafting approach that organizes
various token sources into multiple databases in a hierarchical framework based
on temporal locality. In the drafting step, HD sequentially accesses multiple
databases to obtain draft tokens from the highest to the lowest locality,
ensuring consistent acceleration across diverse tasks and minimizing drafting
latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters
demonstrate that HD outperforms existing database drafting methods, achieving
robust inference speedups across model sizes, tasks, and temperatures.
ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates
Authors: name: Ling Yang, name: Zhaochen Yu, name: Bin Cui, name: Mengdi Wang
Published: 2025-02-10 18:51:47

We present that hierarchical LLM reasoning via scaling thought templates can
effectively optimize the reasoning search space and outperform the mathematical
reasoning capabilities of powerful LLMs like OpenAI o1-preview and DeepSeek V3.
We train our ReasonFlux-32B model with only 8 GPUs and introduces three
innovations: (i) a structured and generic thought template library, containing
around 500 high-level thought templates capable of generalizing to similar or
relevant reasoning problems; (ii) performing hierarchical reinforcement
learning on a sequence of thought templates instead of long CoTs, optimizing a
base LLM to plan out an optimal template trajectory for gradually handling
complex problems; (iii) a brand new inference scaling system that enables
hierarchical LLM reasoning by adaptively scaling thought templates at inference
time. With a template trajectory containing sequential thought templates, our
ReasonFlux-32B significantly advances math reasoning capabilities to
state-of-the-art levels. Notably, on the MATH benchmark, it achieves an
accuracy of 91.2% and surpasses o1-preview by 6.7%. On the USA Math Olympiad
(AIME) benchmark, ReasonFlux-32B solves an average of 56.7% of problems,
surpassing o1-preview and DeepSeek-V3 by 27% and 45%, respectively. Code:
https://github.com/Gen-Verse/ReasonFlux
EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
Authors: name: Haiwen Diao, name: Xiaotong Li, name: Yufeng Cui, name: Yueze Wang, name: Haoge Deng, name: Ting Pan, name: Wenxuan Wang, name: Huchuan Lu, name: Xinlong Wang
Published: 2025-02-10 18:59:58

Existing encoder-free vision-language models (VLMs) are rapidly narrowing the
performance gap with their encoder-based counterparts, highlighting the
promising potential for unified multimodal systems with structural simplicity
and efficient deployment. We systematically clarify the performance gap between
VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist
visual layers from scratch, deeply excavating the under-examined
characteristics of encoder-free VLMs. We develop efficient strategies for
encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth
investigation, we launch EVEv2.0, a new and improved family of encoder-free
VLMs. We show that: (i) Properly decomposing and hierarchically associating
vision and language within a unified model reduces interference between
modalities. (ii) A well-designed training strategy enables effective
optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0
represents a thorough study for developing a decoder-only architecture across
modalities, demonstrating superior data efficiency and strong vision-reasoning
capability. Code is publicly available at: https://github.com/baaivision/EVE.
Dual Caption Preference Optimization for Diffusion Models
Authors: name: Amir Saeidi, name: Yiran Luo, name: Agneet Chatterjee, name: Shamanthak Hegde, name: Bimsara Pathiraja, name: Yezhou Yang, name: Chitta Baral
Published: 2025-02-09 20:34:43

Recent advancements in human preference optimization, originally developed
for Large Language Models (LLMs), have shown significant potential in improving
text-to-image diffusion models. These methods aim to learn the distribution of
preferred samples while distinguishing them from less preferred ones. However,
existing preference datasets often exhibit overlap between these distributions,
leading to a conflict distribution. Additionally, we identified that input
prompts contain irrelevant information for less preferred images, limiting the
denoising network's ability to accurately predict noise in preference
optimization methods, known as the irrelevant prompt issue. To address these
challenges, we propose Dual Caption Preference Optimization (DCPO), a novel
approach that utilizes two distinct captions to mitigate irrelevant prompts. To
tackle conflict distribution, we introduce the Pick-Double Caption dataset, a
modified version of Pick-a-Pic v2 with separate captions for preferred and less
preferred images. We further propose three different strategies for generating
distinct captions: captioning, perturbation, and hybrid methods. Our
experiments show that DCPO significantly improves image quality and relevance
to prompts, outperforming Stable Diffusion (SD) 2.1, SFT_Chosen, Diffusion-DPO,
and MaPO across multiple metrics, including Pickscore, HPSv2.1, GenEval,
CLIPscore, and ImageReward, fine-tuned on SD 2.1 as the backbone.
APE: Faster and Longer Context-Augmented Generation via Adaptive
  Parallel Encoding
Authors: name: Xinyu Yang, name: Tianqi Chen, name: Beidi Chen
Published: 2025-02-08 03:41:16

Context-augmented generation (CAG) techniques, including RAG and ICL, require
the efficient combination of multiple contexts to generate responses to user
queries. Directly inputting these contexts as a sequence introduces a
considerable computational burden by re-encoding the combined selection of
contexts for every request. To address this, we explore the promising potential
of parallel encoding to independently pre-compute and cache each context's KV
states. This approach enables the direct loading of cached states during
inference while accommodating more contexts through position reuse across
contexts. However, due to misalignments in attention distribution, directly
applying parallel encoding results in a significant performance drop. To enable
effective and efficient CAG, we propose Adaptive Parallel Encoding
(APE), which brings shared prefix, attention temperature, and
scaling factor to align the distribution of parallel encoding with sequential
encoding. Results on RAG and ICL tasks demonstrate that APE can preserve 98%
and 93% sequential encoding performance using the same inputs while
outperforming parallel encoding by 3.6% and 7.9%, respectively. It also scales
to many-shot CAG, effectively encoding hundreds of contexts in parallel.
Efficiency evaluation shows that APE can achieve an end-to-end 4.5times
speedup by reducing 28times prefilling time for a 128K-length context.
Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building
  a Chinese-Centric LLM
Authors: name: Qingshui Gu, name: Shu Li, name: Tianyu Zheng, name: Zhaoxiang Zhang
Published: 2025-02-10 16:31:37

Steel-LLM is a Chinese-centric language model developed from scratch with the
goal of creating a high-quality, open-source model despite limited
computational resources. Launched in March 2024, the project aimed to train a
1-billion-parameter model on a large-scale dataset, prioritizing transparency
and the sharing of practical insights to assist others in the community. The
training process primarily focused on Chinese data, with a small proportion of
English data included, addressing gaps in existing open-source LLMs by
providing a more detailed and practical account of the model-building journey.
Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL
and CMMLU, outperforming early models from larger institutions. This paper
provides a comprehensive summary of the project's key contributions, including
data collection, model design, training methodologies, and the challenges
encountered along the way, offering a valuable resource for researchers and
practitioners looking to develop their own LLMs. The model checkpoints and
training script are available at https://github.com/zhanshijinwat/Steel-LLM.
LM2: Large Memory Models
Authors: name: Jikun Kang, name: Wenqi Wu, name: Filippos Christianos, name: Alex J. Chan, name: Fraser Greenlee, name: George Thomas, name: Marvin Purtorab, name: Andy Toulis
Published: 2025-02-09 22:11:42

This paper introduces the Large Memory Model (LM2), a decoder-only
Transformer architecture enhanced with an auxiliary memory module that aims to
address the limitations of standard Transformers in multi-step reasoning,
relational argumentation, and synthesizing information distributed over long
contexts. The proposed LM2 incorporates a memory module that acts as a
contextual representation repository, interacting with input tokens via cross
attention and updating through gating mechanisms. To preserve the Transformers
general-purpose capabilities, LM2 maintains the original information flow while
integrating a complementary memory pathway. Experimental results on the
BABILong benchmark demonstrate that the LM2model outperforms both the
memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3%
on average across tasks. LM2 exhibits exceptional capabilities in multi-hop
inference, numerical reasoning, and large-context question-answering. On the
MMLU dataset, it achieves a 5.0% improvement over a pre-trained vanilla model,
demonstrating that its memory module does not degrade performance on general
tasks. Further, in our analysis, we explore the memory interpretability,
effectiveness of memory modules, and test-time behavior. Our findings emphasize
the importance of explicit memory in enhancing Transformer architectures.
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
  Tile
Authors: name: Hangliang Ding, name: Dacheng Li, name: Runlong Su, name: Peiyuan Zhang, name: Zhijie Deng, name: Ion Stoica, name: Hao Zhang
Published: 2025-02-10 05:00:56

Despite the promise of synthesizing high-fidelity videos, Diffusion
Transformers (DiTs) with 3D full attention suffer from expensive inference due
to the complexity of attention computation and numerous sampling steps. For
example, the popular Open-Sora-Plan model consumes more than 9 minutes for
generating a single video of 29 frames. This paper addresses the inefficiency
issue from two aspects: 1) Prune the 3D full attention based on the redundancy
within video data; We identify a prevalent tile-style repetitive pattern in the
3D attention maps for video data, and advocate a new family of sparse 3D
attention that holds a linear complexity w.r.t. the number of video frames. 2)
Shorten the sampling process by adopting existing multi-step consistency
distillation; We split the entire sampling trajectory into several segments and
perform consistency distillation within each one to activate few-step
generation capacities. We further devise a three-stage training pipeline to
conjoin the low-complexity attention and few-step generation capacities.
Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
generation with a marginal performance trade-off in VBench. In addition, we
demonstrate that our approach is amenable to distributed inference, achieving
an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
The Hidden Life of Tokens: Reducing Hallucination of Large
  Vision-Language Models via Visual Information Steering
Authors: name: Zhuowei Li, name: Haizhou Shi, name: Yunhe Gao, name: Di Liu, name: Zhenting Wang, name: Yuxiao Chen, name: Ting Liu, name: Long Zhao, name: Hao Wang, name: Dimitris N. Metaxas
Published: 2025-02-05 21:34:02

Large Vision-Language Models (LVLMs) can reason effectively over both textual
and visual inputs, but they tend to hallucinate syntactically coherent yet
visually ungrounded contents. In this paper, we investigate the internal
dynamics of hallucination by examining the tokens logits rankings throughout
the generation process, revealing three key patterns in how LVLMs process
information: (1) gradual visual information loss -- visually grounded tokens
gradually become less favored throughout generation, and (2) early excitation
-- semantically meaningful tokens achieve peak activation in the layers earlier
than the final layer. (3) hidden genuine information -- visually grounded
tokens though not being eventually decided still retain relatively high
rankings at inference. Based on these insights, we propose VISTA (Visual
Information Steering with Token-logit Augmentation), a training-free
inference-time intervention framework that reduces hallucination while
promoting genuine information. VISTA works by combining two complementary
approaches: reinforcing visual information in activation space and leveraging
early layer activations to promote semantically meaningful decoding. Compared
to existing methods, VISTA requires no external supervision and is applicable
to various decoding strategies. Extensive experiments show that VISTA on
average reduces hallucination by abount 40% on evaluated open-ended generation
task, and it consistently outperforms existing methods on four benchmarks
across four architectures under three decoding strategies.
Adaptive Semantic Prompt Caching with VectorQ
Authors: name: Luis Gaspar Schroeder, name: Shu Liu, name: Alejandro Cuadron, name: Mark Zhao, name: Stephan Krusche, name: Alfons Kemper, name: Matei Zaharia, name: Joseph E. Gonzalez
Published: 2025-02-06 04:16:20

Semantic prompt caches reduce the latency and cost of large language model
(LLM) inference by reusing cached LLM-generated responses for semantically
similar prompts. Vector similarity metrics assign a numerical score to quantify
the similarity between an embedded prompt and its nearest neighbor in the
cache. Existing systems rely on a static threshold to classify whether the
similarity score is sufficiently high to result in a cache hit. We show that
this one-size-fits-all threshold is insufficient across different prompts. We
propose VectorQ, a framework to learn embedding-specific threshold regions that
adapt to the complexity and uncertainty of an embedding. Through evaluations on
a combination of four diverse datasets, we show that VectorQ consistently
outperforms state-of-the-art systems across all static thresholds, achieving up
to 12x increases in cache hit rate and error rate reductions up to 92%.
SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in
  LLMs
Authors: name: Dinithi Jayasuriya, name: Sina Tayebati, name: Davide Ettori, name: Ranganath Krishnan, name: Amit Ranjan Trivedi
Published: 2025-02-05 06:11:55

We propose SPARC, a lightweight continual learning framework for large
language models (LLMs) that enables efficient task adaptation through prompt
tuning in a lower-dimensional space. By leveraging principal component analysis
(PCA), we identify a compact subspace of the training data. Optimizing prompts
in this lower-dimensional space enhances training efficiency, as it focuses
updates on the most relevant features while reducing computational overhead.
Furthermore, since the model's internal structure remains unaltered, the
extensive knowledge gained from pretraining is fully preserved, ensuring that
previously learned information is not compromised during adaptation. Our method
achieves high knowledge retention in both task-incremental and
domain-incremental continual learning setups while fine-tuning only 0.04% of
the model's parameters. Additionally, by integrating LoRA, we enhance
adaptability to computational constraints, allowing for a tradeoff between
accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate
that our PCA-based prompt tuning combined with LoRA maintains full knowledge
retention while improving accuracy, utilizing only 1% of the model's
parameters. These results establish our approach as a scalable and
resource-efficient solution for continual learning in LLMs.
Intelligent Sensing-to-Action for Robust Autonomy at the Edge:
  Opportunities and Challenges
Authors: name: Amit Ranjan Trivedi, name: Sina Tayebati, name: Hemant Kumawat, name: Nastaran Darabi, name: Divake Kumar, name: Adarsh Kumar Kosta, name: Yeshwanth Venkatesha, name: Dinithi Jayasuriya, name: Nethmi Jayasinghe, name: Priyadarshini Panda, name: Saibal Mukhopadhyay, name: Kaushik Roy
Published: 2025-02-04 20:13:58

Autonomous edge computing in robotics, smart cities, and autonomous vehicles
relies on the seamless integration of sensing, processing, and actuation for
real-time decision-making in dynamic environments. At its core is the
sensing-to-action loop, which iteratively aligns sensor inputs with
computational models to drive adaptive control strategies. These loops can
adapt to hyper-local conditions, enhancing resource efficiency and
responsiveness, but also face challenges such as resource constraints,
synchronization delays in multi-modal data fusion, and the risk of cascading
errors in feedback loops. This article explores how proactive, context-aware
sensing-to-action and action-to-sensing adaptations can enhance efficiency by
dynamically adjusting sensing and computation based on task demands, such as
sensing a very limited part of the environment and predicting the rest. By
guiding sensing through control actions, action-to-sensing pathways can improve
task relevance and resource use, but they also require robust monitoring to
prevent cascading errors and maintain reliability. Multi-agent sensing-action
loops further extend these capabilities through coordinated sensing and actions
across distributed agents, optimizing resource use via collaboration.
Additionally, neuromorphic computing, inspired by biological systems, provides
an efficient framework for spike-based, event-driven processing that conserves
energy, reduces latency, and supports hierarchical control--making it ideal for
multi-agent optimization. This article highlights the importance of end-to-end
co-design strategies that align algorithmic models with hardware and
environmental dynamics and improve cross-layer interdependencies to improve
throughput, precision, and adaptability for energy-efficient edge autonomy in
complex environments.
Continuous 3D Perception Model with Persistent State
Authors: name: Qianqian Wang, name: Yifei Zhang, name: Aleksander Holynski, name: Alexei A. Efros, name: Angjoo Kanazawa
Published: 2025-01-21 18:59:23

We present a unified framework capable of solving a broad range of 3D tasks.
Our approach features a stateful recurrent model that continuously updates its
state representation with each new observation. Given a stream of images, this
evolving state can be used to generate metric-scale pointmaps (per-pixel 3D
points) for each new input in an online fashion. These pointmaps reside within
a common coordinate system, and can be accumulated into a coherent, dense scene
reconstruction that updates as new images arrive. Our model, called CUT3R
(Continuous Updating Transformer for 3D Reconstruction), captures rich priors
of real-world scenes: not only can it predict accurate pointmaps from image
observations, but it can also infer unseen regions of the scene by probing at
virtual, unobserved views. Our method is simple yet highly flexible, naturally
accepting varying lengths of images that may be either video streams or
unordered photo collections, containing both static and dynamic content. We
evaluate our method on various 3D/4D tasks and demonstrate competitive or
state-of-the-art performance in each. Project Page: https://cut3r.github.io/
Value-Based Deep RL Scales Predictably
Authors: name: Oleh Rybkin, name: Michal Nauman, name: Preston Fu, name: Charlie Snell, name: Pieter Abbeel, name: Sergey Levine, name: Aviral Kumar
Published: 2025-02-06 18:59:47

Scaling data and compute is critical to the success of machine learning.
However, scaling demands predictability: we want methods to not only perform
well with more compute or data, but also have their performance be predictable
from small-scale runs, without running the large-scale experiment. In this
paper, we show that value-based off-policy RL methods are predictable despite
community lore regarding their pathological behavior. First, we show that data
and compute requirements to attain a given performance level lie on a Pareto
frontier, controlled by the updates-to-data (UTD) ratio. By estimating this
frontier, we can predict this data requirement when given more compute, and
this compute requirement when given more data. Second, we determine the optimal
allocation of a total resource budget across data and compute for a given
performance and use it to determine hyperparameters that maximize performance
for a given budget. Third, this scaling behavior is enabled by first estimating
predictable relationships between hyperparameters, which is used to manage
effects of overfitting and plasticity loss unique to RL. We validate our
approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI
gym, and IsaacGym, when extrapolating to higher levels of data, compute,
budget, or performance.