Orionnnn/hub-stats huggingface dataset:
GEM: A Gym for Agentic LLMs
Authors: name: Zichen Liu, name: Anya Sims, name: Keyu Duan, name: Changyu Chen, name: Simon Yu, name: Xiangxin Zhou, name: Haotian Xu, name: Shaopan Xiong, name: Bo Liu, name: Chenmien Tan, name: Chuen Yang Beh, name: Weixun Wang, name: Hao Zhu, name: Weiyan Shi, name: Diyi Yang, name: Michael Shieh, name: Yee Whye Teh, name: Wee Sun Lee, name: Min Lin
Published: 2025-10-01 15:55:57

The training paradigm for large language models (LLMs) is moving from static
datasets to experience-based learning, where agents acquire skills via
interacting with complex environments. To facilitate this transition we
introduce GEM (General Experience Maker), an open-source environment simulator
designed for the age of LLMs. Analogous to OpenAI-Gym for traditional
reinforcement learning (RL), GEM provides a standardized framework for the
environment-agent interface, including asynchronous vectorized execution for
high throughput, and flexible wrappers for easy extensibility. GEM also
features a diverse suite of environments, robust integrated tools, and
single-file example scripts demonstrating using GEM with five popular RL
training frameworks. Along with this, we also provide a set of baselines across
24 environments using REINFORCE with Return Batch Normalization (ReBN), which
-- unlike GRPO -- is compatible with the full RL setting of dense per-turn
rewards and offers better credit assignment. We further conduct apple-to-apple
benchmarking of PPO, GRPO and REINFORCE in both single- and multi-turn settings
using GEM to shed light on the algorithmic designs. Lastly, GEM also functions
as a convenient evaluation toolkit besides a training environment. We hope this
framework can help accelerate future agentic LLM research.
GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
Authors: name: Kung-Hsiang Huang, name: Haoyi Qiu, name: Yutong Dai, name: Caiming Xiong, name: Chien-Sheng Wu
Published: 2025-10-01 05:37:54

Graphical user interface (GUI) agents built on vision-language models have
emerged as a promising approach to automate human-computer workflows. However,
they also face the inefficiency challenge as they process long sequences of
high-resolution screenshots and solving long-horizon tasks, making inference
slow, costly and memory-bound. While key-value (KV) caching can mitigate this,
storing the full cache is prohibitive for image-heavy contexts. Existing
cache-compression methods are sub-optimal as they do not account for the
spatial and temporal redundancy of GUIs. In this work, we first analyze
attention patterns in GUI agent workloads and find that, unlike in natural
images, attention sparsity is uniformly high across all transformer layers.
This insight motivates a simple uniform budget allocation strategy, which we
show empirically outperforms more complex layer-varying schemes. Building on
this, we introduce GUI-KV, a plug-and-play KV cache compression method for GUI
agents that requires no retraining. GUI-KV combines two novel techniques: (i)
spatial saliency guidance, which augments attention scores with the L2 norm of
hidden states to better preserve semantically important visual tokens, and (ii)
temporal redundancy scoring, which projects previous frames' keys onto the
current frame's key subspace to preferentially prune redundant history. Across
standard GUI agent benchmarks and models, GUI-KV outperforms competitive KV
compression baselines, closely matching full-cache accuracy at modest budgets.
Notably, in a 5-screenshot setting on the AgentNetBench benchmark, GUI-KV
reduces decoding FLOPs by 38.9% while increasing step accuracy by 4.1% over the
full-cache baseline. These results demonstrate that exploiting GUI-specific
redundancies enables efficient and reliable agent performance.
Beyond Log Likelihood: Probability-Based Objectives for Supervised
  Fine-Tuning across the Model Capability Continuum
Authors: name: Gaotang Li, name: Ruizhong Qiu, name: Xiusi Chen, name: Heng Ji, name: Hanghang Tong
Published: 2025-10-01 05:17:47

Supervised fine-tuning (SFT) is the standard approach for post-training large
language models (LLMs), yet it often shows limited generalization. We trace
this limitation to its default training objective: negative log likelihood
(NLL). While NLL is classically optimal when training from scratch,
post-training operates in a different paradigm and could violate its optimality
assumptions, where models already encode task-relevant priors and supervision
can be long and noisy. To this end, we study a general family of
probability-based objectives and characterize their effectiveness under
different conditions. Through comprehensive experiments and extensive ablation
studies across 7 model backbones, 14 benchmarks, and 3 domains, we uncover a
critical dimension that governs objective behavior: the model-capability
continuum. Near the model-strong end, prior-leaning objectives that downweight
low-probability tokens (e.g., -p, -p^{10}, thresholded variants)
consistently outperform NLL; toward the model-weak end, NLL dominates; in
between, no single objective prevails. Our theoretical analysis further
elucidates how objectives trade places across the continuum, providing a
principled foundation for adapting objectives to model capability. Our code is
available at https://github.com/GaotangLi/Beyond-Log-Likelihood.
VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified
  Rewards in World Simulators
Authors: name: Hengtao Li, name: Pengxiang Ding, name: Runze Suo, name: Yihao Wang, name: Zirui Ge, name: Dongyuan Zang, name: Kexian Yu, name: Mingyang Sun, name: Hongyin Zhang, name: Donglin Wang, name: Weihua Su
Published: 2025-10-01 01:33:10

Vision-Language-Action (VLA) models enable embodied decision-making but rely
heavily on imitation learning, leading to compounding errors and poor
robustness under distribution shift. Reinforcement learning (RL) can mitigate
these issues yet typically demands costly real-world interactions or suffers
from sim-to-real gaps. We introduce VLA-RFT, a reinforcement fine-tuning
framework that leverages a data-driven world model as a controllable simulator.
Trained from real interaction data, the simulator predicts future visual
observations conditioned on actions, allowing policy rollouts with dense,
trajectory-level rewards derived from goal-achieving references. This design
delivers an efficient and action-aligned learning signal, drastically lowering
sample requirements. With fewer than 400 fine-tuning steps, VLA-RFT surpasses
strong supervised baselines and achieves greater efficiency than
simulator-based RL. Moreover, it exhibits strong robustness under perturbed
conditions, sustaining stable task execution. Our results establish
world-model-based RFT as a practical post-training paradigm to enhance the
generalization and robustness of VLA models. For more details, please refer to
https://vla-rft.github.io/.
On Predictability of Reinforcement Learning Dynamics for Large Language
  Models
Authors: name: Yuchen Cai, name: Ding Cao, name: Xin Xu, name: Zijun Yao, name: Yuqing Huang, name: Zhenyu Tan, name: Benyi Zhang, name: Guiquan Liu, name: Junfeng Fang
Published: 2025-10-01 06:13:50

Recent advances in reasoning capabilities of large language models (LLMs) are
largely driven by reinforcement learning (RL), yet the underlying parameter
dynamics during RL training remain poorly understood. This work identifies two
fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1
Dominance, where the top singular subspace of the parameter update matrix
nearly fully determines reasoning improvements, recovering over 99\% of
performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace
evolves linearly throughout training, enabling accurate prediction from early
checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the
generalizability of these properties. More importantly, based on these
findings, we propose AlphaRL, a plug-in acceleration framework that
extrapolates the final parameter update using a short early training window,
achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning
performance without extra modules or hyperparameter tuning. This positions our
finding as a versatile and practical tool for large-scale RL, opening a path
toward principled, interpretable, and efficient training paradigm for LLMs.
Training Vision-Language Process Reward Models for Test-Time Scaling in
  Multimodal Reasoning: Key Insights and Lessons Learned
Authors: name: Brandon Ong, name: Tej Deep Pala, name: Vernon Toh, name: William Chandra Tjhi, name: Soujanya Poria
Published: 2025-09-27 10:56:58

Process Reward Models (PRMs) provide step-level supervision that improves the
reliability of reasoning in large language models. While PRMs have been
extensively studied in text-based domains, their extension to Vision Language
Models (VLMs) remains limited. Existing Vision-Language PRMs (VL-PRMs) rely on
Monte Carlo Tree Search (MCTS) for data construction, which can often produce
noisy supervision signals and limit generalization across tasks. In this work,
we aim to elucidate the design space of VL-PRMs by exploring diverse strategies
for dataset construction, training, and test-time scaling. First, we introduce
a hybrid data synthesis framework that combines MCTS with judgments from a
strong VLM, producing more accurate step-level labels. Second, we propose
perception-focused supervision, enabling our PRM to explicitly detect errors at
the visual grounding stage of reasoning. Third, we systematically evaluate
multiple test-time scaling strategies, showing that our PRMs can reliably guide
VLMs toward more accurate solutions. Our experiments covering five diverse
multimodal benchmarks (MMMU, PuzzleVQA, AlgoPuzzleVQA, MathVista, and
MathVision) reveal several key insights: (i) VL-PRMs when used as Outcome
Reward Models (ORMs) during test-time scaling (TTS) can outperform VL-PRM
guided process step selection, (ii) smaller VL-PRMs can match or even surpass
larger ones in detecting process errors, (iii) VL-PRMs uncover latent reasoning
abilities in stronger VLM backbones, (iv) perception-level supervision leads to
significant gains in test-time scaling, and (v) TTS performance of different
policies improve on advanced math reasoning datasets despite not training
VL-PRMs on such datasets. We hope our work will motivate further research and
support the advancement of VLMs.
In-Place Feedback: A New Paradigm for Guiding LLMs in Multi-Turn
  Reasoning
Authors: name: Youngbin Choi, name: Minjong Lee, name: Saemi Moon, name: Seunghyuk Cho, name: Chaehyeon Chung, name: MoonJeong Park, name: Dongwoo Kim
Published: 2025-10-01 11:16:04

Large language models (LLMs) are increasingly studied in the context of
multi-turn reasoning, where models iteratively refine their outputs based on
user-provided feedback. Such settings are crucial for tasks that require
complex reasoning, yet existing feedback paradigms often rely on issuing new
messages. LLMs struggle to integrate these reliably, leading to inconsistent
improvements. In this work, we introduce in-place feedback, a novel interaction
paradigm in which users directly edit an LLM's previous response, and the model
conditions on this modified response to generate its revision. Empirical
evaluations on diverse reasoning-intensive benchmarks reveal that in-place
feedback achieves better performance than conventional multi-turn feedback
while using 79.1% fewer tokens. Complementary analyses on controlled
environments further demonstrate that in-place feedback resolves a core
limitation of multi-turn feedback: models often fail to apply feedback
precisely to erroneous parts of the response, leaving errors uncorrected and
sometimes introducing new mistakes into previously correct content. These
findings suggest that in-place feedback offers a more natural and effective
mechanism for guiding LLMs in reasoning-intensive tasks.
Code2Video: A Code-centric Paradigm for Educational Video Generation
Authors: name: Yanzhe Chen, name: Kevin Qinghong Lin, name: Mike Zheng Shou
Published: 2025-10-01 17:56:48

While recent generative models advance pixel-space video synthesis, they
remain limited in producing professional educational videos, which demand
disciplinary knowledge, precise visual structures, and coherent transitions,
limiting their applicability in educational scenarios. Intuitively, such
requirements are better addressed through the manipulation of a renderable
environment, which can be explicitly controlled via logical commands (e.g.,
code). In this work, we propose Code2Video, a code-centric agent framework for
generating educational videos via executable Python code. The framework
comprises three collaborative agents: (i) Planner, which structures lecture
content into temporally coherent flows and prepares corresponding visual
assets; (ii) Coder, which converts structured instructions into executable
Python codes while incorporating scope-guided auto-fix to enhance efficiency;
and (iii) Critic, which leverages vision-language models (VLM) with visual
anchor prompts to refine spatial layout and ensure clarity. To support
systematic evaluation, we build MMMC, a benchmark of professionally produced,
discipline-specific educational videos. We evaluate MMMC across diverse
dimensions, including VLM-as-a-Judge aesthetic scores, code efficiency, and
particularly, TeachQuiz, a novel end-to-end metric that quantifies how well a
VLM, after unlearning, can recover knowledge by watching the generated videos.
Our results demonstrate the potential of Code2Video as a scalable,
interpretable, and controllable approach, achieving 40% improvement over direct
code generation and producing videos comparable to human-crafted tutorials. The
code and datasets are available at https://github.com/showlab/Code2Video.
BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model
  Responses
Authors: name: Xin Xu, name: Xunzhi He, name: Churan Zhi, name: Ruizhe Chen, name: Julian McAuley, name: Zexue He
Published: 2025-09-30 19:56:54

Existing studies on bias mitigation methods for large language models (LLMs)
use diverse baselines and metrics to evaluate debiasing performance, leading to
inconsistent comparisons among them. Moreover, their evaluations are mostly
based on the comparison between LLMs' probabilities of biased and unbiased
contexts, which ignores the gap between such evaluations and real-world use
cases where users interact with LLMs by reading model responses and expect fair
and safe outputs rather than LLMs' probabilities. To enable consistent
evaluation across debiasing methods and bridge this gap, we introduce
BiasFreeBench, an empirical benchmark that comprehensively compares eight
mainstream bias mitigation techniques (covering four prompting-based and four
training-based methods) on two test scenarios (multi-choice QA and open-ended
multi-turn QA) by reorganizing existing datasets into a unified query-response
setting. We further introduce a response-level metric, Bias-Free Score, to
measure the extent to which LLM responses are fair, safe, and
anti-stereotypical. Debiasing performances are systematically compared and
analyzed across key dimensions: the prompting vs. training paradigm, model
size, and generalization of different training strategies to unseen bias types.
We will publicly release our benchmark, aiming to establish a unified testbed
for bias mitigation research.
Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals
  Long-Range Dependency Pitfalls
Authors: name: Xiaoyan Bai, name: Itamar Pres, name: Yuntian Deng, name: Chenhao Tan, name: Stuart Shieber, name: Fernanda Viégas, name: Martin Wattenberg, name: Andrew Lee
Published: 2025-09-30 19:03:26

Language models are increasingly capable, yet still fail at a seemingly
simple task of multi-digit multiplication. In this work, we study why, by
reverse-engineering a model that successfully learns multiplication via
implicit chain-of-thought, and report three findings: (1) Evidence of
long-range structure: Logit attributions and linear probes indicate that the
model encodes the necessary long-range dependencies for multi-digit
multiplication. (2) Mechanism: the model encodes long-range dependencies using
attention to construct a directed acyclic graph to ``cache'' and ``retrieve''
pairwise partial products. (3) Geometry: the model implements partial products
in attention heads by forming Minkowski sums between pairs of digits, and
digits are represented using a Fourier basis, both of which are intuitive and
efficient representations that the standard fine-tuning model lacks. With these
insights, we revisit the learning dynamics of standard fine-tuning and find
that the model converges to a local optimum that lacks the required long-range
dependencies. We further validate this understanding by introducing an
auxiliary loss that predicts the ``running sum'' via a linear regression probe,
which provides an inductive bias that enables the model to successfully learn
multi-digit multiplication. In summary, by reverse-engineering the mechanisms
of an implicit chain-of-thought model we uncover a pitfall for learning
long-range dependencies in Transformers and provide an example of how the
correct inductive bias can address this issue.
Rethinking Reward Models for Multi-Domain Test-Time Scaling
Authors: name: Dong Bok Lee, name: Seanie Lee, name: Sangwoo Park, name: Minki Kang, name: Jinheon Baek, name: Dongki Kim, name: Dominik Wagner, name: Jiongdao Jin, name: Heejun Lee, name: Tobias Bocklet, name: Jinyu Wang, name: Jingjing Fu, name: Sung Ju Hwang, name: Jiang Bia, name: Lei Song
Published: 2025-10-01 04:21:14

The reliability of large language models (LLMs) during test-time scaling is
often assessed with external verifiers or reward models that
distinguish correct reasoning from flawed logic. Prior work generally assumes
that process reward models (PRMs), which score every intermediate reasoning
step, outperform outcome reward models (ORMs) that assess only the final
answer. This view is based mainly on evidence from narrow, math-adjacent
domains. We present the first unified evaluation of four reward model variants,
discriminative ORM and PRM (\DisORM, \DisPRM) and generative ORM and PRM
(\GenORM, \GenPRM), across 14 diverse domains. Contrary to conventional wisdom,
we find that (i) \DisORM performs on par with \DisPRM, (ii) \GenPRM is not
competitive, and (iii) overall, \GenORM is the most robust, yielding
significant and consistent gains across every tested domain. We attribute this
to PRM-style stepwise scoring, which inherits label noise from LLM
auto-labeling and has difficulty evaluating long reasoning trajectories,
including those involving self-correcting reasoning. Our theoretical analysis
shows that step-wise aggregation compounds errors as reasoning length grows,
and our empirical observations confirm this effect. These findings challenge
the prevailing assumption that fine-grained supervision is always better and
support generative outcome verification for multi-domain deployment. We
publicly release our code, datasets, and checkpoints at
https://github.com/db-Lee/Multi-RM{\small\texttt{https://github.com/db-Lee/Multi-RM}}
to facilitate future research in multi-domain settings.
The Pitfalls of KV Cache Compression
Authors: name: Alex Chen, name: Renato Geh, name: Aditya Grover, name: Guy Van den Broeck, name: Daniel Israel
Published: 2025-09-30 19:55:26

KV cache compression promises increased throughput and efficiency with
negligible loss in performance. While the gains in throughput are indisputable
and recent literature has indeed shown minimal degradation on particular
benchmarks, in general the consequences of compression in realistic scenarios
such as multi-instruction prompting have been insufficiently studied. In this
paper, we identify several pitfalls practitioners should be aware of when
deploying KV cache compressed LLMs. Importantly, we show that certain
instructions degrade much more rapidly with compression, effectively causing
them to be completely ignored by the LLM. As a practical example of that, we
highlight system prompt leakage as a case study, empirically showing the impact
of compression on leakage and general instruction following. We show several
factors that play a role in prompt leakage: compression method, instruction
order, and KV eviction bias. We then propose simple changes to KV cache
eviction policies that can reduce the impact of these factors and improve the
overall performance in multi-instruction tasks.
Catching the Details: Self-Distilled RoI Predictors for Fine-Grained
  MLLM Perception
Authors: name: Yuheng Shi, name: Xiaohuan Pei, name: Minjing Dong, name: Chang Xu
Published: 2025-09-21 06:54:04

Multimodal Large Language Models (MLLMs) require high-resolution visual
information to perform fine-grained perception, yet processing entire
high-resolution images is computationally prohibitive. While recent methods
leverage a Region-of-Interest (RoI) mechanism to focus on salient areas, they
typically present a difficult trade-off: training-based approaches depend on
large-scale annotated datasets, while training-free methods that utilize the
model's internal attention are computationally inefficient and less accurate,
requiring either multi-pass prefill stages or reliance on the slow
auto-regressive decoding process. In this paper, we propose an efficient,
annotation-free Self-Distilled Region Proposal Network (SD-RPN) that resolves
this trade-off. The SD-RPN is built around a pipeline that transforms the noisy
attention maps from the MLLM's middle layers into high-quality pseudo-RoI
labels by explicitly denoising the signal and resolving ambiguity. We use these
labels to train a lightweight Region Proposal Network (RPN) that learns a more
precise localization. This RPN is also highly efficient, predicting the RoI in
a single forward pass using features from the MLLM's middle layers, decoupling
RoI identification from the auto-regressive generation and avoiding costly
multi-pass operations.To validate our approach, we integrate the framework into
the LLaVA-1.5 architecture. Despite being trained on only a few (e.g. 10K)
question-answer pairs, our method demonstrates exceptional data efficiency and
generalization, achieving over a 10% absolute accuracy improvement on unseen
benchmarks, including TextVQA, DocVQA, and V-Star. Our work presents a
practical and scalable solution for enhancing the fine-grained perception of
MLLMs without requiring costly supervision or full model fine-tuning. Code is
available at https://github.com/YuHengsss/SD-RPN.
Swift: An Autoregressive Consistency Model for Efficient Weather
  Forecasting
Authors: name: Jason Stock, name: Troy Arcomano, name: Rao Kotamarthi
Published: 2025-09-30 00:54:24

Diffusion models offer a physically grounded framework for probabilistic
weather forecasting, but their typical reliance on slow, iterative solvers
during inference makes them impractical for subseasonal-to-seasonal (S2S)
applications where long lead-times and domain-driven calibration are essential.
To address this, we introduce Swift, a single-step consistency model that, for
the first time, enables autoregressive finetuning of a probability flow model
with a continuous ranked probability score (CRPS) objective. This eliminates
the need for multi-model ensembling or parameter perturbations. Results show
that Swift produces skillful 6-hourly forecasts that remain stable for up to 75
days, running 39times faster than state-of-the-art diffusion baselines while
achieving forecast skill competitive with the numerical-based, operational IFS
ENS. This marks a step toward efficient and reliable ensemble forecasting from
medium-range to seasonal-scales.