Orionnnn/hub-stats huggingface dataset:
VisualOverload: Probing Visual Understanding of VLMs in Really Dense
  Scenes
Authors: name: Paul Gavrikov, name: Wei Lin, name: M. Jehanzeb Mirza, name: Soumya Jahagirdar, name: Muhammad Huzaifa, name: Sivan Doveh, name: Serena Yeung-Levy, name: James Glass, name: Hilde Kuehne
Published: 2025-09-29 18:00:25

Is basic visual understanding really solved in state-of-the-art VLMs? We
present VisualOverload, a slightly different visual question answering (VQA)
benchmark comprising 2,720 question-answer pairs, with privately held
ground-truth responses. Unlike prior VQA datasets that typically focus on near
global image understanding, VisualOverload challenges models to perform simple,
knowledge-free vision tasks in densely populated (or, overloaded) scenes. Our
dataset consists of high-resolution scans of public-domain paintings that are
populated with multiple figures, actions, and unfolding subplots set against
elaborately detailed backdrops. We manually annotated these images with
questions across six task categories to probe for a thorough understanding of
the scene. We hypothesize that current benchmarks overestimate the performance
of VLMs, and encoding and reasoning over details is still a challenging task
for them, especially if they are confronted with densely populated scenes.
Indeed, we observe that even the best model (o3) out of 37 tested models only
achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on
all questions. Beyond a thorough evaluation, we complement our benchmark with
an error analysis that reveals multiple failure modes, including a lack of
counting skills, failure in OCR, and striking logical inconsistencies under
complex tasks. Altogether, VisualOverload exposes a critical gap in current
vision models and offers a crucial resource for the community to develop better
models.
  Benchmark: http://paulgavrikov.github.io/visualoverload
TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
Authors: name: Zhepei Wei, name: Xiao Yang, name: Kai Sun, name: Jiaqi Wang, name: Rulin Shao, name: Sean Chen, name: Mohammad Kachuee, name: Teja Gollapudi, name: Tony Liao, name: Nicolas Scheffer, name: Rakesh Wanga, name: Anuj Kumar, name: Yu Meng, name: Wen-tau Yih, name: Xin Luna Dong
Published: 2025-09-30 04:25:17

While large language models (LLMs) have demonstrated strong performance on
factoid question answering, they are still prone to hallucination and
untruthful responses, particularly when tasks demand information outside their
parametric knowledge. Indeed, truthfulness requires more than accuracy --
models must also recognize uncertainty and abstain when unsure to avoid
hallucinations. This presents a fundamental challenge for existing methods:
approaches that optimize for accuracy often amplify hallucinations, while those
that encourage abstention can become overly conservative, sacrificing correct
answers. Both extremes ultimately compromise truthfulness. In this work, we
present TruthRL, a general reinforcement learning (RL) framework that directly
optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using
GRPO with a simple yet effective ternary reward that distinguishes correct
answers, hallucinations, and abstentions. It incentivizes models to reduce
hallucinations not only by providing correct responses, but also by enabling
abstention when uncertain, thereby improving truthfulness. Extensive
experiments across four knowledge-intensive benchmarks show that, compared to
vanilla RL, TruthRL significantly reduces hallucinations by 28.9% and improves
truthfulness by 21.1%, with consistent gains across various backbone models
(e.g., Qwen, Llama) under both retrieval and non-retrieval setups. In-depth
ablation study demonstrates that vanilla accuracy-driven methods, such as
supervised fine-tuning or RL with a binary reward, struggle to balance factual
correctness and uncertainty. In contrast, our proposed truthfulness-driven
TruthRL achieves strong performance in both accuracy and truthfulness,
underscoring the importance of learning objective design for developing
truthful LLMs.
Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified
  Self-Play
Authors: name: Qinsi Wang, name: Bo Liu, name: Tianyi Zhou, name: Jing Shi, name: Yueqian Lin, name: Yiran Chen, name: Hai Helen Li, name: Kun Wan, name: Wentian Zhao
Published: 2025-09-29 21:55:55

Although reinforcement learning (RL) can effectively enhance the reasoning
capabilities of vision-language models (VLMs), current methods remain heavily
dependent on labor-intensive datasets that require extensive manual
construction and verification, leading to extremely high training costs and
consequently constraining the practical deployment of VLMs. To address this
challenge, we propose Vision-Zero, a domain-agnostic framework enabling VLM
self-improvement through competitive visual games generated from arbitrary
image pairs. Specifically, Vision-Zero encompasses three main attributes: (1)
Strategic Self-Play Framework: Vision-Zero trains VLMs in "Who Is the
Spy"-style games, where the models engage in strategic reasoning and actions
across multiple roles. Through interactive gameplay, models autonomously
generate their training data without human annotation. (2) Gameplay from
Arbitrary Images: Unlike existing gamified frameworks, Vision-Zero can generate
games from arbitrary images, thereby enhancing the model's reasoning ability
across diverse domains and showing strong generalization to different tasks. We
demonstrate this versatility using three distinct types of image datasets:
CLEVR-based synthetic scenes, charts, and real-world images. (3) Sustainable
Performance Gain: We introduce Iterative Self-Play Policy Optimization
(Iterative-SPO), a novel training algorithm that alternates between Self-Play
and reinforcement learning with verifiable rewards (RLVR), mitigating the
performance plateau often seen in self-play-only training and achieving
sustained long-term improvements. Despite using label-free data, Vision-Zero
achieves state-of-the-art performance on reasoning, chart question answering,
and vision-centric understanding tasks, surpassing other annotation-based
methods. Models and code has been released at
https://github.com/wangqinsi1/Vision-Zero.
Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Authors: name: Zhen Yang, name: Zi-Yi Dou, name: Di Feng, name: Forrest Huang, name: Anh Nguyen, name: Keen You, name: Omar Attia, name: Yuhao Yang, name: Michael Feng, name: Haotian Zhang, name: Ram Ramrakhya, name: Chao Jia, name: Jeffrey Nichols, name: Alexander Toshev, name: Yinfei Yang, name: Zhe Gan
Published: 2025-09-30 17:13:56

Developing autonomous agents that effectively interact with Graphic User
Interfaces (GUIs) remains a challenging open problem, especially for small
on-device models. In this paper, we present Ferret-UI Lite, a compact,
end-to-end GUI agent that operates across diverse platforms, including mobile,
web, and desktop. Utilizing techniques optimized for developing small models,
we build our 3B Ferret-UI Lite agent through curating a diverse GUI data
mixture from real and synthetic sources, strengthening inference-time
performance through chain-of-thought reasoning and visual tool-use, and
reinforcement learning with designed rewards. Ferret-UI Lite achieves
competitive performance with other small-scale GUI agents. In GUI grounding,
Ferret-UI Lite attains scores of 91.6%, 53.3%, and 61.2% on the
ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI
navigation, Ferret-UI Lite achieves success rates of 28.0% on AndroidWorld
and 19.8% on OSWorld. We share our methods and lessons learned from
developing compact, on-device GUI agents.
Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and
  Multi-Scale Global-Local Attention
Authors: name: Kai Li, name: Kejun Gao, name: Xiaolin Hu
Published: 2025-09-28 03:25:34

Audio-visual speech separation (AVSS) methods leverage visual cues to extract
target speech and have demonstrated strong separation quality in noisy acoustic
environments. However, these methods usually involve a large number of
parameters and require high computational cost, which is unacceptable in many
applications where speech separation serves as only a preprocessing step for
further speech processing. To address this issue, we propose an efficient AVSS
method, named Dolphin. For visual feature extraction, we develop DP-LipCoder, a
dual-path lightweight video encoder that transforms lip-motion into discrete
audio-aligned semantic tokens. For audio separation, we construct a lightweight
encoder-decoder separator, in which each layer incorporates a global-local
attention (GLA) block to efficiently capture multi-scale dependencies.
Experiments on three benchmark datasets showed that Dolphin not only surpassed
the current state-of-the-art (SOTA) model in separation quality but also
achieved remarkable improvements in efficiency: over 50% fewer parameters, more
than 2.4x reduction in MACs, and over 6x faster GPU inference speed. These
results indicate that Dolphin offers a practical and deployable solution for
high-performance AVSS in real-world scenarios. Our code and demo page are
publicly available at http://cslikai.cn/Dolphin/.
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in
  Real-world Applications
Authors: name: Wei He, name: Yueqing Sun, name: Hongyan Hao, name: Xueyuan Hao, name: Zhikang Xia, name: Qi Gu, name: Chengcheng Han, name: Dengchang Zhao, name: Hui Su, name: Kefeng Zhang, name: Man Gao, name: Xi Su, name: Xiaodong Cai, name: Xunliang Cai, name: Yu Yang, name: Yunke Zhao
Published: 2025-09-30 16:33:49

As LLM-based agents are increasingly deployed in real-life scenarios,
existing benchmarks fail to capture their inherent complexity of handling
extensive information, leveraging diverse resources, and managing dynamic user
interactions. To address this gap, we introduce VitaBench, a challenging
benchmark that evaluates agents on versatile interactive tasks grounded in
real-world settings. Drawing from daily applications in food delivery, in-store
consumption, and online travel services, VitaBench presents agents with the
most complex life-serving simulation environment to date, comprising 66 tools.
Through a framework that eliminates domain-specific policies, we enable
flexible composition of these scenarios and tools, yielding 100 cross-scenario
tasks (main results) and 300 single-scenario tasks. Each task is derived from
multiple real user requests and requires agents to reason across temporal and
spatial dimensions, utilize complex tool sets, proactively clarify ambiguous
instructions, and track shifting user intent throughout multi-turn
conversations. Moreover, we propose a rubric-based sliding window evaluator,
enabling robust assessment of diverse solution pathways in complex environments
and stochastic interactions. Our comprehensive evaluation reveals that even the
most advanced models achieve only 30% success rate on cross-scenario tasks, and
less than 50% success rate on others. Overall, we believe VitaBench will serve
as a valuable resource for advancing the development of AI agents in practical
real-world applications. The code, dataset, and leaderboard are available at
https://vitabench.github.io/
Attention as a Compass: Efficient Exploration for Process-Supervised RL
  in Reasoning Models
Authors: name: Runze Liu, name: Jiakang Wang, name: Yuling Shi, name: Zhihui Xie, name: Chenxin An, name: Kaiyan Zhang, name: Jian Zhao, name: Xiaodong Gu, name: Lei Lin, name: Wenping Hu, name: Xiu Li, name: Fuzheng Zhang, name: Guorui Zhou, name: Kun Gai
Published: 2025-09-30 17:58:34

Reinforcement Learning (RL) has shown remarkable success in enhancing the
reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL
(PSRL) has emerged as a more effective paradigm compared to outcome-based RL.
However, existing PSRL approaches suffer from limited exploration efficiency,
both in terms of branching positions and sampling. In this paper, we introduce
a novel PSRL framework (AttnRL), which enables efficient exploration for
reasoning models. Motivated by preliminary observations that steps exhibiting
high attention scores correlate with reasoning behaviors, we propose to branch
from positions with high values. Furthermore, we develop an adaptive sampling
strategy that accounts for problem difficulty and historical batch size,
ensuring that the whole training batch maintains non-zero advantage values. To
further improve sampling efficiency, we design a one-step off-policy training
pipeline for PSRL. Extensive experiments on multiple challenging mathematical
reasoning benchmarks demonstrate that our method consistently outperforms prior
approaches in terms of performance and sampling and training efficiency.
dParallel: Learnable Parallel Decoding for dLLMs
Authors: name: Zigeng Chen, name: Gongfan Fang, name: Xinyin Ma, name: Ruonan Yu, name: Xinchao Wang
Published: 2025-09-30 16:32:52

Diffusion large language models (dLLMs) have recently drawn considerable
attention within the research community as a promising alternative to
autoregressive generation, offering parallel token prediction and lower
inference latency. Yet, their parallel decoding potential remains largely
underexplored, as existing open-source models still require nearly token-length
decoding steps to ensure performance. To address this, we introduce dParallel,
a simple and effective method that unlocks the inherent parallelism of dLLMs
for fast sampling. We identify that the key bottleneck to parallel decoding
arises from the sequential certainty convergence for masked tokens. Building on
this insight, we introduce the core of our approach: certainty-forcing
distillation, a novel training strategy that distills the model to follow its
original sampling trajectories while enforcing it to achieve high certainty on
masked tokens more rapidly and in parallel. Extensive experiments across
various benchmarks demonstrate that our method can dramatically reduce the
number of decoding steps while maintaining performance. When applied to the
LLaDA-8B-Instruct model, dParallel reduces decoding steps from 256 to 30 on
GSM8K, achieving an 8.5x speedup without performance degradation. On the MBPP
benchmark, it cuts decoding steps from 256 to 24, resulting in a 10.5x speedup
while maintaining accuracy. Our code is available at
https://github.com/czg1225/dParallel
Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced
  Performance Gap
Authors: name: Yueqian Lin, name: Zhengmian Hu, name: Qinsi Wang, name: Yudong Liu, name: Hengfan Zhang, name: Jayakumar Subramanian, name: Nikos Vlassis, name: Hai Helen Li, name: Yiran Chen
Published: 2025-09-30 17:17:09

We present Voice Evaluation of Reasoning Ability (VERA), a benchmark for
evaluating reasoning ability in voice-interactive systems under real-time
conversational constraints. VERA comprises 2,931 voice-native episodes derived
from established text benchmarks and organized into five tracks (Math, Web,
Science, Long-Context, Factual). Each item is adapted for speech interaction
while preserving reasoning difficulty. VERA enables direct text-voice
comparison within model families and supports analysis of how architectural
choices affect reliability. We assess 12 contemporary voice systems alongside
strong text baselines and observe large, consistent modality gaps: on
competition mathematics a leading text model attains 74.8% accuracy while its
voice counterpart reaches 6.1%; macro-averaged across tracks the best text
models achieve 54.0% versus 11.3% for voice. Latency-accuracy analyses reveal a
low-latency plateau, where fast voice systems cluster around ~10% accuracy,
while approaching text performance requires sacrificing real-time interaction.
Diagnostic experiments indicate that common mitigations are insufficient.
Increasing "thinking time" yields negligible gains; a decoupled cascade that
separates reasoning from narration improves accuracy but still falls well short
of text and introduces characteristic grounding/consistency errors. Failure
analyses further show distinct error signatures across native streaming,
end-to-end, and cascade designs. VERA provides a reproducible testbed and
targeted diagnostics for architectures that decouple thinking from speaking,
offering a principled way to measure progress toward real-time voice assistants
that are both fluent and reliably reasoned.
Thinking Sparks!: Emergent Attention Heads in Reasoning Models During
  Post Training
Authors: name: Yein Park, name: Minbyul Jeong, name: Jaewoo Kang
Published: 2025-09-30 04:23:43

The remarkable capabilities of modern large reasoning models are largely
unlocked through post-training techniques such as supervised fine-tuning and
reinforcement learning. However, the architectural mechanisms behind such
improvements remain largely opaque. In this work, we use circuit analysis to
demonstrate that post-training for complex reasoning sparks the emergence of
novel, functionally specialized attention heads. These heads collectively
support structured reasoning and computation. Our comparative analysis across
Qwen families and DeepSeek-distilled model reveals that these emergent heads
evolve differently under different training regimes. Distillation and SFT
foster a cumulative addition of stable reasoning heads. In contrast, group
relative policy optimization operates in a dynamic search mode: relatively few
attention heads are iteratively activated, evaluated, and pruned, with their
survival closely tracking fluctuations in the task reward signal. Furthermore,
we find that controllable think on/off models do not possess dedicated thinking
heads. Instead, turning off explicit reasoning triggers a broader-but less
efficient-set of compensatory heads. Through ablation and qualitative analyses,
we connect these circuit-level dynamics to a crucial performance trade-off:
strengthened heads enable sophisticated problem-solving strategies for
difficult problems but can also introduce over-thinking failure modes, such as
calculation errors or logical loops on simpler tasks. These findings connect
circuit-level dynamics to macro-level performance, identifying an inherent
tension where complex reasoning comes at the cost of elementary computations.
More broadly, our work points to future directions for training policy design,
emphasizing the need to balance the development of effective reasoning
strategies with the assurance of reliable, flawless execution.
DC-VideoGen: Efficient Video Generation with Deep Compression Video
  Autoencoder
Authors: name: Junyu Chen, name: Wenkun He, name: Yuchao Gu, name: Yuyang Zhao, name: Jincheng Yu, name: Junsong Chen, name: Dongyun Zou, name: Yujun Lin, name: Zhekai Zhang, name: Muyang Li, name: Haocheng Xi, name: Ligeng Zhu, name: Enze Xie, name: Song Han, name: Han Cai
Published: 2025-09-29 17:59:31

We introduce DC-VideoGen, a post-training acceleration framework for
efficient video generation. DC-VideoGen can be applied to any pre-trained video
diffusion model, improving efficiency by adapting it to a deep compression
latent space with lightweight fine-tuning. The framework builds on two key
innovations: (i) a Deep Compression Video Autoencoder with a novel chunk-causal
temporal design that achieves 32x/64x spatial and 4x temporal compression while
preserving reconstruction quality and generalization to longer videos; and (ii)
AE-Adapt-V, a robust adaptation strategy that enables rapid and stable transfer
of pre-trained models into the new latent space. Adapting the pre-trained
Wan-2.1-14B model with DC-VideoGen requires only 10 GPU days on the NVIDIA H100
GPU. The accelerated models achieve up to 14.8x lower inference latency than
their base counterparts without compromising quality, and further enable
2160x3840 video generation on a single GPU. Code:
https://github.com/dc-ai-projects/DC-VideoGen.
Benefits and Pitfalls of Reinforcement Learning for Language Model
  Planning: A Theoretical Perspective
Authors: name: Siwei Wang, name: Yifei Shen, name: Haoran Sun, name: Shi Feng, name: Shang-Hua Teng, name: Li Dong, name: Yaru Hao, name: Wei Chen
Published: 2025-09-26 17:39:48

Recent reinforcement learning (RL) methods have substantially enhanced the
planning capabilities of Large Language Models (LLMs), yet the theoretical
basis for their effectiveness remains elusive. In this work, we investigate
RL's benefits and limitations through a tractable graph-based abstraction,
focusing on policy gradient (PG) and Q-learning methods. Our theoretical
analyses reveal that supervised fine-tuning (SFT) may introduce
co-occurrence-based spurious solutions, whereas RL achieves correct planning
primarily through exploration, underscoring exploration's role in enabling
better generalization. However, we also show that PG suffers from diversity
collapse, where output diversity decreases during training and persists even
after perfect accuracy is attained. By contrast, Q-learning provides two key
advantages: off-policy learning and diversity preservation at convergence. We
further demonstrate that careful reward design is necessary to prevent reward
hacking in Q-learning. Finally, applying our framework to the real-world
planning benchmark Blocksworld, we confirm that these behaviors manifest in
practice.
Learning to See Before Seeing: Demystifying LLM Visual Priors from
  Language Pre-training
Authors: name: Junlin Han, name: Shengbang Tong, name: David Fan, name: Yufan Ren, name: Koustuv Sinha, name: Philip Torr, name: Filippos Kokkinos
Published: 2025-09-30 17:57:44

Large Language Models (LLMs), despite being trained on text alone,
surprisingly develop rich visual priors. These priors allow latent visual
capabilities to be unlocked for vision tasks with a relatively small amount of
multimodal data, and in some cases, to perform visual tasks without ever having
seen an image. Through systematic analysis, we reveal that visual priors-the
implicit, emergent knowledge about the visual world acquired during language
pre-training-are composed of separable perception and reasoning priors with
unique scaling trends and origins. We show that an LLM's latent visual
reasoning ability is predominantly developed by pre-training on
reasoning-centric data (e.g., code, math, academia) and scales progressively.
This reasoning prior acquired from language pre-training is transferable and
universally applicable to visual reasoning. In contrast, a perception prior
emerges more diffusely from broad corpora, and perception ability is more
sensitive to the vision encoder and visual instruction tuning data. In
parallel, text describing the visual world proves crucial, though its
performance impact saturates rapidly. Leveraging these insights, we propose a
data-centric recipe for pre-training vision-aware LLMs and verify it in 1T
token scale pre-training. Our findings are grounded in over 100 controlled
experiments consuming 500,000 GPU-hours, spanning the full MLLM construction
pipeline-from LLM pre-training to visual alignment and supervised multimodal
fine-tuning-across five model scales, a wide range of data categories and
mixtures, and multiple adaptation setups. Along with our main findings, we
propose and investigate several hypotheses, and introduce the Multi-Level
Existence Bench (MLE-Bench). Together, this work provides a new way of
deliberately cultivating visual priors from language pre-training, paving the
way for the next generation of multimodal LLMs.