Orionnnn/hub-stats huggingface dataset:
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
Authors: name: Ilgee Hong, name: Changlong Yu, name: Liang Qiu, name: Weixiang Yan, name: Zhenghao Xu, name: Haoming Jiang, name: Qingru Zhang, name: Qin Lu, name: Xin Liu, name: Chao Zhang, name: Tuo Zhao
Published: 2025-05-22 05:56:11

Reinforcement learning from human feedback (RLHF) has become a powerful
post-training paradigm for aligning large language models with human
preferences. A core challenge in RLHF is constructing accurate reward signals,
where the conventional Bradley-Terry reward models (BT RMs) often suffer from
sensitivity to data size and coverage, as well as vulnerability to reward
hacking. Generative reward models (GenRMs) offer a more robust alternative by
generating chain-of-thought (CoT) rationales followed by a final reward.
However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting
their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks.
Moreover, their pairwise preference outputs are incompatible with standard RLHF
algorithms that require pointwise reward signals. In this work, we introduce
Think-RM, a training framework that enables long-horizon reasoning in GenRMs by
modeling an internal thinking process. Rather than producing structured,
externally provided rationales, Think-RM generates flexible, self-guided
reasoning traces that support advanced capabilities such as self-reflection,
hypothetical reasoning, and divergent reasoning. To elicit these reasoning
abilities, we first warm-up the models by supervised fine-tuning (SFT) over
long CoT data. We then further improve the model's long-horizon abilities by
rule-based reinforcement learning (RL). In addition, we propose a novel
pairwise RLHF pipeline that directly optimizes policies using pairwise
preference rewards, eliminating the need for pointwise reward conversion and
enabling more effective use of Think-RM outputs. Experiments show that Think-RM
achieves state-of-the-art results on RM-Bench, outperforming both BT RM and
vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline,
it demonstrates superior end-policy performance compared to traditional
approaches.
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement
  Learning
Authors: name: Zhepei Wei, name: Wenlin Yao, name: Yao Liu, name: Weizhi Zhang, name: Qin Lu, name: Liang Qiu, name: Changlong Yu, name: Puyang Xu, name: Chao Zhang, name: Bing Yin, name: Hyokun Yun, name: Lihong Li
Published: 2025-05-22 09:07:43

While reinforcement learning (RL) has demonstrated remarkable success in
enhancing large language models (LLMs), it has primarily focused on single-turn
tasks such as solving math problems. Training effective web agents for
multi-turn interactions remains challenging due to the complexity of
long-horizon decision-making across dynamic web interfaces. In this work, we
present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework
for training web agents. It learns directly from online interactions with web
environments by asynchronously generating diverse trajectories, entirely guided
by binary rewards depending on task success. Experiments on the WebArena-Lite
benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task
success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to
44.8%, significantly outperforming existing state-of-the-art methods and strong
proprietary models such as OpenAI o3. In-depth analyses reveal the
effectiveness of the thinking-based prompting strategy and test-time scaling
through increased interactions for web tasks. We further investigate different
RL initialization policies by introducing two variants, namely WebAgent-R1-Zero
and WebAgent-R1-CoT, which highlight the importance of the warm-up training
stage (i.e., behavior cloning) and provide insights on incorporating long
chain-of-thought (CoT) reasoning in web agents.
SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based
  on Speech and Audio Information
Authors: name: Chih-Kai Yang, name: Neo Ho, name: Yen-Ting Piao, name: Hung-yi Lee
Published: 2025-05-19 15:20:32

Large audio-language models (LALMs) extend the large language models with
multimodal understanding in speech, audio, etc. While their performances on
speech and audio-processing tasks are extensively studied, their reasoning
abilities remain underexplored. Particularly, their multi-hop reasoning, the
ability to recall and integrate multiple facts, lacks systematic evaluation.
Existing benchmarks focus on general speech and audio-processing tasks,
conversational abilities, and fairness but overlook this aspect. To bridge this
gap, we introduce SAKURA, a benchmark assessing LALMs' multi-hop reasoning
based on speech and audio information. Results show that LALMs struggle to
integrate speech/audio representations for multi-hop reasoning, even when they
extract the relevant information correctly, highlighting a fundamental
challenge in multimodal reasoning. Our findings expose a critical limitation in
LALMs, offering insights and resources for future research.
Training Step-Level Reasoning Verifiers with Formal Verification Tools
Authors: name: Ryo Kamoi, name: Yusen Zhang, name: Nan Zhang, name: Sarkar Snigdha Sarathi Das, name: Rui Zhang
Published: 2025-05-21 19:23:45

Process Reward Models (PRMs), which provide step-by-step feedback on the
reasoning generated by Large Language Models (LLMs), are receiving increasing
attention. However, two key research gaps remain: collecting accurate
step-level error labels for training typically requires costly human
annotation, and existing PRMs are limited to math reasoning problems. In
response to these gaps, this paper aims to address the challenges of automatic
dataset creation and the generalization of PRMs to diverse reasoning tasks. To
achieve this goal, we propose FoVer, an approach for training PRMs on
step-level error labels automatically annotated by formal verification tools,
such as Z3 for formal logic and Isabelle for theorem proof, which provide
automatic and accurate verification for symbolic tasks. Using this approach, we
synthesize a training dataset with error labels on LLM responses for formal
logic and theorem proof tasks without human annotation. Although this data
synthesis is feasible only for tasks compatible with formal verification, we
observe that LLM-based PRMs trained on our dataset exhibit cross-task
generalization, improving verification across diverse reasoning tasks.
Specifically, PRMs trained with FoVer significantly outperform baseline PRMs
based on the original LLMs and achieve competitive or superior results compared
to state-of-the-art PRMs trained on labels annotated by humans or stronger
models, as measured by step-level verification on ProcessBench and Best-of-K
performance across 12 reasoning benchmarks, including MATH, AIME, ANLI, MMLU,
and BBH. The datasets, models, and code are provided at
https://github.com/psunlpgroup/FoVer.
RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture
  Understanding
Authors: name: Jiaang Li, name: Yifei Yuan, name: Wenyan Li, name: Mohammad Aliannejadi, name: Daniel Hershcovich, name: Anders Søgaard, name: Ivan Vulić, name: Wenxuan Zhang, name: Paul Pu Liang, name: Yang Deng, name: Serge Belongie
Published: 2025-05-20 14:57:16

As vision-language models (VLMs) become increasingly integrated into daily
life, the need for accurate visual culture understanding is becoming critical.
Yet, these models frequently fall short in interpreting cultural nuances
effectively. Prior work has demonstrated the effectiveness of
retrieval-augmented generation (RAG) in enhancing cultural understanding in
text-only settings, while its application in multimodal scenarios remains
underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented
Visual culturE uNdErstAnding), a new benchmark designed to advance visual
culture understanding through retrieval, focusing on two tasks: culture-focused
visual question answering (cVQA) and culture-informed image captioning (cIC).
RAVENEA extends existing datasets by integrating over 10,000 Wikipedia
documents curated and ranked by human annotators. With RAVENEA, we train and
evaluate seven multimodal retrievers for each image query, and measure the
downstream impact of retrieval-augmented inputs across fourteen
state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented
with culture-aware retrieval, outperform their non-augmented counterparts (by
at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the
value of retrieval-augmented methods and culturally inclusive benchmarks for
multimodal understanding.
Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal
  Large Language Models
Authors: name: Runsen Xu, name: Weiyao Wang, name: Hao Tang, name: Xingyu Chen, name: Xiaodong Wang, name: Fu-Jen Chu, name: Dahua Lin, name: Matt Feiszli, name: Kevin J. Liang
Published: 2025-05-22 17:59:39

Multi-modal large language models (MLLMs) have rapidly advanced in visual
tasks, yet their spatial understanding remains limited to single images,
leaving them ill-suited for robotics and other real-world applications that
require multi-frame reasoning. In this paper, we propose a framework to equip
MLLMs with robust multi-frame spatial understanding by integrating depth
perception, visual correspondence, and dynamic perception. Central to our
approach is the MultiSPA dataset, a novel, large-scale collection of more than
27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we
introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks
under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves
significant gains over baselines and proprietary systems, demonstrating
scalable, generalizable multi-frame reasoning. We further observe multi-task
benefits and early indications of emergent capabilities in challenging
scenarios, and showcase how our model can serve as a multi-frame reward
annotator for robotics.
Let Androids Dream of Electric Sheep: A Human-like Image Implication
  Understanding and Reasoning Framework
Authors: name: Chenhao Zhang, name: Yazhe Niu
Published: 2025-05-22 17:59:53

Metaphorical comprehension in images remains a critical challenge for AI
systems, as existing models struggle to grasp the nuanced cultural, emotional,
and contextual implications embedded in visual content. While multimodal large
language models (MLLMs) excel in basic Visual Question Answer (VQA) tasks, they
struggle with a fundamental limitation on image implication tasks: contextual
gaps that obscure the relationships between different visual elements and their
abstract meanings. Inspired by the human cognitive process, we propose Let
Androids Dream (LAD), a novel framework for image implication understanding and
reasoning. LAD addresses contextual missing through the three-stage framework:
(1) Perception: converting visual information into rich and multi-level textual
representations, (2) Search: iteratively searching and integrating cross-domain
knowledge to resolve ambiguity, and (3) Reasoning: generating context-alignment
image implication via explicit reasoning. Our framework with the lightweight
GPT-4o-mini model achieves SOTA performance compared to 15+ MLLMs on English
image implication benchmark and a huge improvement on Chinese benchmark,
performing comparable with the GPT-4o model on Multiple-Choice Question (MCQ)
and outperforms 36.7% on Open-Style Question (OSQ). Additionally, our work
provides new insights into how AI can more effectively interpret image
implications, advancing the field of vision-language reasoning and human-AI
interaction. Our project is publicly available at
https://github.com/MING-ZCH/Let-Androids-Dream-of-Electric-Sheep.
Steering Large Language Models for Machine Translation Personalization
Authors: name: Daniel Scalena, name: Gabriele Sarti, name: Arianna Bisazza, name: Elisabetta Fersini, name: Malvina Nissim
Published: 2025-05-22 12:47:16

High-quality machine translation systems based on large language models
(LLMs) have simplified the production of personalized translations reflecting
specific stylistic constraints. However, these systems still struggle in
settings where stylistic requirements are less explicit and might be harder to
convey via prompting. We explore various strategies for personalizing
LLM-generated translations in low-resource settings, focusing on the
challenging literary translation domain. We explore prompting strategies and
inference-time interventions for steering model generations towards a
personalized style, and propose a contrastive framework exploiting latent
concepts extracted from sparse autoencoders to identify salient personalization
properties. Our results show that steering achieves strong personalization
while preserving translation quality. We further examine the impact of steering
on LLM representations, finding model layers with a relevant impact for
personalization are impacted similarly by multi-shot prompting and our steering
method, suggesting similar mechanism at play.
MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation
  Capabilities in Any Language
Authors: name: Seyoung Song, name: Seogyeong Jeong, name: Eunsu Kim, name: Jiho Jin, name: Dongkwan Kim, name: Jay Shin, name: Alice Oh
Published: 2025-05-20 14:14:00

Evaluating text generation capabilities of large language models (LLMs) is
challenging, particularly for low-resource languages where methods for direct
assessment are scarce. We propose MUG-Eval, a novel framework that evaluates
LLMs' multilingual generation capabilities by transforming existing benchmarks
into conversational tasks and measuring the LLMs' accuracies on those tasks. We
specifically designed these conversational tasks to require effective
communication in the target language. Then, we simply use task success rate as
a proxy of successful conversation generation. Our approach offers two key
advantages: it is independent of language-specific NLP tools or annotated
datasets, which are limited for most languages, and it does not rely on
LLMs-as-judges, whose evaluation quality degrades outside a few high-resource
languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and
low-resource categories, and we find that MUG-Eval correlates strongly with
established benchmarks (r > 0.75) while enabling standardized comparisons
across languages and models. Our framework provides a robust and
resource-efficient solution for evaluating multilingual generation that can be
extended to thousands of languages.
VLM-R^3: Region Recognition, Reasoning, and Refinement for Enhanced
  Multimodal Chain-of-Thought
Authors: name: Chaoya Jiang, name: Yongrui Heng, name: Wei Ye, name: Han Yang, name: Haiyang Xu, name: Ming Yan, name: Ji Zhang, name: Fei Huang, name: Shikun Zhang
Published: 2025-05-22 03:50:13

Recently, reasoning-based MLLMs have achieved a degree of success in
generating long-form textual reasoning chains. However, they still struggle
with complex tasks that necessitate dynamic and iterative focusing on and
revisiting of visual regions to achieve precise grounding of textual reasoning
in visual evidence. We introduce VLM-R^3 (Visual
Language Model with Region Recognition and
Reasoning), a framework that equips an MLLM with the ability to (i)
decide when additional visual evidence is needed, (ii) determine
where to ground within the image, and (iii) seamlessly weave the
relevant sub-image content back into an interleaved chain-of-thought. The core
of our method is Region-Conditioned Reinforcement Policy Optimization
(R-GRPO), a training paradigm that rewards the model for selecting informative
regions, formulating appropriate transformations (e.g.\ crop, zoom), and
integrating the resulting visual context into subsequent reasoning steps. To
bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual
Interleaved Rationale (VLIR) corpus that provides step-level supervision on
region selection and textual justification. Extensive experiments on MathVista,
ScienceQA, and other benchmarks show that VLM-R^3 sets a new state of the art
in zero-shot and few-shot settings, with the largest gains appearing on
questions demanding subtle spatial reasoning or fine-grained visual cue
extraction.
Risk-Averse Reinforcement Learning with Itakura-Saito Loss
Authors: name: Igor Udovichenko, name: Olivier Croissant, name: Anita Toleutaeva, name: Evgeny Burnaev, name: Alexander Korotin
Published: 2025-05-22 17:18:07

Risk-averse reinforcement learning finds application in various high-stakes
fields. Unlike classical reinforcement learning, which aims to maximize
expected returns, risk-averse agents choose policies that minimize risk,
occasionally sacrificing expected value. These preferences can be framed
through utility theory. We focus on the specific case of the exponential
utility function, where we can derive the Bellman equations and employ various
reinforcement learning algorithms with few modifications. However, these
methods suffer from numerical instability due to the need for exponent
computation throughout the process. To address this, we introduce a numerically
stable and mathematically sound loss function based on the Itakura-Saito
divergence for learning state-value and action-value functions. We evaluate our
proposed loss function against established alternatives, both theoretically and
empirically. In the experimental section, we explore multiple financial
scenarios, some with known analytical solutions, and show that our loss
function outperforms the alternatives.
Date Fragments: A Hidden Bottleneck of Tokenization for Temporal
  Reasoning
Authors: name: Gagan Bhatia, name: Maxime Peyrard, name: Wei Zhao
Published: 2025-05-22 00:06:29

Modern BPE tokenizers often split calendar dates into meaningless fragments,
e.g., 20250312 rightarrow 202, 503, 12, inflating token counts and obscuring
the inherent structure needed for robust temporal reasoning. In this work, we
(1) introduce a simple yet interpretable metric, termed date fragmentation
ratio, that measures how faithfully a tokenizer preserves multi-digit date
components; (2) release DateAugBench, a suite of 6500 examples spanning three
temporal reasoning tasks: context-based date resolution, format-invariance
puzzles, and date arithmetic across historical, contemporary, and future
regimes; and (3) through layer-wise probing and causal attention-hop analyses,
uncover an emergent date-abstraction mechanism whereby large language models
stitch together the fragments of month, day, and year components for temporal
reasoning. Our experiments show that excessive fragmentation correlates with
accuracy drops of up to 10 points on uncommon dates like historical and
futuristic dates. Further, we find that the larger the model, the faster the
emergent date abstraction that heals date fragments is accomplished. Lastly, we
observe a reasoning path that LLMs follow to assemble date fragments, typically
differing from human interpretation (year rightarrow month rightarrow
day).
AGENTIF: Benchmarking Instruction Following of Large Language Models in
  Agentic Scenarios
Authors: name: Yunjia Qi, name: Hao Peng, name: Xiaozhi Wang, name: Amy Xin, name: Youfeng Liu, name: Bin Xu, name: Lei Hou, name: Juanzi Li
Published: 2025-05-22 17:31:10

Large Language Models (LLMs) have demonstrated advanced capabilities in
real-world agentic applications. Growing research efforts aim to develop
LLM-based agents to address practical demands, introducing a new challenge:
agentic scenarios often involve lengthy instructions with complex constraints,
such as extended system prompts and detailed tool specifications. While
adherence to such instructions is crucial for agentic applications, whether
LLMs can reliably follow them remains underexplored. In this paper, we
introduce AgentIF, the first benchmark for systematically evaluating LLM
instruction following ability in agentic scenarios. AgentIF features three key
characteristics: (1) Realistic, constructed from 50 real-world agentic
applications. (2) Long, averaging 1,723 words with a maximum of 15,630 words.
(3) Complex, averaging 11.9 constraints per instruction, covering diverse
constraint types, such as tool specifications and condition constraints. To
construct AgentIF, we collect 707 human-annotated instructions across 50
agentic tasks from industrial application agents and open-source agentic
systems. For each instruction, we annotate the associated constraints and
corresponding evaluation metrics, including code-based evaluation, LLM-based
evaluation, and hybrid code-LLM evaluation. We use AgentIF to systematically
evaluate existing advanced LLMs. We observe that current models generally
perform poorly, especially in handling complex constraint structures and tool
specifications. We further conduct error analysis and analytical experiments on
instruction length and meta constraints, providing some findings about the
failure modes of existing LLMs. We have released the code and data to
facilitate future research.
SPhyR: Spatial-Physical Reasoning Benchmark on Material Distribution
Authors: name: Philipp D. Siedler
Published: 2025-05-21 22:00:20

We introduce a novel dataset designed to benchmark the physical and spatial
reasoning capabilities of Large Language Models (LLM) based on topology
optimization, a method for computing optimal material distributions within a
design space under prescribed loads and supports. In this dataset, LLMs are
provided with conditions such as 2D boundary, applied forces and supports, and
must reason about the resulting optimal material distribution. The dataset
includes a variety of tasks, ranging from filling in masked regions within
partial structures to predicting complete material distributions. Solving these
tasks requires understanding the flow of forces and the required material
distribution under given constraints, without access to simulation tools or
explicit physical models, challenging models to reason about structural
stability and spatial organization. Our dataset targets the evaluation of
spatial and physical reasoning abilities in 2D settings, offering a
complementary perspective to traditional language and logic benchmarks.
How Do Large Vision-Language Models See Text in Image? Unveiling the
  Distinctive Role of OCR Heads
Authors: name: Ingeol Baek, name: Hwan Chang, name: Sunghyun Ryu, name: Hwanhee Lee
Published: 2025-05-21 10:53:41

Despite significant advancements in Large Vision Language Models (LVLMs), a
gap remains, particularly regarding their interpretability and how they locate
and interpret textual information within images. In this paper, we explore
various LVLMs to identify the specific heads responsible for recognizing text
from images, which we term the Optical Character Recognition Head (OCR Head).
Our findings regarding these heads are as follows: (1) Less Sparse: Unlike
previous retrieval heads, a large number of heads are activated to extract
textual information from images. (2) Qualitatively Distinct: OCR heads possess
properties that differ significantly from general retrieval heads, exhibiting
low similarity in their characteristics. (3) Statically Activated: The
frequency of activation for these heads closely aligns with their OCR scores.
We validate our findings in downstream tasks by applying Chain-of-Thought (CoT)
to both OCR and conventional retrieval heads and by masking these heads. We
also demonstrate that redistributing sink-token values within the OCR heads
improves performance. These insights provide a deeper understanding of the
internal mechanisms LVLMs employ in processing embedded textual information in
images.