Orionnnn/hub-stats huggingface dataset:
HumaniBench: A Human-Centric Framework for Large Multimodal Models
  Evaluation
Authors: name: Shaina Raza, name: Aravind Narayanan, name: Vahid Reza Khazaie, name: Ashmal Vayani, name: Mukund S. Chettiar, name: Amandeep Singh, name: Mubarak Shah, name: Deval Pandya
Published: 2025-05-16 17:09:44

Large multimodal models (LMMs) now excel on many vision language benchmarks,
however, they still struggle with human centered criteria such as fairness,
ethics, empathy, and inclusivity, key to aligning with human values. We
introduce HumaniBench, a holistic benchmark of 32K real-world image question
pairs, annotated via a scalable GPT4o assisted pipeline and exhaustively
verified by domain experts. HumaniBench evaluates seven Human Centered AI
(HCAI) principles: fairness, ethics, understanding, reasoning, language
inclusivity, empathy, and robustness, across seven diverse tasks, including
open and closed ended visual question answering (VQA), multilingual QA, visual
grounding, empathetic captioning, and robustness tests. Benchmarking 15 state
of the art LMMs (open and closed source) reveals that proprietary models
generally lead, though robustness and visual grounding remain weak points. Some
open-source models also struggle to balance accuracy with adherence to
human-aligned principles. HumaniBench is the first benchmark purpose built
around HCAI principles. It provides a rigorous testbed for diagnosing alignment
gaps and guiding LMMs toward behavior that is both accurate and socially
responsible. Dataset, annotation prompts, and evaluation code are available at:
https://vectorinstitute.github.io/HumaniBench
Prior Prompt Engineering for Reinforcement Fine-Tuning
Authors: name: Pittawat Taveekitworachai, name: Potsawee Manakul, name: Sarana Nutanong, name: Kunat Pipatanakul
Published: 2025-05-20 10:05:11

This paper investigates prior prompt engineering (pPE) in the context of
reinforcement fine-tuning (RFT), where language models (LMs) are incentivized
to exhibit behaviors that maximize performance through reward signals. While
existing RFT research has primarily focused on algorithms, reward shaping, and
data curation, the design of the prior prompt--the instructions prepended to
queries during training to elicit behaviors such as step-by-step
reasoning--remains underexplored. We investigate whether different pPE
approaches can guide LMs to internalize distinct behaviors after RFT. Inspired
by inference-time prompt engineering (iPE), we translate five representative
iPE strategies--reasoning, planning, code-based reasoning, knowledge recall,
and null-example utilization--into corresponding pPE approaches. We experiment
with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on
in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and
GPQA-Diamond). Our results show that all pPE-trained models surpass their
iPE-prompted counterparts, with the null-example pPE approach achieving the
largest average performance gain and the highest improvement on AIME2024 and
GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by
adapting a behavior-classification framework, we demonstrate that different pPE
strategies instill distinct behavioral styles in the resulting models. These
findings position pPE as a powerful yet understudied axis for RFT.
RL Tango: Reinforcing Generator and Verifier Together for Language
  Reasoning
Authors: name: Kaiwen Zha, name: Zhengqi Gao, name: Maohao Shen, name: Zhang-Wei Hong, name: Duane S. Boning, name: Dina Katabi
Published: 2025-05-21 02:43:15

Reinforcement learning (RL) has recently emerged as a compelling approach for
enhancing the reasoning capabilities of large language models (LLMs), where an
LLM generator serves as a policy guided by a verifier (reward model). However,
current RL post-training methods for LLMs typically use verifiers that are
fixed (rule-based or frozen pretrained) or trained discriminatively via
supervised fine-tuning (SFT). Such designs are susceptible to reward hacking
and generalize poorly beyond their training distributions. To overcome these
limitations, we propose Tango, a novel framework that uses RL to concurrently
train both an LLM generator and a verifier in an interleaved manner. A central
innovation of Tango is its generative, process-level LLM verifier, which is
trained via RL and co-evolves with the generator. Importantly, the verifier is
trained solely based on outcome-level verification correctness rewards without
requiring explicit process-level annotations. This generative RL-trained
verifier exhibits improved robustness and superior generalization compared to
deterministic or SFT-trained verifiers, fostering effective mutual
reinforcement with the generator. Extensive experiments demonstrate that both
components of Tango achieve state-of-the-art results among 7B/8B-scale models:
the generator attains best-in-class performance across five competition-level
math benchmarks and four challenging out-of-domain reasoning tasks, while the
verifier leads on the ProcessBench dataset. Remarkably, both components exhibit
particularly substantial improvements on the most difficult mathematical
reasoning problems. Code is at: https://github.com/kaiwenzha/rl-tango.
Learning to Reason via Mixture-of-Thought for Logical Reasoning
Authors: name: Tong Zheng, name: Lichang Chen, name: Simeng Han, name: R. Thomas McCoy, name: Heng Huang
Published: 2025-05-21 17:59:54

Human beings naturally utilize multiple reasoning modalities to learn and
solve logical problems, i.e., different representational formats such as
natural language, code, and symbolic logic. In contrast, most existing
LLM-based approaches operate with a single reasoning modality during training,
typically natural language. Although some methods explored modality selection
or augmentation at inference time, the training process remains modality-blind,
limiting synergy among modalities. To fill in this gap, we propose
Mixture-of-Thought (MoT), a framework that enables LLMs to reason across three
complementary modalities: natural language, code, and a newly introduced
symbolic modality, truth-table, which systematically enumerates logical cases
and partially mitigates key failure modes in natural language reasoning. MoT
adopts a two-phase design: (1) self-evolving MoT training, which jointly learns
from filtered, self-generated rationales across modalities; and (2) MoT
inference, which fully leverages the synergy of three modalities to produce
better predictions. Experiments on logical reasoning benchmarks including FOLIO
and ProofWriter demonstrate that our MoT framework consistently and
significantly outperforms strong LLM baselines with single-modality
chain-of-thought approaches, achieving up to +11.7pp average accuracy gain.
Further analyses show that our MoT framework benefits both training and
inference stages; that it is particularly effective on harder logical reasoning
problems; and that different modalities contribute complementary strengths,
with truth-table reasoning helping to overcome key bottlenecks in natural
language inference.
When to Continue Thinking: Adaptive Thinking Mode Switching for
  Efficient Reasoning
Authors: name: Xiaoyun Zhang, name: Jingqing Ruan, name: Xing Ma, name: Yawen Zhu, name: Haodong Zhao, name: Hao Li, name: Jiansong Chen, name: Ke Zeng, name: Xunliang Cai
Published: 2025-05-21 11:41:39

Large reasoning models (LRMs) achieve remarkable performance via long
reasoning chains, but often incur excessive computational overhead due to
redundant reasoning, especially on simple tasks. In this work, we
systematically quantify the upper bounds of LRMs under both Long-Thinking and
No-Thinking modes, and uncover the phenomenon of "Internal Self-Recovery
Mechanism" where models implicitly supplement reasoning during answer
generation. Building on this insight, we propose Adaptive Self-Recovery
Reasoning (ASRR), a framework that suppresses unnecessary reasoning and enables
implicit recovery. By introducing accuracy-aware length reward regulation, ASRR
adaptively allocates reasoning effort according to problem difficulty,
achieving high efficiency with negligible performance sacrifice. Experiments
across multiple benchmarks and models show that, compared with GRPO, ASRR
reduces reasoning budget by up to 32.5% (1.5B) and 25.7% (7B) with minimal
accuracy loss (1.2% and 0.6% pass@1), and significantly boosts harmless rates
on safety benchmarks (up to +21.7%). Our results highlight the potential of
ASRR for enabling efficient, adaptive, and safer reasoning in LRMs.
ConvSearch-R1: Enhancing Query Reformulation for Conversational Search
  with Reasoning via Reinforcement Learning
Authors: name: Changtai Zhu, name: Siyin Wang, name: Ruijun Feng, name: Kai Song, name: Xipeng Qiu
Published: 2025-05-21 17:27:42

Conversational search systems require effective handling of context-dependent
queries that often contain ambiguity, omission, and coreference. Conversational
Query Reformulation (CQR) addresses this challenge by transforming these
queries into self-contained forms suitable for off-the-shelf retrievers.
However, existing CQR approaches suffer from two critical constraints: high
dependency on costly external supervision from human annotations or large
language models, and insufficient alignment between the rewriting model and
downstream retrievers. We present ConvSearch-R1, the first self-driven
framework that completely eliminates dependency on external rewrite supervision
by leveraging reinforcement learning to optimize reformulation directly through
retrieval signals. Our novel two-stage approach combines Self-Driven Policy
Warm-Up to address the cold-start problem through retrieval-guided
self-distillation, followed by Retrieval-Guided Reinforcement Learning with a
specially designed rank-incentive reward shaping mechanism that addresses the
sparsity issue in conventional retrieval metrics. Extensive experiments on
TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly
outperforms previous state-of-the-art methods, achieving over 10% improvement
on the challenging TopiOCQA dataset while using smaller 3B parameter models
without any external supervision.
Language Specific Knowledge: Do Models Know Better in X than in English?
Authors: name: Ishika Agarwal, name: Nimet Beyza Bozdag, name: Dilek Hakkani-Tür
Published: 2025-05-21 00:31:13

Code-switching is a common phenomenon of alternating between different
languages in the same utterance, thought, or conversation. We posit that humans
code-switch because they feel more comfortable talking about certain topics and
domains in one language than another. With the rise of knowledge-intensive
language models, we ask ourselves the next, natural question: Could models hold
more knowledge on some topics in some language X? More importantly, could we
improve reasoning by changing the language that reasoning is performed in? We
coin the term Language Specific Knowledge (LSK) to represent this phenomenon.
As ethnic cultures tend to develop alongside different languages, we employ
culture-specific datasets (that contain knowledge about cultural and social
behavioral norms). We find that language models can perform better when using
chain-of-thought reasoning in some languages other than English, sometimes even
better in low-resource languages. Paired with previous works showing that
semantic similarity does not equate to representational similarity, we
hypothesize that culturally specific texts occur more abundantly in
corresponding languages, enabling specific knowledge to occur only in specific
"expert" languages. Motivated by our initial results, we design a simple
methodology called LSKExtractor to benchmark the language-specific knowledge
present in a language model and, then, exploit it during inference. We show our
results on various models and datasets, showing an average relative improvement
of 10% in accuracy. Our research contributes to the open-source development of
language models that are inclusive and more aligned with the cultural and
linguistic contexts in which they are deployed.
IA-T2I: Internet-Augmented Text-to-Image Generation
Authors: name: Chuanhao Li, name: Jianwen Sun, name: Yukang Feng, name: Mingliang Zhai, name: Yifan Chang, name: Kaipeng Zhang
Published: 2025-05-21 17:31:49

Current text-to-image (T2I) generation models achieve promising results, but
they fail on the scenarios where the knowledge implied in the text prompt is
uncertain. For example, a T2I model released in February would struggle to
generate a suitable poster for a movie premiering in April, because the
character designs and styles are uncertain to the model. To solve this problem,
we propose an Internet-Augmented text-to-image generation (IA-T2I) framework to
compel T2I models clear about such uncertain knowledge by providing them with
reference images. Specifically, an active retrieval module is designed to
determine whether a reference image is needed based on the given text prompt; a
hierarchical image selection module is introduced to find the most suitable
image returned by an image search engine to enhance the T2I model; a
self-reflection mechanism is presented to continuously evaluate and refine the
generated image to ensure faithful alignment with the text prompt. To evaluate
the proposed framework's performance, we collect a dataset named Img-Ref-T2I,
where text prompts include three types of uncertain knowledge: (1) known but
rare. (2) unknown. (3) ambiguous. Moreover, we carefully craft a complex prompt
to guide GPT-4o in making preference evaluation, which has been shown to have
an evaluation accuracy similar to that of human preference evaluation.
Experimental results demonstrate the effectiveness of our framework,
outperforming GPT-4o by about 30% in human evaluation.
Text Generation Beyond Discrete Token Sampling
Authors: name: Yufan Zhuang, name: Liyuan Liu, name: Chandan Singh, name: Jingbo Shang, name: Jianfeng Gao
Published: 2025-05-20 18:41:46

In standard autoregressive generation, an LLM predicts the next-token
distribution, samples a discrete token, and then discards the distribution,
passing only the sampled token as new input. To preserve this distribution's
rich information, we propose Mixture of Inputs (MoI), a training-free method
for autoregressive generation. After generating a token following the standard
paradigm, we construct a new input that blends the generated discrete token
with the previously discarded token distribution. Specifically, we employ a
Bayesian estimation method that treats the token distribution as the prior, the
sampled token as the observation, and replaces the conventional one-hot vector
with the continuous posterior expectation as the new model input. MoI allows
the model to maintain a richer internal representation throughout the
generation process, resulting in improved text quality and reasoning
capabilities. On mathematical reasoning, code generation, and PhD-level QA
tasks, MoI consistently improves performance across multiple models including
QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional
training and negligible computational overhead.
dKV-Cache: The Cache for Diffusion Language Models
Authors: name: Xinyin Ma, name: Runpeng Yu, name: Gongfan Fang, name: Xinchao Wang
Published: 2025-05-21 17:32:10

Diffusion Language Models (DLMs) have been seen as a promising competitor for
autoregressive language models. However, diffusion language models have long
been constrained by slow inference. A core challenge is that their
non-autoregressive architecture and bidirectional attention preclude the
key-value cache that accelerates decoding. We address this bottleneck by
proposing a KV-cache-like mechanism, delayed KV-Cache, for the denoising
process of DLMs. Our approach is motivated by the observation that different
tokens have distinct representation dynamics throughout the diffusion process.
Accordingly, we propose a delayed and conditioned caching strategy for key and
value states. We design two complementary variants to cache key and value
step-by-step: (1) dKV-Cache-Decode, which provides almost lossless
acceleration, and even improves performance on long sequences, suggesting that
existing DLMs may under-utilise contextual information during inference. (2)
dKV-Cache-Greedy, which has aggressive caching with reduced lifespan, achieving
higher speed-ups with quadratic time complexity at the cost of some performance
degradation. dKV-Cache, in final, achieves from 2-10x speedup in inference,
largely narrowing the gap between ARs and DLMs. We evaluate our dKV-Cache on
several benchmarks, delivering acceleration across general language
understanding, mathematical, and code-generation benchmarks. Experiments
demonstrate that cache can also be used in DLMs, even in a training-free manner
from current DLMs.
lmgame-Bench: How Good are LLMs at Playing Games?
Authors: name: Lanxiang Hu, name: Mingjia Huo, name: Yuxuan Zhang, name: Haoyang Yu, name: Eric P. Xing, name: Ion Stoica, name: Tajana Rosing, name: Haojian Jin, name: Hao Zhang
Published: 2025-05-21 06:02:55

Playing video games requires perception, memory, and planning, exactly the
faculties modern large language model (LLM) agents are expected to master. We
study the major challenges in using popular video games to evaluate modern LLMs
and find that directly dropping LLMs into games cannot make an effective
evaluation, for three reasons -- brittle vision perception, prompt sensitivity,
and potential data contamination. We introduce lmgame-Bench to turn games into
reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and
narrative games delivered through a unified Gym-style API and paired with
lightweight perception and memory scaffolds, and is designed to stabilize
prompt variance and remove contamination. Across 13 leading models, we show
lmgame-Bench is challenging while still separating models well. Correlation
analysis shows that every game probes a unique blend of capabilities often
tested in isolation elsewhere. More interestingly, performing reinforcement
learning on a single game from lmgame-Bench transfers both to unseen games and
to external planning tasks. Our evaluation code is available at
https://github.com/lmgame-org/GamingAgent/lmgame-bench.
Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
Authors: name: Wei Liu, name: Ruochen Zhou, name: Yiyun Deng, name: Yuzhen Huang, name: Junteng Liu, name: Yuntian Deng, name: Yizhe Zhang, name: Junxian He
Published: 2025-05-21 15:03:26

Large Reasoning Models (LRMs) have shown remarkable capabilities in solving
complex problems through reinforcement learning (RL), particularly by
generating long reasoning traces. However, these extended outputs often exhibit
substantial redundancy, which limits the efficiency of LRMs. In this paper, we
investigate RL-based approaches to promote reasoning efficiency. Specifically,
we first present a unified framework that formulates various efficient
reasoning methods through the lens of length-based reward shaping. Building on
this perspective, we propose a novel Length-bAsed StEp Reward shaping method
(LASER), which employs a step function as the reward, controlled by a target
length. LASER surpasses previous methods, achieving a superior Pareto-optimal
balance between performance and efficiency. Next, we further extend LASER based
on two key intuitions: (1) The reasoning behavior of the model evolves during
training, necessitating reward specifications that are also adaptive and
dynamic; (2) Rather than uniformly encouraging shorter or longer chains of
thought (CoT), we posit that length-based reward shaping should be
difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries.
This approach is expected to facilitate a combination of fast and slow
thinking, leading to a better overall tradeoff. The resulting method is termed
LASER-D (Dynamic and Difficulty-aware). Experiments on
DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and
DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both
reasoning performance and response length efficiency. For instance, LASER-D and
its variant achieve a +6.1 improvement on AIME2024 while reducing token usage
by 63%. Further analysis reveals our RL-based compression produces more concise
reasoning patterns with less redundant "self-reflections". Resources are at
https://github.com/hkust-nlp/Laser.
Diffusion vs. Autoregressive Language Models: A Text Embedding
  Perspective
Authors: name: Siyue Zhang, name: Yilun Zhao, name: Liyuan Geng, name: Arman Cohan, name: Anh Tuan Luu, name: Chen Zhao
Published: 2025-05-21 02:59:14

Large language model (LLM)-based embedding models, benefiting from large
scale pre-training and post-training, have begun to surpass BERT and T5-based
models on general-purpose text embedding tasks such as document retrieval.
However, a fundamental limitation of LLM embeddings lies in the unidirectional
attention used during autoregressive pre-training, which misaligns with the
bidirectional nature of text embedding tasks. To this end, We propose adopting
diffusion language models for text embeddings, motivated by their inherent
bidirectional architecture and recent success in matching or surpassing LLMs
especially on reasoning tasks. We present the first systematic study of the
diffusion language embedding model, which outperforms the LLM-based embedding
model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval,
2% on instruction-following retrieval, and achieve competitive performance on
traditional text embedding benchmarks. Our analysis verifies that bidirectional
attention is crucial for encoding global context in long and complex text.
VARD: Efficient and Dense Fine-Tuning for Diffusion Models with
  Value-based RL
Authors: name: Fengyuan Dai, name: Zifeng Zhuang, name: Yufei Huang, name: Siteng Huang, name: Bangyan Liao, name: Donglin Wang, name: Fajie Yuan
Published: 2025-05-21 17:44:37

Diffusion models have emerged as powerful generative tools across various
domains, yet tailoring pre-trained models to exhibit specific desirable
properties remains challenging. While reinforcement learning (RL) offers a
promising solution,current methods struggle to simultaneously achieve stable,
efficient fine-tuning and support non-differentiable rewards. Furthermore,
their reliance on sparse rewards provides inadequate supervision during
intermediate steps, often resulting in suboptimal generation quality. To
address these limitations, dense and differentiable signals are required
throughout the diffusion process. Hence, we propose VAlue-based Reinforced
Diffusion (VARD): a novel approach that first learns a value function
predicting expection of rewards from intermediate states, and subsequently uses
this value function with KL regularization to provide dense supervision
throughout the generation process. Our method maintains proximity to the
pretrained model while enabling effective and stable training via
backpropagation. Experimental results demonstrate that our approach facilitates
better trajectory guidance, improves training efficiency and extends the
applicability of RL to diffusion models optimized for complex,
non-differentiable reward functions.