Orionnnn/hub-stats huggingface dataset:
Vid2World: Crafting Video Diffusion Models to Interactive World Models
Authors: name: Siqiao Huang, name: Jialong Wu, name: Qixing Zhou, name: Shangchen Miao, name: Mingsheng Long
Published: 2025-05-20 13:41:45

World models, which predict transitions based on history observation and
action sequences, have shown great promise in improving data efficiency for
sequential decision making. However, existing world models often require
extensive domain-specific training and still produce low-fidelity, coarse
predictions, limiting their applicability in complex environments. In contrast,
video diffusion models trained on large, internet-scale datasets have
demonstrated impressive capabilities in generating high-quality videos that
capture diverse real-world dynamics. In this work, we present Vid2World, a
general approach for leveraging and transferring pre-trained video diffusion
models into interactive world models. To bridge the gap, Vid2World performs
casualization of a pre-trained video diffusion model by crafting its
architecture and training objective to enable autoregressive generation.
Furthermore, it introduces a causal action guidance mechanism to enhance action
controllability in the resulting interactive world model. Extensive experiments
in robot manipulation and game simulation domains show that our method offers a
scalable and effective approach for repurposing highly capable video diffusion
models to interactive world models.
UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement
  Learning
Authors: name: Sule Bai, name: Mingxing Li, name: Yong Liu, name: Jing Tang, name: Haoji Zhang, name: Lei Sun, name: Xiangxiang Chu, name: Yansong Tang
Published: 2025-05-20 11:40:43

Traditional visual grounding methods primarily focus on single-image
scenarios with simple textual references. However, extending these methods to
real-world scenarios that involve implicit and complex instructions,
particularly in conjunction with multiple images, poses significant challenges,
which is mainly due to the lack of advanced reasoning ability across diverse
multi-modal contexts. In this work, we aim to address the more practical
universal grounding task, and propose UniVG-R1, a reasoning guided multimodal
large language model (MLLM) for universal visual grounding, which enhances
reasoning capabilities through reinforcement learning (RL) combined with
cold-start data. Specifically, we first construct a high-quality
Chain-of-Thought (CoT) grounding dataset, annotated with detailed reasoning
chains, to guide the model towards correct reasoning paths via supervised
fine-tuning. Subsequently, we perform rule-based reinforcement learning to
encourage the model to identify correct reasoning chains, thereby incentivizing
its reasoning capabilities. In addition, we identify a difficulty bias arising
from the prevalence of easy samples as RL training progresses, and we propose a
difficulty-aware weight adjustment strategy to further strengthen the
performance. Experimental results demonstrate the effectiveness of UniVG-R1,
which achieves state-of-the-art performance on MIG-Bench with a 9.1%
improvement over the previous method. Furthermore, our model exhibits strong
generalizability, achieving an average improvement of 23.4% in zero-shot
performance across four image and video reasoning grounding benchmarks. The
project page can be accessed at https://amap-ml.github.io/UniVG-R1-page/.
The Hallucination Tax of Reinforcement Finetuning
Authors: name: Linxin Song, name: Taiwei Shi, name: Jieyu Zhao
Published: 2025-05-20 06:36:45

Reinforcement finetuning (RFT) has become a standard approach for enhancing
the reasoning capabilities of large language models (LLMs). However, its impact
on model trustworthiness remains underexplored. In this work, we identify and
systematically study a critical side effect of RFT, which we term the
hallucination tax: a degradation in refusal behavior causing models to produce
hallucinated answers to unanswerable questions confidently. To investigate
this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of
unanswerable math problems designed to probe models' ability to recognize an
unanswerable question by reasoning from the insufficient or ambiguous
information. Our results show that standard RFT training could reduce model
refusal rates by more than 80%, which significantly increases model's tendency
to hallucinate. We further demonstrate that incorporating just 10% SUM during
RFT substantially restores appropriate refusal behavior, with minimal accuracy
trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage
inference-time compute to reason about their own uncertainty and knowledge
boundaries, improving generalization not only to out-of-domain math problems
but also to factual question answering tasks.
Visual Instruction Bottleneck Tuning
Authors: name: Changdae Oh, name: Jiatong Li, name: Shawn Im, name: Yixuan Li
Published: 2025-05-20 05:24:53

Despite widespread adoption, multimodal large language models (MLLMs) suffer
performance degradation when encountering unfamiliar queries under distribution
shifts. Existing methods to improve MLLM generalization typically require
either more instruction data or larger advanced model architectures, both of
which incur non-trivial human labor or computational costs. In this work, we
take an alternative approach to enhance the robustness of MLLMs under
distribution shifts, from a representation learning perspective. Inspired by
the information bottleneck (IB) principle, we derive a variational lower bound
of the IB for MLLMs and devise a practical implementation, Visual Instruction
Bottleneck Tuning (Vittle). We then provide a theoretical justification of
Vittle by revealing its connection to an information-theoretic robustness
metric of MLLM. Empirical validation of three MLLMs on open-ended and
closed-form question answering and object hallucination detection tasks over 45
datasets, including 30 shift scenarios, demonstrates that Vittle consistently
improves the MLLM's robustness under shifts by pursuing the learning of a
minimal sufficient representation.
Void in Language Models
Authors: name: Mani Shemiranifar
Published: 2025-05-20 15:01:56

Despite advances in transformer-based language models (LMs), a fundamental
question remains largely unanswered: Are all layers activated during inference?
We investigate this question by detecting unactivated layers (which we refer to
as Voids) using a non-trainable and parameter-free adaptive computation method
called L2 Adaptive Computation (LAC). We adapt LAC from its original
efficiency-focused application to trace activated layers during inference. This
method monitors changes in the L2-norm of activations to identify voids. We
analyze layer activation in instruction-tuned LMs across two phases: Prompt
Processing (PP), where we trace activated layers for each token in the input
prompts, and Response Generation (RG), where we trace activated layers for each
generated token. We further demonstrate that distinct layers are activated
during these two phases. To show the effectiveness of our method, we evaluated
three distinct instruction-tuned LMs from the Llama, Mistral, and Qwen families
on three benchmarks: MMLU, GPQA Diamond, and BoolQ. For example, on MMLU with a
zero-shot setting, skipping voids in Qwen2.5-7B-Instruct resulted in an
improvement from 69.24 to 71.29 while the model uses only 30% of the layers.
Similarly, Mistral-7B-Instruct-v0.3 on GPQA Diamond improved from 13.88 to
18.36 when using 70% of the layers during both the PP and RG phases. These
results show that not all layers contribute equally during inference, and that
selectively skipping most of them can improve the performance of models on
certain tasks.
GeoRanker: Distance-Aware Ranking for Worldwide Image Geolocalization
Authors: name: Pengyue Jia, name: Seongheon Park, name: Song Gao, name: Xiangyu Zhao, name: Yixuan Li
Published: 2025-05-19 21:04:46

Worldwide image geolocalization-the task of predicting GPS coordinates from
images taken anywhere on Earth-poses a fundamental challenge due to the vast
diversity in visual content across regions. While recent approaches adopt a
two-stage pipeline of retrieving candidates and selecting the best match, they
typically rely on simplistic similarity heuristics and point-wise supervision,
failing to model spatial relationships among candidates. In this paper, we
propose GeoRanker, a distance-aware ranking framework that leverages large
vision-language models to jointly encode query-candidate interactions and
predict geographic proximity. In addition, we introduce a multi-order distance
loss that ranks both absolute and relative distances, enabling the model to
reason over structured spatial relationships. To support this, we curate
GeoRanking, the first dataset explicitly designed for geographic ranking tasks
with multimodal candidate information. GeoRanker achieves state-of-the-art
results on two well-established benchmarks (IM2GPS3K and YFCC4K), significantly
outperforming current best methods.
CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM
  APIs
Authors: name: Guoheng Sun, name: Ziyao Wang, name: Bowei Tian, name: Meng Liu, name: Zheyu Shen, name: Shwai He, name: Yexiao He, name: Wanghao Ye, name: Yiting Wang, name: Ang Li
Published: 2025-05-19 23:39:23

As post-training techniques evolve, large language models (LLMs) are
increasingly augmented with structured multi-step reasoning abilities, often
optimized through reinforcement learning. These reasoning-enhanced models
outperform standard LLMs on complex tasks and now underpin many commercial LLM
APIs. However, to protect proprietary behavior and reduce verbosity, providers
typically conceal the reasoning traces while returning only the final answer.
This opacity introduces a critical transparency gap: users are billed for
invisible reasoning tokens, which often account for the majority of the cost,
yet have no means to verify their authenticity. This opens the door to token
count inflation, where providers may overreport token usage or inject
synthetic, low-effort tokens to inflate charges. To address this issue, we
propose CoIn, a verification framework that audits both the quantity and
semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from
token embedding fingerprints to check token counts, and uses embedding-based
relevance matching to detect fabricated reasoning content. Experiments
demonstrate that CoIn, when deployed as a trusted third-party auditor, can
effectively detect token count inflation with a success rate reaching up to
94.7%, showing the strong ability to restore billing transparency in opaque LLM
services. The dataset and code are available at
https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.
Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values
  Prioritization with AIRiskDilemmas
Authors: name: Yu Ying Chiu, name: Zhilin Wang, name: Sharan Maiya, name: Yejin Choi, name: Kyle Fish, name: Sydney Levine, name: Evan Hubinger
Published: 2025-05-20 17:24:09

Detecting AI risks becomes more challenging as stronger models emerge and
find novel methods such as Alignment Faking to circumvent these detection
attempts. Inspired by how risky behaviors in humans (i.e., illegal activities
that may hurt others) are sometimes guided by strongly-held values, we believe
that identifying values within AI models can be an early warning system for
AI's risky behaviors. We create LitmusValues, an evaluation pipeline to reveal
AI models' priorities on a range of AI value classes. Then, we collect
AIRiskDilemmas, a diverse collection of dilemmas that pit values against one
another in scenarios relevant to AI safety risks such as Power Seeking. By
measuring an AI model's value prioritization using its aggregate choices, we
obtain a self-consistent set of predicted value priorities that uncover
potential risks. We show that values in LitmusValues (including seemingly
innocuous ones like Care) can predict for both seen risky behaviors in
AIRiskDilemmas and unseen risky behaviors in HarmBench.
Vox-Profile: A Speech Foundation Model Benchmark for Characterizing
  Diverse Speaker and Speech Traits
Authors: name: Tiantian Feng, name: Jihwan Lee, name: Anfeng Xu, name: Yoonjeong Lee, name: Thanathai Lertpetchpun, name: Xuan Shi, name: Helin Wang, name: Thomas Thebaud, name: Laureano Moro-Velazquez, name: Dani Byrd, name: Najim Dehak, name: Shrikanth Narayanan
Published: 2025-05-20 17:36:41

We introduce Vox-Profile, a comprehensive benchmark to characterize rich
speaker and speech traits using speech foundation models. Unlike existing works
that focus on a single dimension of speaker traits, Vox-Profile provides
holistic and multi-dimensional profiles that reflect both static speaker traits
(e.g., age, sex, accent) and dynamic speech properties (e.g., emotion, speech
flow). This benchmark is grounded in speech science and linguistics, developed
with domain experts to accurately index speaker and speech characteristics. We
report benchmark experiments using over 15 publicly available speech datasets
and several widely used speech foundation models that target various static and
dynamic speaker and speech properties. In addition to benchmark experiments, we
showcase several downstream applications supported by Vox-Profile. First, we
show that Vox-Profile can augment existing speech recognition datasets to
analyze ASR performance variability. Vox-Profile is also used as a tool to
evaluate the performance of speech generation systems. Finally, we assess the
quality of our automated profiles through comparison with human evaluation and
show convergent validity. Vox-Profile is publicly available at:
https://github.com/tiantiaf0627/vox-profile-release.
Rethinking Optimal Verification Granularity for Compute-Efficient
  Test-Time Scaling
Authors: name: Hao Mark Chen, name: Guanxi Lu, name: Yasuyuki Okoshi, name: Zhiwen Mo, name: Masato Motomura, name: Hongxiang Fan
Published: 2025-05-16 22:24:48

Test-time scaling (TTS) has proven effective in enhancing the reasoning
capabilities of large language models (LLMs). Verification plays a key role in
TTS, simultaneously influencing (1) reasoning performance and (2) compute
efficiency, due to the quality and computational cost of verification. In this
work, we challenge the conventional paradigms of verification, and make the
first attempt toward systematically investigating the impact of verification
granularity-that is, how frequently the verifier is invoked during generation,
beyond verifying only the final output or individual generation steps. To this
end, we introduce Variable Granularity Search (VG-Search), a unified algorithm
that generalizes beam search and Best-of-N sampling via a tunable granularity
parameter g. Extensive experiments with VG-Search under varying compute
budgets, generator-verifier configurations, and task attributes reveal that
dynamically selecting g can improve the compute efficiency and scaling
behavior. Building on these findings, we propose adaptive VG-Search strategies
that achieve accuracy gains of up to 3.1\% over Beam Search and 3.6\% over
Best-of-N, while reducing FLOPs by over 52\%. We will open-source the code to
support future research.
Learning to Highlight Audio by Watching Movies
Authors: name: Chao Huang, name: Ruohan Gao, name: J. M. F. Tsang, name: Jan Kurcius, name: Cagdas Bilen, name: Chenliang Xu, name: Anurag Kumar, name: Sanjeel Parekh
Published: 2025-05-17 22:03:57

Recent years have seen a significant increase in video content creation and
consumption. Crafting engaging content requires the careful curation of both
visual and audio elements. While visual cue curation, through techniques like
optimal viewpoint selection or post-editing, has been central to media
production, its natural counterpart, audio, has not undergone equivalent
advancements. This often results in a disconnect between visual and acoustic
saliency. To bridge this gap, we introduce a novel task: visually-guided
acoustic highlighting, which aims to transform audio to deliver appropriate
highlighting effects guided by the accompanying video, ultimately creating a
more harmonious audio-visual experience. We propose a flexible,
transformer-based multimodal framework to solve this task. To train our model,
we also introduce a new dataset -- the muddy mix dataset, leveraging the
meticulous audio and video crafting found in movies, which provides a form of
free supervision. We develop a pseudo-data generation process to simulate
poorly mixed audio, mimicking real-world scenarios through a three-step process
-- separation, adjustment, and remixing. Our approach consistently outperforms
several baselines in both quantitative and subjective evaluation. We also
systematically study the impact of different types of contextual guidance and
difficulty levels of the dataset. Our project page is here:
https://wikichao.github.io/VisAH/.
Towards Embodied Cognition in Robots via Spatially Grounded Synthetic
  Worlds
Authors: name: Joel Currie, name: Gioele Migno, name: Enrico Piacenti, name: Maria Elena Giannaccini, name: Patric Bach, name: Davide De Tommaso, name: Agnieszka Wykowska
Published: 2025-05-20 13:49:09

We present a conceptual framework for training Vision-Language Models (VLMs)
to perform Visual Perspective Taking (VPT), a core capability for embodied
cognition essential for Human-Robot Interaction (HRI). As a first step toward
this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse,
that enables supervised learning for spatial reasoning tasks. Each instance
includes an RGB image, a natural language description, and a ground-truth 4X4
transformation matrix representing object pose. We focus on inferring Z-axis
distance as a foundational skill, with future extensions targeting full 6
Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to
support further research. This work serves as a foundational step toward
embodied AI systems capable of spatial understanding in interactive human-robot
scenarios.
The Distracting Effect: Understanding Irrelevant Passages in RAG
Authors: name: Chen Amiraz, name: Florin Cuconasu, name: Simone Filice, name: Zohar Karnin
Published: 2025-05-11 09:25:05

A well-known issue with Retrieval Augmented Generation (RAG) is that
retrieved passages that are irrelevant to the query sometimes distract the
answer-generating LLM, causing it to provide an incorrect response. In this
paper, we shed light on this core issue and formulate the distracting effect of
a passage w.r.t. a query (and an LLM). We provide a quantifiable measure of the
distracting effect of a passage and demonstrate its robustness across LLMs.
  Our research introduces novel methods for identifying and using hard
distracting passages to improve RAG systems. By fine-tuning LLMs with these
carefully selected distracting passages, we achieve up to a 7.5% increase in
answering accuracy compared to counterparts fine-tuned on conventional RAG
datasets. Our contribution is two-fold: first, we move beyond the simple binary
classification of irrelevant passages as either completely unrelated vs.
distracting, and second, we develop and analyze multiple methods for finding
hard distracting passages. To our knowledge, no other research has provided
such a comprehensive framework for identifying and utilizing hard distracting
passages.
Dynadiff: Single-stage Decoding of Images from Continuously Evolving
  fMRI
Authors: name: Marlène Careil, name: Yohann Benchetrit, name: Jean-Rémi King
Published: 2025-05-20 16:14:37

Brain-to-image decoding has been recently propelled by the progress in
generative AI models and the availability of large ultra-high field functional
Magnetic Resonance Imaging (fMRI). However, current approaches depend on
complicated multi-stage pipelines and preprocessing steps that typically
collapse the temporal dimension of brain recordings, thereby limiting
time-resolved brain decoders. Here, we introduce Dynadiff (Dynamic Neural
Activity Diffusion for Image Reconstruction), a new single-stage diffusion
model designed for reconstructing images from dynamically evolving fMRI
recordings. Our approach offers three main contributions. First, Dynadiff
simplifies training as compared to existing approaches. Second, our model
outperforms state-of-the-art models on time-resolved fMRI signals, especially
on high-level semantic image reconstruction metrics, while remaining
competitive on preprocessed fMRI data that collapse time. Third, this approach
allows a precise characterization of the evolution of image representations in
brain activity. Overall, this work lays the foundation for time-resolved
brain-to-image decoding.
KERL: Knowledge-Enhanced Personalized Recipe Recommendation using Large
  Language Models
Authors: name: Fnu Mohbat, name: Mohammed J Zaki
Published: 2025-05-20 17:19:57

Recent advances in large language models (LLMs) and the abundance of food
data have resulted in studies to improve food understanding using LLMs. Despite
several recommendation systems utilizing LLMs and Knowledge Graphs (KGs), there
has been limited research on integrating food related KGs with LLMs. We
introduce KERL, a unified system that leverages food KGs and LLMs to provide
personalized food recommendations and generates recipes with associated
micro-nutritional information. Given a natural language question, KERL extracts
entities, retrieves subgraphs from the KG, which are then fed into the LLM as
context to select the recipes that satisfy the constraints. Next, our system
generates the cooking steps and nutritional information for each recipe. To
evaluate our approach, we also develop a benchmark dataset by curating recipe
related questions, combined with constraints and personal preferences. Through
extensive experiments, we show that our proposed KG-augmented LLM significantly
outperforms existing approaches, offering a complete and coherent solution for
food recommendation, recipe generation, and nutritional analysis. Our code and
benchmark datasets are publicly available at
https://github.com/mohbattharani/KERL.
Object-Centric Representations Improve Policy Generalization in Robot
  Manipulation
Authors: name: Alexandre Chapin, name: Bruno Machado, name: Emmanuel Dellandrea, name: Liming Chen
Published: 2025-05-16 07:06:37

Visual representations are central to the learning and generalization
capabilities of robotic manipulation policies. While existing methods rely on
global or dense features, such representations often entangle task-relevant and
irrelevant scene information, limiting robustness under distribution shifts. In
this work, we investigate object-centric representations (OCR) as a structured
alternative that segments visual input into a finished set of entities,
introducing inductive biases that align more naturally with manipulation tasks.
We benchmark a range of visual encoders-object-centric, global and dense
methods-across a suite of simulated and real-world manipulation tasks ranging
from simple to complex, and evaluate their generalization under diverse visual
conditions including changes in lighting, texture, and the presence of
distractors. Our findings reveal that OCR-based policies outperform dense and
global representations in generalization settings, even without task-specific
pretraining. These insights suggest that OCR is a promising direction for
designing visual systems that generalize effectively in dynamic, real-world
robotic environments.