Orionnnn/hub-stats huggingface dataset:
Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace
  Disentanglement
Authors: name: Sekh Mainul Islam, name: Pepa Atanasova, name: Isabelle Augenstein
Published: 2025-11-03 16:15:06

Natural Language Explanations (NLEs) describe how Large Language Models
(LLMs) make decisions, drawing on both external Context Knowledge (CK) and
Parametric Knowledge (PK) stored in model weights. Understanding their
interaction is key to assessing the grounding of NLEs, yet it remains
underexplored. Prior work has largely examined only single-step generation,
typically the final answer, and has modelled PK and CK interaction only as a
binary choice in a rank-1 subspace. This overlooks richer forms of interaction,
such as complementary or supportive knowledge. We propose a novel rank-2
projection subspace that disentangles PK and CK contributions more accurately
and use it for the first multi-step analysis of knowledge interactions across
longer NLE sequences. Experiments on four QA datasets and three open-weight
instruction-tuned LLMs show that diverse knowledge interactions are poorly
represented in a rank-1 subspace but are effectively captured in our rank-2
formulation. Our multi-step analysis reveals that hallucinated NLEs align
strongly with the PK direction, context-faithful ones balance PK and CK, and
Chain-of-Thought prompting for NLEs shifts generated NLEs toward CK by reducing
PK reliance. This work provides the first framework for systematic studies of
multi-step knowledge interactions in LLMs through a richer rank-2 subspace
disentanglement. Code and data:
https://github.com/copenlu/pk-ck-knowledge-disentanglement.
Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers
Authors: name: Mohamed Eltahir, name: Ali Habibullah, name: Lama Ayash, name: Tanveer Hussain, name: Naeemullah Khan
Published: 2025-11-03 14:25:12

In the retrieval domain, candidates' fusion from heterogeneous retrievers is
a long-standing challenge, particularly for complex, multi-modal data such as
videos. While typical fusion techniques are training-free, they rely solely on
rank or score signals, disregarding candidates' representations. This work
introduces Vote-in-Context (ViC), a generalized, training-free framework that
re-thinks list-wise reranking and fusion as a zero-shot reasoning task for a
Vision-Language Model (VLM). The core insight is to serialize both content
evidence and retriever metadata directly within the VLM's prompt, allowing the
model to adaptively weigh retriever consensus against visual-linguistic
content. We demonstrate the generality of this framework by applying it to the
challenging domain of cross-modal video retrieval. To this end, we introduce
the S-Grid, a compact serialization map that represents each video as an image
grid, optionally paired with subtitles to enable list-wise reasoning over video
candidates. ViC is evaluated both as a single-list reranker, where it
dramatically improves the precision of individual retrievers, and as an
ensemble fuser, where it consistently outperforms strong baselines like
CombSUM. Across video retrieval benchmarks including ActivityNet and VATEX, the
framework establishes new state-of-the-art zero-shot retrieval performance,
demonstrating its effectiveness in handling complex visual and temporal signals
alongside text. In zero-shot settings, ViC achieves Recall@1 scores of 87.1%
(t2v) / 89.0% (v2t) on MSR-VTT and 99.6% (v2t) on VATEX, representing massive
gains of up to +40 Recall@1 over previous state-of-the-art baselines. We
present ViC as a simple, reproducible, and highly effective recipe for turning
modern VLMs into powerful zero-shot rerankers and fusers. Code and resources
are publicly available at: https://github.com/mohammad2012191/ViC
NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
Authors: name: Tim Windecker, name: Manthan Patel, name: Moritz Reuss, name: Richard Schwarzkopf, name: Cesar Cadena, name: Rudolf Lioutikov, name: Marco Hutter, name: Jonas Frey
Published: 2025-10-30 18:16:32

Vision-language models demonstrate unprecedented performance and
generalization across a wide range of tasks and scenarios. Integrating these
foundation models into robotic navigation systems opens pathways toward
building general-purpose robots. Yet, evaluating these models' navigation
capabilities remains constrained by costly real-world trials, overly simplified
simulations, and limited benchmarks. We introduce NaviTrace, a high-quality
Visual Question Answering benchmark where a model receives an instruction and
embodiment type (human, legged robot, wheeled robot, bicycle) and must output a
2D navigation trace in image space. Across 1000 scenarios and more than 3000
expert traces, we systematically evaluate eight state-of-the-art VLMs using a
newly introduced semantic-aware trace score. This metric combines Dynamic Time
Warping distance, goal endpoint error, and embodiment-conditioned penalties
derived from per-pixel semantics and correlates with human preferences. Our
evaluation reveals consistent gap to human performance caused by poor spatial
grounding and goal localization. NaviTrace establishes a scalable and
reproducible benchmark for real-world robotic navigation. The benchmark and
leaderboard can be found at
https://leggedrobotics.github.io/navitrace_webpage/.
OpenSIR: Open-Ended Self-Improving Reasoner
Authors: name: Wai-Chung Kwan, name: Joshua Ong Jun Leang, name: Pavlos Vougiouklis, name: Jeff Z. Pan, name: Marco Valentino, name: Pasquale Minervini
Published: 2025-11-01 16:08:28

Recent advances in large language model (LLM) reasoning through reinforcement
learning rely on annotated datasets for verifiable rewards, which may limit
models' ability to surpass human-level performance. While self-play offers a
promising alternative, existing approaches depend on external verifiers or
cannot learn open-endedly. We present Open-Ended Self-Improving Reasoner
(OpenSIR), a self-play framework where an LLM learns to generate and solve
novel problems by alternating teacher and student roles without external
supervision. To generate novel problems, OpenSIR optimises for both difficulty
and diversity, rewarding problems that challenge appropriately while exploring
distinct concepts, enabling open-ended mathematical discovery. Starting from a
single trivial seed problem, OpenSIR substantially improves instruction models:
Llama-3.2-3B-Instruct advances from 73.9 to 78.3 on GSM8K, and from 28.8 to
34.4 on College Math, while Gemma-2-2B-Instruct rises from 38.5 to 58.7 on
GSM8K. Our analyses reveal that OpenSIR achieves open-ended learning through
co-evolving teacher-student roles that adaptively calibrate difficulty and
drive diverse exploration, progressing autonomously from basic to advanced
mathematics.
PHUMA: Physically-Grounded Humanoid Locomotion Dataset
Authors: name: Kyungmin Lee, name: Sibeen Kim, name: Minho Park, name: Hyunseung Kim, name: Dongyoon Hwang, name: Hojoon Lee, name: Jaegul Choo
Published: 2025-10-30 08:13:12

Motion imitation is a promising approach for humanoid locomotion, enabling
agents to acquire humanlike behaviors. Existing methods typically rely on
high-quality motion capture datasets such as AMASS, but these are scarce and
expensive, limiting scalability and diversity. Recent studies attempt to scale
data collection by converting large-scale internet videos, exemplified by
Humanoid-X. However, they often introduce physical artifacts such as floating,
penetration, and foot skating, which hinder stable imitation. In response, we
introduce PHUMA, a Physically-grounded HUMAnoid locomotion dataset that
leverages human video at scale, while addressing physical artifacts through
careful data curation and physics-constrained retargeting. PHUMA enforces joint
limits, ensures ground contact, and eliminates foot skating, producing motions
that are both large-scale and physically reliable. We evaluated PHUMA in two
sets of conditions: (i) imitation of unseen motion from self-recorded test
videos and (ii) path following with pelvis-only guidance. In both cases,
PHUMA-trained policies outperform Humanoid-X and AMASS, achieving significant
gains in imitating diverse motions. The code is available at
https://davian-robotics.github.io/PHUMA.
MotionStream: Real-Time Video Generation with Interactive Motion
  Controls
Authors: name: Joonghyuk Shin, name: Zhengqi Li, name: Richard Zhang, name: Jun-Yan Zhu, name: Jaesik Park, name: Eli Schechtman, name: Xun Huang
Published: 2025-11-03 06:37:53

Current motion-conditioned video generation methods suffer from prohibitive
latency (minutes per video) and non-causal processing that prevents real-time
interaction. We present MotionStream, enabling sub-second latency with up to 29
FPS streaming generation on a single GPU. Our approach begins by augmenting a
text-to-video model with motion control, which generates high-quality videos
that adhere to the global text prompt and local motion guidance, but does not
perform inference on the fly. As such, we distill this bidirectional teacher
into a causal student through Self Forcing with Distribution Matching
Distillation, enabling real-time streaming inference. Several key challenges
arise when generating videos of long, potentially infinite time-horizons: (1)
bridging the domain gap from training on finite length and extrapolating to
infinite horizons, (2) sustaining high quality by preventing error
accumulation, and (3) maintaining fast inference, without incurring growth in
computational cost due to increasing context windows. A key to our approach is
introducing carefully designed sliding-window causal attention, combined with
attention sinks. By incorporating self-rollout with attention sinks and KV
cache rolling during training, we properly simulate inference-time
extrapolations with a fixed context window, enabling constant-speed generation
of arbitrarily long videos. Our models achieve state-of-the-art results in
motion following and video quality while being two orders of magnitude faster,
uniquely enabling infinite-length streaming. With MotionStream, users can paint
trajectories, control cameras, or transfer motion, and see results unfold in
real-time, delivering a truly interactive experience.
ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool
  Use
Authors: name: Mengjie Deng, name: Guanting Dong, name: Zhicheng Dou
Published: 2025-10-31 10:51:27

Recently, large language models (LLMs) have demonstrated remarkable
problem-solving capabilities by autonomously integrating with external tools
for collaborative reasoning. However, due to the inherently complex and diverse
nature of multimodal information, enabling multimodal large language models
(MLLMs) to flexibly and efficiently utilize external tools during reasoning
remains an underexplored challenge. In this work, we introduce ToolScope, an
agentic framework designed to unify global planning with local multimodal
perception, adopting a specialized Perceive tool to mitigates visual context
degradation in long-horizon VQA task. ToolScope comprises three primary
components: the Global Navigator, the Agentic Executor, and the Response
Synthesizer. The Global Navigator functions as a "telescope", offering
high-level strategic guidance. The Agentic Executor operates iteratively to
augment MLLM with local perception through the integration of external
tools-Search, Code, and Perceive. Finally, the Response Synthesizer
consolidates and organizes the reasoning process into a coherent, user-friendly
output. We evaluate ToolScope on four VQA benchmarks across diverse domains,
including VQA 2.0, ScienceQA, MAT-Search and MathVista. It demonstrates strong
generalization capabilities, achieving an average performance improvement of up
to +6.69% across all datasets.
UniLumos: Fast and Unified Image and Video Relighting with
  Physics-Plausible Feedback
Authors: name: Ropeway Liu, name: Hangjie Yuan, name: Bo Dong, name: Jiazheng Xing, name: Jinwang Wang, name: Rui Zhao, name: Yan Xing, name: Weihua Chen, name: Fan Wang
Published: 2025-11-03 15:41:41

Relighting is a crucial task with both practical demand and artistic value,
and recent diffusion models have shown strong potential by enabling rich and
controllable lighting effects. However, as they are typically optimized in
semantic latent space, where proximity does not guarantee physical correctness
in visual space, they often produce unrealistic results, such as overexposed
highlights, misaligned shadows, and incorrect occlusions. We address this with
UniLumos, a unified relighting framework for both images and videos that brings
RGB-space geometry feedback into a flow matching backbone. By supervising the
model with depth and normal maps extracted from its outputs, we explicitly
align lighting effects with the scene structure, enhancing physical
plausibility. Nevertheless, this feedback requires high-quality outputs for
supervision in visual space, making standard multi-step denoising
computationally expensive. To mitigate this, we employ path consistency
learning, allowing supervision to remain effective even under few-step training
regimes. To enable fine-grained relighting control and supervision, we design a
structured six-dimensional annotation protocol capturing core illumination
attributes. Building upon this, we propose LumosBench, a disentangled
attribute-level benchmark that evaluates lighting controllability via large
vision-language models, enabling automatic and interpretable assessment of
relighting precision across individual dimensions. Extensive experiments
demonstrate that UniLumos achieves state-of-the-art relighting quality with
significantly improved physical consistency, while delivering a 20x speedup for
both image and video relighting. Code is available at
https://github.com/alibaba-damo-academy/Lumos-Custom.
Actial: Activate Spatial Reasoning Ability of Multimodal Large Language
  Models
Authors: name: Xiaoyu Zhan, name: Wenxuan Huang, name: Hao Sun, name: Xinyu Fu, name: Changfeng Ma, name: Shaosheng Cao, name: Bohan Jia, name: Shaohui Lin, name: Zhenfei Yin, name: Lei Bai, name: Wanli Ouyang, name: Yuanqi Li, name: Jie Guo, name: Yanwen Guo
Published: 2025-11-03 14:27:00

Recent advances in Multimodal Large Language Models (MLLMs) have
significantly improved 2D visual understanding, prompting interest in their
application to complex 3D reasoning tasks. However, it remains unclear whether
these models can effectively capture the detailed spatial information required
for robust real-world performance, especially cross-view consistency, a key
requirement for accurate 3D reasoning. Considering this issue, we introduce
Viewpoint Learning, a task designed to evaluate and improve the spatial
reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset,
consisting of 100K object-centric image pairs with diverse viewpoints and
corresponding question-answer pairs. Our approach employs a two-stage
fine-tuning strategy: first, foundational knowledge is injected to the baseline
MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in
significant improvements across multiple tasks; second, generalization is
enhanced through Reinforcement Learning using the Group Relative Policy
Optimization (GRPO) algorithm on a broader set of questions. Additionally, we
introduce a hybrid cold-start initialization method designed to simultaneously
learn viewpoint representations and maintain coherent reasoning thinking.
Experimental results show that our approach significantly activates the spatial
reasoning ability of MLLM, improving performance on both in-domain and
out-of-domain reasoning tasks. Our findings highlight the value of developing
foundational spatial skills in MLLMs, supporting future progress in robotics,
autonomous systems, and 3D scene understanding.
Do Vision-Language Models Measure Up? Benchmarking Visual Measurement
  Reading with MeasureBench
Authors: name: Fenfen Lin, name: Yesheng Liu, name: Haiyu Xu, name: Chen Yue, name: Zheqi He, name: Mingxuan Zhao, name: Miguel Hu Chen, name: Jiakang Liu, name: JG Yao, name: Xi Yang
Published: 2025-10-30 17:20:51

Reading measurement instruments is effortless for humans and requires
relatively little domain expertise, yet it remains surprisingly challenging for
current vision-language models (VLMs) as we find in preliminary evaluation. In
this work, we introduce MeasureBench, a benchmark on visual measurement reading
covering both real-world and synthesized images of various types of
measurements, along with an extensible pipeline for data synthesis. Our
pipeline procedurally generates a specified type of gauge with controllable
visual appearance, enabling scalable variation in key details such as pointers,
scales, fonts, lighting, and clutter. Evaluation on popular proprietary and
open-weight VLMs shows that even the strongest frontier VLMs struggle
measurement reading in general. A consistent failure mode is indicator
localization: models can read digits or labels but misidentify the key
positions of pointers or alignments, leading to big numeric errors despite
plausible textual reasoning. We have also conducted preliminary experiments
with reinforcement learning over synthetic data, and find encouraging results
on in-domain synthetic subset but less promising for real-world images. Our
analysis highlights a fundamental limitation of current VLMs in fine-grained
spatial grounding. We hope this resource can help future advances on visually
grounded numeracy and precise spatial perception of VLMs, bridging the gap
between recognizing numbers and measuring the world.
ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal
  Generation
Authors: name: Yongyuan Liang, name: Wei Chow, name: Feng Li, name: Ziqiao Ma, name: Xiyao Wang, name: Jiageng Mao, name: Jiuhai Chen, name: Jiatao Gu, name: Yue Wang, name: Furong Huang
Published: 2025-11-03 02:27:46

Unified multimodal models (UMMs) have emerged as a powerful paradigm for
seamlessly unifying text and image understanding and generation. However,
prevailing evaluations treat these abilities in isolation, such that tasks with
multimodal inputs and outputs are scored primarily through unimodal reasoning,
i.e., textual benchmarks emphasize language-based reasoning, while visual
benchmarks emphasize reasoning outcomes manifested in the pixels. We introduce
ROVER to address this pressing need to test reciprocal cross-modal reasoning,
the use of one modality to guide, verify, or refine outputs in the other, an
ability central to the vision of unified multimodal intelligence. ROVER is a
human-annotated benchmark that explicitly targets reciprocal cross-modal
reasoning, which contains 1312 tasks grounded in 1876 images, spanning two
complementary settings. Verbally-augmented reasoning for visual generation
evaluates whether models can use verbal prompts and reasoning chains to guide
faithful image synthesis. Visually-augmented reasoning for verbal generation
evaluates whether models can generate intermediate visualizations that
strengthen their own reasoning processes for question answering. Experiments on
17 unified models reveal two key findings: (i) Cross-modal reasoning determines
visual generation quality, with interleaved models significantly outperforming
non-interleaved ones; notably, combining strong unimodal models fails to
achieve comparable reasoning. (ii) Models show dissociation between physical
and symbolic reasoning: they succeed at interpreting perceptual concepts
literally but fail to construct visual abstractions for symbolic tasks, where
faulty reasoning harms performance. These results highlight reciprocal
cross-modal reasoning as a critical frontier for enabling true omnimodal
generation.
UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings
Authors: name: Zhibin Lan, name: Liqiang Niu, name: Fandong Meng, name: Jie Zhou, name: Jinsong Su
Published: 2025-11-01 05:04:23

The remarkable success of multimodal large language models (MLLMs) has driven
advances in multimodal embeddings, yet existing models remain inherently
discriminative, limiting their ability to benefit from reasoning-driven
generation paradigm. In this work, we pioneer the exploration of generative
embeddings, unifying embedding tasks within a generative paradigm. We propose
UME-R1, a universal multimodal embedding framework consisting of a two-stage
training strategy: a cold-start supervised fine-tuning equips the model with
reasoning capabilities and enables it to generate both discriminative and
generative embeddings; a subsequent reinforcement learning enhances reasoning
and further optimizes generative embedding quality. This pioneering work
reveals four key insights: 1) generative embeddings unlock substantial
performance gains over conventional discriminative embeddings by leveraging the
powerful generative reasoning capabilities of MLLMs; 2) discriminative and
generative embeddings are complementary, whose combined oracle performance far
exceeding that of either alone; 3) RL can effectively enhance generative
embeddings, establishing a scalable optimization paradigm.; 4) repeated
sampling at inference boosts downstream task coverage (pass@k), highlighting
the inference-time scalability potential of generative embeddings. Evaluated on
the MMEB-V2 benchmark across 78 tasks spanning video, image, and visual
documents, UME-R1 significantly outperforms conventional discriminative
embedding models and offers a foundation for more interpretable,
reasoning-driven generative multimodal embeddings. Our code, models, and
datasets will be publicly available at https://github.com/XMUDeepLIT/UME-R1.
Data-Efficient RLVR via Off-Policy Influence Guidance
Authors: name: Erle Zhu, name: Dazhi Jiang, name: Yuan Wang, name: Xujun Li, name: Jiale Cheng, name: Yuxian Gu, name: Yilin Niu, name: Aohan Zeng, name: Jie Tang, name: Minlie Huang, name: Hongning Wang
Published: 2025-10-30 13:40:52

Data selection is a critical aspect of Reinforcement Learning with Verifiable
Rewards (RLVR) for enhancing the reasoning capabilities of large language
models (LLMs). Current data selection methods are largely heuristic-based,
lacking theoretical guarantees and generalizability. This work proposes a
theoretically-grounded approach using influence functions to estimate the
contribution of each data point to the learning objective. To overcome the
prohibitive computational cost of policy rollouts required for online influence
estimation, we introduce an off-policy influence estimation method that
efficiently approximates data influence using pre-collected offline
trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we
employ sparse random projection to reduce dimensionality and improve storage
and computation efficiency. Leveraging these techniques, we develop
Curriculum RL with Off-Policy
Influence guidance (CROPI), a multi-stage RL framework that
iteratively selects the most influential data for the current policy.
Experiments on models up to 7B parameters demonstrate that CROPI significantly
accelerates training. On a 1.5B model, it achieves a 2.66x step-level
acceleration while using only 10\% of the data per stage compared to
full-dataset training. Our results highlight the substantial potential of
influence-based data selection for efficient RLVR.