Orionnnn/hub-stats huggingface dataset:
OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for
  Vision Language Models
Authors: name: Mengdi Jia, name: Zekun Qi, name: Shaochen Zhang, name: Wenyao Zhang, name: Xinqiang Yu, name: Jiawei He, name: He Wang, name: Li Yi
Published: 2025-06-03 17:58:29

Spatial reasoning is a key aspect of cognitive psychology and remains a major
bottleneck for current vision-language models (VLMs). While extensive research
has aimed to evaluate or improve VLMs' understanding of basic spatial
relations, such as distinguishing left from right, near from far, and object
counting, these tasks represent only the most fundamental level of spatial
reasoning. In this work, we introduce OmniSpatial, a comprehensive and
challenging benchmark for spatial reasoning, grounded in cognitive psychology.
OmniSpatial covers four major categories: dynamic reasoning, complex spatial
logic, spatial interaction, and perspective-taking, with 50 fine-grained
subcategories. Through Internet data crawling and careful manual annotation, we
construct over 1.5K question-answer pairs. Extensive experiments show that both
open- and closed-source VLMs, as well as existing reasoning and spatial
understanding models, exhibit significant limitations in comprehensive spatial
understanding. We further analyze failure cases and propose potential
directions for future research.
ORV: 4D Occupancy-centric Robot Video Generation
Authors: name: Xiuyu Yang, name: Bohan Li, name: Shaocong Xu, name: Nan Wang, name: Chongjie Ye, name: Zhaoxi Chen, name: Minghan Qin, name: Yikang Ding, name: Xin Jin, name: Hang Zhao, name: Hao Zhao
Published: 2025-06-03 17:00:32

Acquiring real-world robotic simulation data through teleoperation is
notoriously time-consuming and labor-intensive. Recently, action-driven
generative models have gained widespread adoption in robot learning and
simulation, as they eliminate safety concerns and reduce maintenance efforts.
However, the action sequences used in these methods often result in limited
control precision and poor generalization due to their globally coarse
alignment. To address these limitations, we propose ORV, an Occupancy-centric
Robot Video generation framework, which utilizes 4D semantic occupancy
sequences as a fine-grained representation to provide more accurate semantic
and geometric guidance for video generation. By leveraging occupancy-based
representations, ORV enables seamless translation of simulation data into
photorealistic robot videos, while ensuring high temporal consistency and
precise controllability. Furthermore, our framework supports the simultaneous
generation of multi-view videos of robot gripping operations - an important
capability for downstream robotic learning tasks. Extensive experimental
results demonstrate that ORV consistently outperforms existing baseline methods
across various datasets and sub-tasks. Demo, Code and Model:
https://orangesodahub.github.io/ORV
Revisiting LRP: Positional Attribution as the Missing Ingredient for
  Transformer Explainability
Authors: name: Yarden Bakish, name: Itamar Zimerman, name: Hila Chefer, name: Lior Wolf
Published: 2025-06-02 18:07:55

The development of effective explainability tools for Transformers is a
crucial pursuit in deep learning research. One of the most promising approaches
in this domain is Layer-wise Relevance Propagation (LRP), which propagates
relevance scores backward through the network to the input space by
redistributing activation values based on predefined rules. However, existing
LRP-based methods for Transformer explainability entirely overlook a critical
component of the Transformer architecture: its positional encoding (PE),
resulting in violation of the conservation property, and the loss of an
important and unique type of relevance, which is also associated with
structural and positional features. To address this limitation, we reformulate
the input space for Transformer explainability as a set of position-token
pairs. This allows us to propose specialized theoretically-grounded LRP rules
designed to propagate attributions across various positional encoding methods,
including Rotary, Learnable, and Absolute PE. Extensive experiments with both
fine-tuned classifiers and zero-shot foundation models, such as LLaMA 3,
demonstrate that our method significantly outperforms the state-of-the-art in
both vision and NLP explainability tasks. Our code is publicly available.
ReFoCUS: Reinforcement-guided Frame Optimization for Contextual
  Understanding
Authors: name: Hosu Lee, name: Junho Kim, name: Hyunjun Kim, name: Yong Man Ro
Published: 2025-06-02 03:08:07

Recent progress in Large Multi-modal Models (LMMs) has enabled effective
vision-language reasoning, yet the ability to understand video content remains
constrained by suboptimal frame selection strategies. Existing approaches often
rely on static heuristics or external retrieval modules to feed frame
information into video-LLMs, which may fail to provide the query-relevant
information. In this work, we introduce ReFoCUS (Reinforcement-guided Frame
Optimization for Contextual UnderStanding), a novel frame-level policy
optimization framework that shifts the optimization target from textual
responses to visual input selection. ReFoCUS learns a frame selection policy
via reinforcement learning, using reward signals derived from a reference LMM
to reflect the model's intrinsic preferences for frames that best support
temporally grounded responses. To efficiently explore the large combinatorial
frame space, we employ an autoregressive, conditional selection architecture
that ensures temporal coherence while reducing complexity. Our approach does
not require explicit supervision at the frame-level and consistently improves
reasoning performance across multiple video QA benchmarks, highlighting the
benefits of aligning frame selection with model-internal utility.
Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate
  Video Diffusion Transformers
Authors: name: Pengtao Chen, name: Xianfang Zeng, name: Maosen Zhao, name: Peng Ye, name: Mingzhu Shen, name: Wei Cheng, name: Gang Yu, name: Tao Chen
Published: 2025-06-03 16:42:37

While Diffusion Transformers (DiTs) have achieved breakthroughs in video
generation, this long sequence generation task remains constrained by the
quadratic complexity of attention mechanisms, resulting in significant
inference latency. Through detailed analysis of attention maps in Video
Diffusion Transformer (vDiT), we identify three recurring sparsity patterns:
diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\%
attention heads can be skipped. Crucially, these patterns exhibit strong
layer-depth and head-position correlations but show limited dependence on the
input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity
acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels
that replace dense attention with computationally efficient implementations for
each identified sparsity pattern. 2) An offline sparse diffusion search
algorithm that selects the optimal sparse computation strategy per layer and
head via hardware-aware cost modeling. After determining the optimal
configuration, we fuse heads within the same layer that share the same
attention strategy, enhancing inference efficiency. Integrated into
state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1),
Sparse-vDiT achieves 2.09times, 2.38times, and 1.67times theoretical
FLOP reduction, and actual inference speedups of 1.76times, 1.85times,
and 1.58times, respectively, while maintaining high visual fidelity, with
PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent
structural sparsity in vDiTs can be systematically exploited for long video
synthesis.
AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video
  Generation
Authors: name: Lu Qiu, name: Yizhuo Li, name: Yuying Ge, name: Yixiao Ge, name: Ying Shan, name: Xihui Liu
Published: 2025-06-03 17:55:18

Recent advances in AI-generated content (AIGC) have significantly accelerated
animation production. To produce engaging animations, it is essential to
generate coherent multi-shot video clips with narrative scripts and character
references. However, existing public datasets primarily focus on real-world
scenarios with global descriptions, and lack reference images for consistent
character guidance. To bridge this gap, we present AnimeShooter, a
reference-guided multi-shot animation dataset. AnimeShooter features
comprehensive hierarchical annotations and strong visual consistency across
shots through an automated pipeline. Story-level annotations provide an
overview of the narrative, including the storyline, key scenes, and main
character profiles with reference images, while shot-level annotations
decompose the story into consecutive shots, each annotated with scene,
characters, and both narrative and descriptive visual captions. Additionally, a
dedicated subset, AnimeShooter-audio, offers synchronized audio tracks for each
shot, along with audio descriptions and sound sources. To demonstrate the
effectiveness of AnimeShooter and establish a baseline for the reference-guided
multi-shot video generation task, we introduce AnimeShooterGen, which leverages
Multimodal Large Language Models (MLLMs) and video diffusion models. The
reference image and previously generated shots are first processed by MLLM to
produce representations aware of both reference and context, which are then
used as the condition for the diffusion model to decode the subsequent shot.
Experimental results show that the model trained on AnimeShooter achieves
superior cross-shot visual consistency and adherence to reference visual
guidance, which highlight the value of our dataset for coherent animated video
generation.
Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in
  Robotics
Authors: name: Dongyoung Kim, name: Sumin Park, name: Huiwon Jang, name: Jinwoo Shin, name: Jaehyung Kim, name: Younggyo Seo
Published: 2025-05-29 16:41:12

Large Vision-Language Models (LVLMs) have recently shown great promise in
advancing robotics by combining embodied reasoning with robot control. A common
approach involves training on embodied reasoning tasks related to robot control
using Supervised Fine-Tuning (SFT). However, SFT datasets are often
heuristically constructed and not explicitly optimized for improving robot
control. Furthermore, SFT often leads to issues such as catastrophic forgetting
and reduced generalization performance. To address these limitations, we
introduce Robot-R1, a novel framework that leverages reinforcement learning to
enhance embodied reasoning specifically for robot control. Robot-R1 learns to
predict the next keypoint state required for task completion, conditioned on
the current scene image and environment metadata derived from expert
demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples
reasoning-based responses and reinforces those that lead to more accurate
predictions. Our experiments show that models trained with Robot-R1 outperform
SFT methods on embodied reasoning tasks. Despite having only 7B parameters,
Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action
control, such as spatial and primitive movement reasoning.
Visual Embodied Brain: Let Multimodal Large Language Models See, Think,
  and Control in Spaces
Authors: name: Gen Luo, name: Ganlin Yang, name: Ziyang Gong, name: Guanzhou Chen, name: Haonan Duan, name: Erfei Cui, name: Ronglei Tong, name: Zhi Hou, name: Tianyi Zhang, name: Zhe Chen, name: Shenglong Ye, name: Lewei Lu, name: Jingbo Wang, name: Wenhai Wang, name: Jifeng Dai, name: Yu Qiao, name: Rongrong Ji, name: Xizhou Zhu
Published: 2025-05-30 18:00:34

The remarkable progress of Multimodal Large Language Models (MLLMs) has
attracted increasing attention to extend them to physical entities like legged
robot. This typically requires MLLMs to not only grasp multimodal understanding
abilities, but also integrate visual-spatial reasoning and physical interaction
capabilities. Nevertheless,existing methods struggle to unify these
capabilities due to their fundamental differences.In this paper, we present the
Visual Embodied Brain (VeBrain), a unified framework for perception, reasoning,
and control in real world. VeBrain reformulates robotic control into common
text-based MLLM tasks in the 2D visual space, thus unifying the objectives and
mapping spaces of different tasks. Then, a novel robotic adapter is proposed to
convert textual control signals from MLLMs to motion policies of real robots.
From the data perspective, we further introduce VeBrain-600k, a high-quality
instruction dataset encompassing various capabilities of VeBrain. In
VeBrain-600k, we take hundreds of hours to collect, curate and annotate the
data, and adopt multimodal chain-of-thought(CoT) to mix the different
capabilities into a single conversation. Extensive experiments on 13 multimodal
benchmarks and 5 spatial intelligence benchmarks demonstrate the superior
performance of VeBrain to existing MLLMs like Qwen2.5-VL. When deployed to
legged robots and robotic arms, VeBrain shows strong adaptability, flexibility,
and compositional capabilities compared to existing methods. For example,
compared to Qwen2.5-VL, VeBrain not only achieves substantial gains on MMVet by
+5.6%, but also excels in legged robot tasks with +50% average gains.
PCoreSet: Effective Active Learning through Knowledge Distillation from
  Vision-Language Models
Authors: name: Seongjae Kang, name: Dong Bok Lee, name: Hyungjoon Jang, name: Dongseop Kim, name: Sung Ju Hwang
Published: 2025-06-01 08:54:37

Knowledge distillation (KD) is a widely used framework for training compact,
task-specific models by leveraging the knowledge of teacher models. However,
its application to active learning (AL), which aims to minimize annotation
costs through iterative sample selection, remains underexplored. This gap stems
from the fact that KD typically assumes access to sufficient labeled data,
whereas AL operates in data-scarce scenarios where task-specific teacher models
are often unavailable. In this paper, we introduce ActiveKD, a framework that
integrates AL with KD by leveraging the zero- and few-shot capabilities of
large vision-language models (VLMs). A key aspect of ActiveKD is the structured
prediction bias of VLMs -- i.e., their predictions form clusters in the
probability space. We regard this structure as an inductive bias of the teacher
model, capturing generalizable output patterns beneficial to student learning.
To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection
strategy that maximizes coverage in the probability space rather than the
feature space. PCoreSet strategically selects categorically diverse unlabeled
samples, facilitating more efficient transfer of teacher knowledge under
limited annotation budgets. Evaluations on 11 datasets show that PCoreSet
consistently outperforms existing selection methods within the ActiveKD
framework, advancing research at the intersection of AL and KD.
Accelerating Diffusion LLMs via Adaptive Parallel Decoding
Authors: name: Daniel Israel, name: Guy Van den Broeck, name: Aditya Grover
Published: 2025-05-31 06:10:10

The generation speed of LLMs are bottlenecked by autoregressive decoding,
where tokens are predicted sequentially one by one. Alternatively, diffusion
large language models (dLLMs) theoretically allow for parallel token
generation, but in practice struggle to achieve the speed of autoregressive
models without significantly sacrificing quality. We therefore introduce
adaptive parallel decoding (APD), a novel method that dynamically adjusts the
number of tokens sampled in parallel. We achieve this by defining a
multiplicative mixture between the dLLM marginal probabilities and the joint
probability of sequences under a small auxiliary autoregressive model. This
inverts the standard setup of speculative decoding, where the goal is to sample
from a large autoregressive verifier by drafting from a smaller model. We
further optimize APD by enabling KV caching and limiting the size of the masked
input. Altogether, our method puts forward three tunable parameters to flexibly
tradeoff throughput and quality. We show that APD provides markedly higher
throughput with minimal quality degradations on downstream benchmarks.
FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video
  Generation
Authors: name: Ariel Shaulov, name: Itay Hazan, name: Lior Wolf, name: Hila Chefer
Published: 2025-06-01 19:55:33

Text-to-video diffusion models are notoriously limited in their ability to
model temporal aspects such as motion, physics, and dynamic interactions.
Existing approaches address this limitation by retraining the model or
introducing external conditioning signals to enforce temporal consistency. In
this work, we explore whether a meaningful temporal representation can be
extracted directly from the predictions of a pre-trained model without any
additional training or auxiliary inputs. We introduce FlowMo, a novel
training-free guidance method that enhances motion coherence using only the
model's own predictions in each diffusion step. FlowMo first derives an
appearance-debiased temporal representation by measuring the distance between
latents corresponding to consecutive frames. This highlights the implicit
temporal structure predicted by the model. It then estimates motion coherence
by measuring the patch-wise variance across the temporal dimension and guides
the model to reduce this variance dynamically during sampling. Extensive
experiments across multiple text-to-video models demonstrate that FlowMo
significantly improves motion coherence without sacrificing visual quality or
prompt alignment, offering an effective plug-and-play solution for enhancing
the temporal fidelity of pre-trained video diffusion models.
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Authors: name: Qianhui Wu, name: Kanzhi Cheng, name: Rui Yang, name: Chaoyun Zhang, name: Jianwei Yang, name: Huiqiang Jiang, name: Jian Mu, name: Baolin Peng, name: Bo Qiao, name: Reuben Tan, name: Si Qin, name: Lars Liden, name: Qingwei Lin, name: Huan Zhang, name: Tong Zhang, name: Jianbing Zhang, name: Dongmei Zhang, name: Jianfeng Gao
Published: 2025-06-03 17:59:08

One of the principal challenges in building VLM-powered GUI agents is visual
grounding, i.e., localizing the appropriate screen region for action execution
based on both the visual content and the textual plans. Most existing work
formulates this as a text-based coordinate generation task. However, these
approaches suffer from several limitations: weak spatial-semantic alignment,
inability to handle ambiguous supervision targets, and a mismatch between the
dense nature of screen coordinates and the coarse, patch-level granularity of
visual features extracted by models like Vision Transformers. In this paper, we
propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its
core, GUI-Actor introduces an attention-based action head that learns to align
a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the
model to propose one or more action regions in a single forward pass. In line
with this, we further design a grounding verifier to evaluate and select the
most plausible action region from the candidates proposed for action execution.
Extensive experiments show that GUI-Actor outperforms prior state-of-the-art
methods on multiple GUI action grounding benchmarks, with improved
generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B
even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7
with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by
incorporating the verifier, we find that fine-tuning only the newly introduced
action head (~100M parameters for 7B model) while keeping the VLM backbone
frozen is sufficient to achieve performance comparable to previous
state-of-the-art models, highlighting that GUI-Actor can endow the underlying
VLM with effective grounding capabilities without compromising its
general-purpose strengths.
DINGO: Constrained Inference for Diffusion LLMs
Authors: name: Tarun Suresh, name: Debangshu Banerjee, name: Shubham Ugare, name: Sasa Misailovic, name: Gagandeep Singh
Published: 2025-05-29 04:04:54

Diffusion LLMs have emerged as a promising alternative to conventional
autoregressive LLMs, offering significant potential for improved runtime
efficiency. However, existing diffusion models lack the ability to provably
enforce user-specified formal constraints, such as regular expressions, which
makes them unreliable for tasks that require structured outputs, such as
fixed-schema JSON generation. Unlike autoregressive models that generate tokens
sequentially, diffusion LLMs predict a block of tokens in parallel. This
parallelism makes traditional constrained decoding algorithms, which are
designed for sequential token prediction, ineffective at preserving the true
output distribution. To address this limitation, we propose DINGO, a dynamic
programming-based constrained decoding strategy that is both efficient and
provably distribution-preserving. DINGO enables sampling of output strings with
the highest probability under the model's predicted distribution, while
strictly satisfying any user-specified regular expression. On standard symbolic
math and JSON generation benchmarks, DINGO achieves up to a 68 percentage point
improvement over unconstrained inference
LumosFlow: Motion-Guided Long Video Generation
Authors: name: Jiahao Chen, name: Hangjie Yuan, name: Yichen Qian, name: Jingyun Liang, name: Jiazheng Xing, name: Pengwei Liu, name: Weihua Chen, name: Fan Wang, name: Bing Su
Published: 2025-06-03 06:25:00

Long video generation has gained increasing attention due to its widespread
applications in fields such as entertainment and simulation. Despite advances,
synthesizing temporally coherent and visually compelling long sequences remains
a formidable challenge. Conventional approaches often synthesize long videos by
sequentially generating and concatenating short clips, or generating key frames
and then interpolate the intermediate frames in a hierarchical manner. However,
both of them still remain significant challenges, leading to issues such as
temporal repetition or unnatural transitions. In this paper, we revisit the
hierarchical long video generation pipeline and introduce LumosFlow, a
framework introduce motion guidance explicitly. Specifically, we first employ
the Large Motion Text-to-Video Diffusion Model (LMTV-DM) to generate key frames
with larger motion intervals, thereby ensuring content diversity in the
generated long videos. Given the complexity of interpolating contextual
transitions between key frames, we further decompose the intermediate frame
interpolation into motion generation and post-hoc refinement. For each pair of
key frames, the Latent Optical Flow Diffusion Model (LOF-DM) synthesizes
complex and large-motion optical flows, while MotionControlNet subsequently
refines the warped results to enhance quality and guide intermediate frame
generation. Compared with traditional video frame interpolation, we achieve 15x
interpolation, ensuring reasonable and continuous motion between adjacent
frames. Experiments show that our method can generate long videos with
consistent motion and appearance. Code and models will be made publicly
available upon acceptance. Our project page:
https://jiahaochen1.github.io/LumosFlow/