Orionnnn/hub-stats huggingface dataset:
MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process
  Errors Identification
Authors: name: Zhaopan Xu, name: Pengfei Zhou, name: Jiaxin Ai, name: Wangbo Zhao, name: Kai Wang, name: Xiaojiang Peng, name: Wenqi Shao, name: Hongxun Yao, name: Kaipeng Zhang
Published: 2025-03-16 13:50:38

Reasoning is an essential capacity for large language models (LLMs) to
address complex tasks, where the identification of process errors is vital for
improving this ability. Recently, process-level reward models (PRMs) were
proposed to provide step-wise rewards that facilitate reinforcement learning
and data production during training and guide LLMs toward correct steps during
inference, thereby improving reasoning accuracy. However, existing benchmarks
of PRMs are text-based and focus on error detection, neglecting other scenarios
like reasoning search. To address this gap, we introduce MPBench, a
comprehensive, multi-task, multimodal benchmark designed to systematically
assess the effectiveness of PRMs in diverse scenarios. MPBench employs three
evaluation paradigms, each targeting a specific role of PRMs in the reasoning
process: (1) Step Correctness, which assesses the correctness of each
intermediate reasoning step; (2) Answer Aggregation, which aggregates multiple
solutions and selects the best one; and (3) Reasoning Process Search, which
guides the search for optimal reasoning steps during inference. Through these
paradigms, MPBench makes comprehensive evaluations and provides insights into
the development of multimodal PRMs.
PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for
  Multimodal Large Language Models
Authors: name: Zhaopan Xu, name: Pengfei Zhou, name: Weidong Tang, name: Jiaxin Ai, name: Wangbo Zhao, name: Xiaojiang Peng, name: Kai Wang, name: Yang You, name: Wenqi Shao, name: Hongxun Yao, name: Kaipeng Zhang
Published: 2025-03-16 15:26:20

In recent years, Multimodal Large Language Models (MLLMs) have demonstrated
remarkable advancements in tasks such as visual question answering, visual
understanding, and reasoning. However, this impressive progress relies on vast
amounts of data collected from the internet, raising significant concerns about
privacy and security. To address these issues, machine unlearning (MU) has
emerged as a promising solution, enabling the removal of specific knowledge
from an already trained model without requiring retraining from scratch.
Although MU for MLLMs has gained attention, current evaluations of its efficacy
remain incomplete, and the underlying problem is often poorly defined, which
hinders the development of strategies for creating more secure and trustworthy
systems. To bridge this gap, we introduce a benchmark, named PEBench, which
includes a dataset of personal entities and corresponding general event scenes,
designed to comprehensively assess the performance of MU for MLLMs. Through
PEBench, we aim to provide a standardized and robust framework to advance
research in secure and privacy-preserving multimodal models. We benchmarked 6
MU methods, revealing their strengths and limitations, and shedding light on
key challenges and opportunities for MU in MLLMs.
Impossible Videos
Authors: name: Zechen Bai, name: Hai Ci, name: Mike Zheng Shou
Published: 2025-03-18 16:10:24

Synthetic videos nowadays is widely used to complement data scarcity and
diversity of real-world videos. Current synthetic datasets primarily replicate
real-world scenarios, leaving impossible, counterfactual and anti-reality video
concepts underexplored. This work aims to answer two questions: 1) Can today's
video generation models effectively follow prompts to create impossible video
content? 2) Are today's video understanding models good enough for
understanding impossible videos? To this end, we introduce IPV-Bench, a novel
benchmark designed to evaluate and foster progress in video understanding and
generation. IPV-Bench is underpinned by a comprehensive taxonomy, encompassing
4 domains, 14 categories. It features diverse scenes that defy physical,
biological, geographical, or social laws. Based on the taxonomy, a prompt suite
is constructed to evaluate video generation models, challenging their prompt
following and creativity capabilities. In addition, a video benchmark is
curated to assess Video-LLMs on their ability of understanding impossible
videos, which particularly requires reasoning on temporal dynamics and world
knowledge. Comprehensive evaluations reveal limitations and insights for future
directions of video models, paving the way for next-generation video models.
Frac-Connections: Fractional Extension of Hyper-Connections
Authors: name: Defa Zhu, name: Hongzhi Huang, name: Jundong Zhou, name: Zihao Huang, name: Yutao Zeng, name: Banggu Wu, name: Qiyang Min, name: Xun Zhou
Published: 2025-03-18 10:37:50

Residual connections are central to modern deep learning architectures,
enabling the training of very deep networks by mitigating gradient vanishing.
Hyper-Connections recently generalized residual connections by introducing
multiple connection strengths at different depths, thereby addressing the
seesaw effect between gradient vanishing and representation collapse. However,
Hyper-Connections increase memory access costs by expanding the width of hidden
states. In this paper, we propose Frac-Connections, a novel approach that
divides hidden states into multiple parts rather than expanding their width.
Frac-Connections retain partial benefits of Hyper-Connections while reducing
memory consumption. To validate their effectiveness, we conduct large-scale
experiments on language tasks, with the largest being a 7B MoE model trained on
up to 3T tokens, demonstrating that Frac-Connections significantly outperform
residual connections.
RoCo-Sim: Enhancing Roadside Collaborative Perception through Foreground
  Simulation
Authors: name: Yuwen Du, name: Anning Hu, name: Zichen Chao, name: Yifan Lu, name: Junhao Ge, name: Genjia Liu, name: Weitao Wu, name: Lanjun Wang, name: Siheng Chen
Published: 2025-03-13 14:33:42

Roadside Collaborative Perception refers to a system where multiple roadside
units collaborate to pool their perceptual data, assisting vehicles in
enhancing their environmental awareness. Existing roadside perception methods
concentrate on model design but overlook data issues like calibration errors,
sparse information, and multi-view consistency, leading to poor performance on
recent published datasets. To significantly enhance roadside collaborative
perception and address critical data issues, we present the first simulation
framework RoCo-Sim for road-side collaborative perception. RoCo-Sim is capable
of generating diverse, multi-view consistent simulated roadside data through
dynamic foreground editing and full-scene style transfer of a single image.
RoCo-Sim consists of four components: (1) Camera Extrinsic Optimization ensures
accurate 3D to 2D projection for roadside cameras; (2) A novel Multi-View
Occlusion-Aware Sampler (MOAS) determines the placement of diverse digital
assets within 3D space; (3) DepthSAM innovatively models foreground-background
relationships from single-frame fixed-view images, ensuring multi-view
consistency of foreground; and (4) Scalable Post-Processing Toolkit generates
more realistic and enriched scenes through style transfer and other
enhancements. RoCo-Sim significantly improves roadside 3D object detection,
outperforming SOTA methods by 83.74 on Rcooper-Intersection and 83.12 on
TUMTraf-V2X for AP70. RoCo-Sim fills a critical gap in roadside perception
simulation. Code and pre-trained models will be released soon:
https://github.com/duyuwen-duen/RoCo-Sim
KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
  Open-Vocabulary Robotic Manipulation
Authors: name: Zixian Liu, name: Mingtong Zhang, name: Yunzhu Li
Published: 2025-03-13 16:59:17

With the rapid advancement of large language models (LLMs) and
vision-language models (VLMs), significant progress has been made in developing
open-vocabulary robotic manipulation systems. However, many existing approaches
overlook the importance of object dynamics, limiting their applicability to
more complex, dynamic tasks. In this work, we introduce KUDA, an
open-vocabulary manipulation system that integrates dynamics learning and
visual prompting through keypoints, leveraging both VLMs and learning-based
neural dynamics models. Our key insight is that a keypoint-based target
specification is simultaneously interpretable by VLMs and can be efficiently
translated into cost functions for model-based planning. Given language
instructions and visual observations, KUDA first assigns keypoints to the RGB
image and queries the VLM to generate target specifications. These abstract
keypoint-based representations are then converted into cost functions, which
are optimized using a learned dynamics model to produce robotic trajectories.
We evaluate KUDA on a range of manipulation tasks, including free-form language
instructions across diverse object categories, multi-object interactions, and
deformable or granular objects, demonstrating the effectiveness of our
framework. The project page is available at http://kuda-dynamics.github.io.
Using Mechanistic Interpretability to Craft Adversarial Attacks against
  Large Language Models
Authors: name: Thomas Winninger, name: Boussad Addad, name: Katarzyna Kapusta
Published: 2025-03-08 16:29:45

Traditional white-box methods for creating adversarial perturbations against
LLMs typically rely only on gradient computation from the targeted model,
ignoring the internal mechanisms responsible for attack success or failure.
Conversely, interpretability studies that analyze these internal mechanisms
lack practical applications beyond runtime interventions. We bridge this gap by
introducing a novel white-box approach that leverages mechanistic
interpretability techniques to craft practical adversarial inputs.
Specifically, we first identify acceptance subspaces - sets of feature vectors
that do not trigger the model's refusal mechanisms - then use gradient-based
optimization to reroute embeddings from refusal subspaces to acceptance
subspaces, effectively achieving jailbreaks. This targeted approach
significantly reduces computation cost, achieving attack success rates of
80-95\% on state-of-the-art models including Gemma2, Llama3.2, and Qwen2.5
within minutes or even seconds, compared to existing techniques that often fail
or require hours of computation. We believe this approach opens a new direction
for both attack research and defense development. Furthermore, it showcases a
practical application of mechanistic interpretability where other methods are
less efficient, which highlights its utility. The code and generated datasets
are available at https://github.com/Sckathach/subspace-rerouting.
Training Video Foundation Models with NVIDIA NeMo
Authors: name: Zeeshan Patel, name: Ethan He, name: Parth Mannan, name: Xiaowei Ren, name: Ryan Wolf, name: Niket Agarwal, name: Jacob Huffman, name: Zhuoyao Wang, name: Carl Wang, name: Jack Chang, name: Yan Bai, name: Tommy Huang, name: Linnan Wang, name: Sahil Jain, name: Shanmugam Ramasamy, name: Joseph Jennings, name: Ekaterina Sirazitdinova, name: Oleg Sudakov, name: Mingyuan Ma, name: Bobby Chen, name: Forrest Lin, name: Hao Wang, name: Vasanth Rao Naik Sabavat, name: Sriharsha Niverty, name: Rong Ou, name: Pallab Bhattacharya, name: David Page, name: Nima Tajbakhsh, name: Ashwath Aithal
Published: 2025-03-17 09:19:12

Video Foundation Models (VFMs) have recently been used to simulate the real
world to train physical AI systems and develop creative visual experiences.
However, there are significant challenges in training large-scale, high quality
VFMs that can generate high-quality videos. We present a scalable, open-source
VFM training pipeline with NVIDIA NeMo, providing accelerated video dataset
curation, multimodal data loading, and parallelized video diffusion model
training and inference. We also provide a comprehensive performance analysis
highlighting best practices for efficient VFM training and inference.
GenStereo: Towards Open-World Generation of Stereo Images and
  Unsupervised Matching
Authors: name: Feng Qiao, name: Zhexiao Xiong, name: Eric Xing, name: Nathan Jacobs
Published: 2025-03-17 01:19:28

Stereo images are fundamental to numerous applications, including extended
reality (XR) devices, autonomous driving, and robotics. Unfortunately,
acquiring high-quality stereo images remains challenging due to the precise
calibration requirements of dual-camera setups and the complexity of obtaining
accurate, dense disparity maps. Existing stereo image generation methods
typically focus on either visual quality for viewing or geometric accuracy for
matching, but not both. We introduce GenStereo, a diffusion-based approach, to
bridge this gap. The method includes two primary innovations (1) conditioning
the diffusion process on a disparity-aware coordinate embedding and a warped
input image, allowing for more precise stereo alignment than previous methods,
and (2) an adaptive fusion mechanism that intelligently combines the
diffusion-generated image with a warped image, improving both realism and
disparity consistency. Through extensive training on 11 diverse stereo
datasets, GenStereo demonstrates strong generalization ability. GenStereo
achieves state-of-the-art performance in both stereo image generation and
unsupervised stereo matching tasks. Our framework eliminates the need for
complex hardware setups while enabling high-quality stereo image generation,
making it valuable for both real-world applications and unsupervised learning
scenarios. Project page is available at https://qjizhi.github.io/genstereo
WISA: World Simulator Assistant for Physics-Aware Text-to-Video
  Generation
Authors: name: Jing Wang, name: Ao Ma, name: Ke Cao, name: Jun Zheng, name: Zhanjie Zhang, name: Jiasong Feng, name: Shanyuan Liu, name: Yuhang Ma, name: Bo Cheng, name: Dawei Leng, name: Yuhui Yin, name: Xiaodan Liang
Published: 2025-03-11 08:10:03

Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and
Kling, have shown great potential for building world simulators. However,
current T2V models struggle to grasp abstract physical principles and generate
videos that adhere to physical laws. This challenge arises primarily from a
lack of clear guidance on physical information due to a significant gap between
abstract physical principles and generation models. To this end, we introduce
the World Simulator Assistant (WISA), an effective framework for decomposing
and incorporating physical principles into T2V models. Specifically, WISA
decomposes physical principles into textual physical descriptions, qualitative
physical categories, and quantitative physical properties. To effectively embed
these physical attributes into the generation process, WISA incorporates
several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and
a Physical Classifier, enhancing the model's physics awareness. Furthermore,
most existing datasets feature videos where physical phenomena are either
weakly represented or entangled with multiple co-occurring processes, limiting
their suitability as dedicated resources for learning explicit physical
principles. We propose a novel video dataset, WISA-32K, collected based on
qualitative physical categories. It consists of 32,000 videos, representing 17
physical laws across three domains of physics: dynamics, thermodynamics, and
optics. Experimental results demonstrate that WISA can effectively enhance the
compatibility of T2V models with real-world physical laws, achieving a
considerable improvement on the VideoPhy benchmark. The visual exhibitions of
WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.
SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?
Authors: name: Jianzhu Yao, name: Kevin Wang, name: Ryan Hsieh, name: Haisu Zhou, name: Tianqing Zou, name: Zerui Cheng, name: Zhangyang Wang, name: Pramod Viswanath
Published: 2025-03-16 04:10:53

Reasoning and strategic behavior in social interactions is a hallmark
of intelligence. This form of reasoning is significantly more sophisticated
than isolated planning or reasoning tasks in static settings (e.g., math
problem solving). In this paper, we present Strategic Planning,
Interaction, and Negotiation (SPIN-Bench), a new multi-domain
evaluation designed to measure the intelligence of strategic planning
and social reasoning. While many existing benchmarks focus on narrow
planning or single-agent reasoning, SPIN-Bench combines classical PDDL tasks,
competitive board games, cooperative card games, and multi-agent negotiation
scenarios in one unified framework. The framework includes both a benchmark as
well as an arena to simulate and evaluate the variety of social settings to
test reasoning and strategic behavior of AI agents. We formulate the benchmark
SPIN-Bench by systematically varying action spaces, state complexity, and the
number of interacting agents to simulate a variety of social settings where
success depends on not only methodical and step-wise decision making, but also
conceptual inference of other (adversarial or cooperative) participants.
Our experiments reveal that while contemporary LLMs handle basic fact
retrieval and short-range planning reasonably well, they encounter
significant performance bottlenecks in tasks requiring deep multi-hop
reasoning over large state spaces and socially adept coordination under
uncertainty. We envision SPIN-Bench as a catalyst for future research on robust
multi-agent planning, social reasoning, and human--AI teaming.
Free-form language-based robotic reasoning and grasping
Authors: name: Runyu Jiao, name: Alice Fasoli, name: Francesco Giuliari, name: Matteo Bortolon, name: Sergio Povoli, name: Guofeng Mei, name: Yiming Wang, name: Fabio Poiesi
Published: 2025-03-17 11:41:16

Performing robotic grasping from a cluttered bin based on human instructions
is a challenging task, as it requires understanding both the nuances of
free-form language and the spatial relationships between objects.
Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have
demonstrated remarkable reasoning capabilities across both text and images. But
can they truly be used for this task in a zero-shot setting? And what are their
limitations? In this paper, we explore these research questions via the
free-form language-based robotic grasping task, and propose a novel method,
FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about
human instructions and object spatial arrangements. Our method detects all
objects as keypoints and uses these keypoints to annotate marks on images,
aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our
method to determine whether a requested object is directly graspable or if
other objects must be grasped and removed first. Since no existing dataset is
specifically designed for this task, we introduce a synthetic dataset
FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated
instructions and ground-truth grasping sequences. We conduct extensive analyses
with both FreeGraspData and real-world validation with a gripper-equipped
robotic arm, demonstrating state-of-the-art performance in grasp reasoning and
execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey
Authors: name: Yaoting Wang, name: Shengqiong Wu, name: Yuecheng Zhang, name: William Wang, name: Ziwei Liu, name: Jiebo Luo, name: Hao Fei
Published: 2025-03-16 18:39:13

By extending the advantage of chain-of-thought (CoT) reasoning in human-like
step-by-step processes to multimodal contexts, multimodal CoT (MCoT) reasoning
has recently garnered significant research attention, especially in the
integration with multimodal large language models (MLLMs). Existing MCoT
studies design various methodologies and innovative reasoning paradigms to
address the unique challenges of image, video, speech, audio, 3D, and
structured data across different modalities, achieving extensive success in
applications such as robotics, healthcare, autonomous driving, and multimodal
generation. However, MCoT still presents distinct challenges and opportunities
that require further focus to ensure consistent thriving in this field, where,
unfortunately, an up-to-date review of this domain is lacking. To bridge this
gap, we present the first systematic survey of MCoT reasoning, elucidating the
relevant foundational concepts and definitions. We offer a comprehensive
taxonomy and an in-depth analysis of current methodologies from diverse
perspectives across various application scenarios. Furthermore, we provide
insights into existing challenges and future research directions, aiming to
foster innovation toward multimodal AGI.
Sightation Counts: Leveraging Sighted User Feedback in Building a
  BLV-aligned Dataset of Diagram Descriptions
Authors: name: Wan Ju Kang, name: Eunki Kim, name: Na Min An, name: Sangryul Kim, name: Haemin Choi, name: Ki Hoon Kwak, name: James Thorne
Published: 2025-03-17 16:52:46

Often, the needs and visual abilities differ between the annotator group and
the end user group. Generating detailed diagram descriptions for blind and
low-vision (BLV) users is one such challenging domain. Sighted annotators could
describe visuals with ease, but existing studies have shown that direct
generations by them are costly, bias-prone, and somewhat lacking by BLV
standards. In this study, we ask sighted individuals to assess -- rather than
produce -- diagram descriptions generated by vision-language models (VLM) that
have been guided with latent supervision via a multi-pass inference. The
sighted assessments prove effective and useful to professional educators who
are themselves BLV and teach visually impaired learners. We release Sightation,
a collection of diagram description datasets spanning 5k diagrams and 137k
samples for completion, preference, retrieval, question answering, and
reasoning training purposes and demonstrate their fine-tuning potential in
various downstream tasks.
Long-Video Audio Synthesis with Multi-Agent Collaboration
Authors: name: Yehang Zhang, name: Xinli Xu, name: Xiaojie Xu, name: Li Liu, name: Yingcong Chen
Published: 2025-03-13 07:58:23

Video-to-audio synthesis, which generates synchronized audio for visual
content, critically enhances viewer immersion and narrative coherence in film
and interactive media. However, video-to-audio dubbing for long-form content
remains an unsolved challenge due to dynamic semantic shifts, temporal
misalignment, and the absence of dedicated datasets. While existing methods
excel in short videos, they falter in long scenarios (e.g., movies) due to
fragmented synthesis and inadequate cross-scene consistency. We propose
LVAS-Agent, a novel multi-agent framework that emulates professional dubbing
workflows through collaborative role specialization. Our approach decomposes
long-video synthesis into four steps including scene segmentation, script
generation, sound design and audio synthesis. Central innovations include a
discussion-correction mechanism for scene/script refinement and a
generation-retrieval loop for temporal-semantic alignment. To enable systematic
evaluation, we introduce LVAS-Bench, the first benchmark with 207
professionally curated long videos spanning diverse scenarios. Experiments
demonstrate superior audio-visual alignment over baseline methods. Project
page: https://lvas-agent.github.io