Orionnnn/hub-stats huggingface dataset:
Exploring the Effect of Reinforcement Learning on Video Understanding:
  Insights from SEED-Bench-R1
Authors: name: Yi Chen, name: Yuying Ge, name: Rui Wang, name: Yixiao Ge, name: Lu Qiu, name: Ying Shan, name: Xihui Liu
Published: 2025-03-31 17:55:23

Recent advancements in Chain of Thought (COT) generation have significantly
improved the reasoning capabilities of Large Language Models (LLMs), with
reinforcement learning (RL) emerging as an effective post-training approach.
Multimodal Large Language Models (MLLMs) inherit this reasoning potential but
remain underexplored in tasks requiring both perception and logical reasoning.
To address this, we introduce SEED-Bench-R1, a benchmark designed to
systematically evaluate post-training methods for MLLMs in video understanding.
It includes intricate real-world videos and complex everyday planning tasks in
the format of multiple-choice questions, requiring sophisticated perception and
reasoning. SEED-Bench-R1 assesses generalization through a three-level
hierarchy: in-distribution, cross-environment, and cross-environment-task
scenarios, equipped with a large-scale training dataset with easily verifiable
ground-truth answers. Using Qwen2-VL-Instruct-7B as a base model, we compare RL
with supervised fine-tuning (SFT), demonstrating RL's data efficiency and
superior performance on both in-distribution and out-of-distribution tasks,
even outperforming SFT on general video understanding benchmarks like
LongVideoBench. Our detailed analysis reveals that RL enhances visual
perception but often produces less logically coherent reasoning chains. We
identify key limitations such as inconsistent reasoning and overlooked visual
cues, and suggest future improvements in base model reasoning, reward modeling,
and RL robustness against noisy signals.
AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models
  with Unsupervised Coefficient Optimization
Authors: name: Yiyang Du, name: Xiaochen Wang, name: Chi Chen, name: Jiabo Ye, name: Yiru Wang, name: Peng Li, name: Ming Yan, name: Ji Zhang, name: Fei Huang, name: Zhifang Sui, name: Maosong Sun, name: Yang Liu
Published: 2025-03-31 05:13:02

Recently, model merging methods have demonstrated powerful strengths in
combining abilities on various tasks from multiple Large Language Models
(LLMs). While previous model merging methods mainly focus on merging
homogeneous models with identical architecture, they meet challenges when
dealing with Multimodal Large Language Models (MLLMs) with inherent
heterogeneous property, including differences in model architecture and the
asymmetry in the parameter space. In this work, we propose AdaMMS, a novel
model merging method tailored for heterogeneous MLLMs. Our method tackles the
challenges in three steps: mapping, merging and searching. Specifically, we
first design mapping function between models to apply model merging on MLLMs
with different architecture. Then we apply linear interpolation on model
weights to actively adapt the asymmetry in the heterogeneous MLLMs. Finally in
the hyper-parameter searching step, we propose an unsupervised hyper-parameter
selection method for model merging. As the first model merging method capable
of merging heterogeneous MLLMs without labeled data, extensive experiments on
various model combinations demonstrated that AdaMMS outperforms previous model
merging methods on various vision-language benchmarks.
ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via
  Residual Learning
Authors: name: Kailin Li, name: Puhao Li, name: Tengyu Liu, name: Yuyang Li, name: Siyuan Huang
Published: 2025-03-27 17:50:30

Human hands play a central role in interacting, motivating increasing
research in dexterous robotic manipulation. Data-driven embodied AI algorithms
demand precise, large-scale, human-like manipulation sequences, which are
challenging to obtain with conventional reinforcement learning or real-world
teleoperation. To address this, we introduce ManipTrans, a novel two-stage
method for efficiently transferring human bimanual skills to dexterous robotic
hands in simulation. ManipTrans first pre-trains a generalist trajectory
imitator to mimic hand motion, then fine-tunes a specific residual module under
interaction constraints, enabling efficient learning and accurate execution of
complex bimanual tasks. Experiments show that ManipTrans surpasses
state-of-the-art methods in success rate, fidelity, and efficiency. Leveraging
ManipTrans, we transfer multiple hand-object datasets to robotic hands,
creating DexManipNet, a large-scale dataset featuring previously unexplored
tasks like pen capping and bottle unscrewing. DexManipNet comprises 3.3K
episodes of robotic manipulation and is easily extensible, facilitating further
policy training for dexterous hands and enabling real-world deployments.
GeometryCrafter: Consistent Geometry Estimation for Open-world Videos
  with Diffusion Priors
Authors: name: Tian-Xing Xu, name: Xiangjun Gao, name: Wenbo Hu, name: Xiaoyu Li, name: Song-Hai Zhang, name: Ying Shan
Published: 2025-04-01 17:58:03

Despite remarkable advancements in video depth estimation, existing methods
exhibit inherent limitations in achieving geometric fidelity through the
affine-invariant predictions, limiting their applicability in reconstruction
and other metrically grounded downstream tasks. We propose GeometryCrafter, a
novel framework that recovers high-fidelity point map sequences with temporal
coherence from open-world videos, enabling accurate 3D/4D reconstruction,
camera parameter estimation, and other depth-based applications. At the core of
our approach lies a point map Variational Autoencoder (VAE) that learns a
latent space agnostic to video latent distributions for effective point map
encoding and decoding. Leveraging the VAE, we train a video diffusion model to
model the distribution of point map sequences conditioned on the input videos.
Extensive evaluations on diverse datasets demonstrate that GeometryCrafter
achieves state-of-the-art 3D accuracy, temporal consistency, and generalization
capability.
When To Solve, When To Verify: Compute-Optimal Problem Solving and
  Generative Verification for LLM Reasoning
Authors: name: Nishad Singhi, name: Hritik Bansal, name: Arian Hosseini, name: Aditya Grover, name: Kai-Wei Chang, name: Marcus Rohrbach, name: Anna Rohrbach
Published: 2025-04-01 17:41:57

Scaling test-time compute has emerged as a key strategy for enhancing the
reasoning capabilities of large language models (LLMs), particularly in tasks
like mathematical problem-solving. A traditional approach, Self-Consistency
(SC), generates multiple solutions to a problem and selects the most common
answer via majority voting. Another common method involves scoring each
solution with a reward model (verifier) and choosing the best one. Recent
advancements in Generative Reward Models (GenRM) reframe verification as a
next-token prediction task, enabling inference-time scaling along a new axis.
Specifically, GenRM generates multiple verification chains-of-thought to score
each solution. Under a limited inference budget, this introduces a fundamental
trade-off: should you spend the budget on scaling solutions via SC or generate
fewer solutions and allocate compute to verification via GenRM? To address
this, we evaluate GenRM against SC under a fixed inference budget.
Interestingly, we find that SC is more compute-efficient than GenRM for most
practical inference budgets across diverse models and datasets. For instance,
GenRM first matches SC after consuming up to 8x the inference compute and
requires significantly more compute to outperform it. Furthermore, we derive
inference scaling laws for the GenRM paradigm, revealing that compute-optimal
inference favors scaling solution generation more aggressively than scaling the
number of verifications. Our work provides practical guidance on optimizing
test-time scaling by balancing solution generation and verification. The code
is available at https://github.com/nishadsinghi/sc-genrm-scaling.
Discovering Knowledge Deficiencies of Language Models on Massive
  Knowledge Base
Authors: name: Linxin Song, name: Xuwei Ding, name: Jieyu Zhang, name: Taiwei Shi, name: Ryotaro Shimizu, name: Rahul Gupta, name: Yang Liu, name: Jian Kang, name: Jieyu Zhao
Published: 2025-03-30 08:33:56

Large language models (LLMs) possess impressive linguistic capabilities but
often fail to faithfully retain factual knowledge, leading to hallucinations
and unreliable outputs. Understanding LLMs' knowledge deficiencies by
exhaustively evaluating against full-scale knowledge bases is computationally
prohibitive, especially for closed-weight models. We propose stochastic error
ascent (SEA), a scalable and efficient framework for discovering knowledge
deficiencies (errors) in closed-weight LLMs under a strict query budget. Rather
than naively probing all knowledge candidates, SEA formulates error discovery
as a stochastic optimization process: it iteratively retrieves new high-error
candidates by leveraging the semantic similarity to previously observed
failures. To further enhance search efficiency and coverage, SEA employs
hierarchical retrieval across document and paragraph levels, and constructs a
relation directed acyclic graph to model error propagation and identify
systematic failure modes. Empirically, SEA uncovers 40.7x more knowledge errors
than Automated Capability Discovery and 26.7% more than AutoBencher, while
reducing the cost-per-error by 599x and 9x, respectively. Human evaluation
confirms the high quality of generated questions, while ablation and
convergence analyses validate the contribution of each component in SEA.
Further analysis on the discovered errors reveals correlated failure patterns
across LLM families and recurring deficits, highlighting the need for better
data coverage and targeted fine-tuning in future LLM development.
Agent S2: A Compositional Generalist-Specialist Framework for Computer
  Use Agents
Authors: name: Saaket Agashe, name: Kyle Wong, name: Vincent Tu, name: Jiachen Yang, name: Ang Li, name: Xin Eric Wang
Published: 2025-04-01 15:40:27

Computer use agents automate digital tasks by directly interacting with
graphical user interfaces (GUIs) on computers and mobile devices, offering
significant potential to enhance human productivity by completing an open-ended
space of user queries. However, current agents face significant challenges:
imprecise grounding of GUI elements, difficulties with long-horizon task
planning, and performance bottlenecks from relying on single generalist models
for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
compositional framework that delegates cognitive responsibilities across
various generalist and specialist models. We propose a novel
Mixture-of-Grounding technique to achieve precise GUI localization and
introduce Proactive Hierarchical Planning, dynamically refining action plans at
multiple temporal scales in response to evolving observations. Evaluations
demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
on three prominent computer use benchmarks. Specifically, Agent S2 achieves
18.9% and 32.7% relative improvements over leading baseline agents such as
Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
Moreover, Agent S2 generalizes effectively to other operating systems and
applications, surpassing previous best methods by 52.8% on WindowsAgentArena
and by 16.52% on AndroidWorld relatively. Code available at
https://github.com/simular-ai/Agent-S.
DSO: Aligning 3D Generators with Simulation Feedback for Physical
  Soundness
Authors: name: Ruining Li, name: Chuanxia Zheng, name: Christian Rupprecht, name: Andrea Vedaldi
Published: 2025-03-28 17:59:53

Most 3D object generators focus on aesthetic quality, often neglecting
physical constraints necessary in applications. One such constraint is that the
3D object should be self-supporting, i.e., remains balanced under gravity.
Prior approaches to generating stable 3D objects used differentiable physics
simulators to optimize geometry at test-time, which is slow, unstable, and
prone to local optima. Inspired by the literature on aligning generative models
to external feedback, we propose Direct Simulation Optimization (DSO), a
framework to use the feedback from a (non-differentiable) simulator to increase
the likelihood that the 3D generator outputs stable 3D objects directly. We
construct a dataset of 3D objects labeled with a stability score obtained from
the physics simulator. We can then fine-tune the 3D generator using the
stability score as the alignment metric, via direct preference optimization
(DPO) or direct reward optimization (DRO), a novel objective, which we
introduce, to align diffusion models without requiring pairwise preferences.
Our experiments show that the fine-tuned feed-forward generator, using either
DPO or DRO objective, is much faster and more likely to produce stable objects
than test-time optimization. Notably, the DSO framework works even without any
ground-truth 3D objects for training, allowing the 3D generator to self-improve
by automatically collecting simulation feedback on its own outputs.
AvatarArtist: Open-Domain 4D Avatarization
Authors: name: Hongyu Liu, name: Xuan Wang, name: Ziyu Wan, name: Yue Ma, name: Jingye Chen, name: Yanbo Fan, name: Yujun Shen, name: Yibing Song, name: Qifeng Chen
Published: 2025-03-25 17:59:03

This work focuses on open-domain 4D avatarization, with the purpose of
creating a 4D avatar from a portrait image in an arbitrary style. We select
parametric triplanes as the intermediate 4D representation and propose a
practical training paradigm that takes advantage of both generative adversarial
networks (GANs) and diffusion models. Our design stems from the observation
that 4D GANs excel at bridging images and triplanes without supervision yet
usually face challenges in handling diverse data distributions. A robust 2D
diffusion prior emerges as the solution, assisting the GAN in transferring its
expertise across various domains. The synergy between these experts permits the
construction of a multi-domain image-triplane dataset, which drives the
development of a general 4D avatar creator. Extensive experiments suggest that
our model, AvatarArtist, is capable of producing high-quality 4D avatars with
strong robustness to various source image domains. The code, the data, and the
models will be made publicly available to facilitate future studies..
ActionStudio: A Lightweight Framework for Data and Training of Large
  Action Models
Authors: name: Jianguo Zhang, name: Thai Hoang, name: Ming Zhu, name: Zuxin Liu, name: Shiyu Wang, name: Tulika Awalgaonkar, name: Akshara Prabhakar, name: Haolin Chen, name: Weiran Yao, name: Zhiwei Liu, name: Juntao Tan, name: Juan Carlos Niebles, name: Shelby Heinecke, name: Huan Wang, name: Silvio Savarese, name: Caiming Xiong
Published: 2025-03-28 17:58:33

Action models are essential for enabling autonomous agents to perform complex
tasks. However, training large action models remains challenging due to the
diversity of agent environments and the complexity of agentic data. Despite
growing interest, existing infrastructure provides limited support for
scalable, agent-specific fine-tuning. We present ActionStudio, a lightweight
and extensible data and training framework designed for large action models.
ActionStudio unifies heterogeneous agent trajectories through a standardized
format, supports diverse training paradigms including LoRA, full fine-tuning,
and distributed setups, and integrates robust preprocessing and verification
tools. We validate its effectiveness across both public and realistic industry
benchmarks, demonstrating strong performance and practical scalability. We
open-sourced code and data at https://github.com/SalesforceAIResearch/xLAM to
facilitate research in the community.
PAVE: Patching and Adapting Video Large Language Models
Authors: name: Zhuoming Liu, name: Yiquan Li, name: Khoi Duc Nguyen, name: Yiwu Zhong, name: Yin Li
Published: 2025-03-25 16:02:37

Pre-trained video large language models (Video LLMs) exhibit remarkable
reasoning capabilities, yet adapting these models to new tasks involving
additional modalities or data types (e.g., audio or 3D information) remains
challenging. In this paper, we present PAVE, a flexible framework for adapting
pre-trained Video LLMs to downstream tasks with side-channel signals, such as
audio, 3D cues, or multi-view videos. PAVE introduces lightweight adapters,
referred to as "patches," which add a small number of parameters and operations
to a base model without modifying its architecture or pre-trained weights. In
doing so, PAVE can effectively adapt the pre-trained base model to support
diverse downstream tasks, including audio-visual question answering, 3D
reasoning, multi-view video recognition, and high frame rate video
understanding. Across these tasks, PAVE significantly enhances the performance
of the base model, surpassing state-of-the-art task-specific models while
incurring a minor cost of ~0.1% additional FLOPs and parameters. Further, PAVE
supports multi-task learning and generalizes well across different Video LLMs.
Our code is available at https://github.com/dragonlzm/PAVE.
Understanding Co-speech Gestures in-the-wild
Authors: name: Sindhu B Hegde, name: K R Prajwal, name: Taein Kwon, name: Andrew Zisserman
Published: 2025-03-28 17:55:52

Co-speech gestures play a vital role in non-verbal communication. In this
paper, we introduce a new framework for co-speech gesture understanding in the
wild. Specifically, we propose three new tasks and benchmarks to evaluate a
model's capability to comprehend gesture-text-speech associations: (i)
gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker
detection using gestures. We present a new approach that learns a tri-modal
speech-text-video-gesture representation to solve these tasks. By leveraging a
combination of global phrase contrastive loss and local gesture-word coupling
loss, we demonstrate that a strong gesture representation can be learned in a
weakly supervised manner from videos in the wild. Our learned representations
outperform previous methods, including large vision-language models (VLMs),
across all three tasks. Further analysis reveals that speech and text
modalities capture distinct gesture-related signals, underscoring the
advantages of learning a shared tri-modal embedding space. The dataset, model,
and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal
Unicorn: Text-Only Data Synthesis for Vision Language Model Training
Authors: name: Xiaomin Yu, name: Pengxiang Ding, name: Wenjie Zhang, name: Siteng Huang, name: Songyang Gao, name: Chengwei Qin, name: Kejian Wu, name: Zhaoxin Fan, name: Ziyue Qiao, name: Donglin Wang
Published: 2025-03-28 17:43:00

Training vision-language models (VLMs) typically requires large-scale,
high-quality image-text pairs, but collecting or synthesizing such data is
costly. In contrast, text data is abundant and inexpensive, prompting the
question: can high-quality multimodal training data be synthesized purely from
text? To tackle this, we propose a cross-integrated three-stage multimodal data
synthesis framework, which generates two datasets: Unicorn-1.2M and
Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
construct 1.2M semantically diverse high-quality captions by expanding sparse
caption seeds using large language models (LLMs). In Stage 2:
Instruction-Tuning Data Generation, we further process 471K captions into
multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
Stage 3: Modality Representation Transfer, these textual captions
representations are transformed into visual representations, resulting in
diverse synthetic image representations. This three-stage process enables us to
construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
instruction-tuning, without relying on real images. By eliminating the
dependency on real images while maintaining data quality and diversity, our
framework offers a cost-effective and scalable solution for VLMs training. Code
is available at https://github.com/Yu-xm/Unicorn.git.
Entropy-Based Adaptive Weighting for Self-Training
Authors: name: Xiaoxuan Wang, name: Yihe Deng, name: Mingyu Derek Ma, name: Wei Wang
Published: 2025-03-31 10:04:35

The mathematical problem-solving capabilities of large language models have
become a focal point of research, with growing interests in leveraging
self-generated reasoning paths as a promising way to refine and enhance these
models. These paths capture step-by-step logical processes while requiring only
the correct answer for supervision. The self-training method has been shown to
be effective in reasoning tasks while eliminating the need for external models
and manual annotations. However, optimizing the use of self-generated data for
model training remains an open challenge. In this work, we propose
Entropy-Based Adaptive Weighting for Self-Training (EAST), an adaptive
weighting strategy designed to prioritize uncertain data during self-training.
Specifically, EAST employs a mapping function with a tunable parameter that
controls the sharpness of the weighting, assigning higher weights to data where
the model exhibits greater uncertainty. This approach guides the model to focus
on more informative and challenging examples, thereby enhancing its reasoning
ability. We evaluate our approach on GSM8K and MATH benchmarks. Empirical
results show that, while the vanilla method yields virtually no improvement
(0%) on MATH, EAST achieves around a 1% gain over backbone model. On GSM8K,
EAST attains a further 1-2% performance boost compared to the vanilla method.
MeshCraft: Exploring Efficient and Controllable Mesh Generation with
  Flow-based DiTs
Authors: name: Xianglong He, name: Junyi Chen, name: Di Huang, name: Zexiang Liu, name: Xiaoshui Huang, name: Wanli Ouyang, name: Chun Yuan, name: Yangguang Li
Published: 2025-03-29 09:21:50

In the domain of 3D content creation, achieving optimal mesh topology through
AI models has long been a pursuit for 3D artists. Previous methods, such as
MeshGPT, have explored the generation of ready-to-use 3D objects via mesh
auto-regressive techniques. While these methods produce visually impressive
results, their reliance on token-by-token predictions in the auto-regressive
process leads to several significant limitations. These include extremely slow
generation speeds and an uncontrollable number of mesh faces. In this paper, we
introduce MeshCraft, a novel framework for efficient and controllable mesh
generation, which leverages continuous spatial diffusion to generate discrete
triangle faces. Specifically, MeshCraft consists of two core components: 1) a
transformer-based VAE that encodes raw meshes into continuous face-level tokens
and decodes them back to the original meshes, and 2) a flow-based diffusion
transformer conditioned on the number of faces, enabling the generation of
high-quality 3D meshes with a predefined number of faces. By utilizing the
diffusion model for the simultaneous generation of the entire mesh topology,
MeshCraft achieves high-fidelity mesh generation at significantly faster speeds
compared to auto-regressive methods. Specifically, MeshCraft can generate an
800-face mesh in just 3.2 seconds (35times faster than existing baselines).
Extensive experiments demonstrate that MeshCraft outperforms state-of-the-art
techniques in both qualitative and quantitative evaluations on ShapeNet dataset
and demonstrates superior performance on Objaverse dataset. Moreover, it
integrates seamlessly with existing conditional guidance strategies, showcasing
its potential to relieve artists from the time-consuming manual work involved
in mesh creation.