Orionnnn/hub-stats huggingface dataset:
ProJudge: A Multi-Modal Multi-Discipline Benchmark and
  Instruction-Tuning Dataset for MLLM-based Process Judges
Authors: name: Jiaxin Ai, name: Pengfei Zhou, name: Zhaopan Xu, name: Ming Li, name: Fanrui Zhang, name: Zizhen Li, name: Jianwen Sun, name: Yukang Feng, name: Baojin Huang, name: Zhongyuan Wang, name: Kaipeng Zhang
Published: 2025-03-09 10:55:51

As multi-modal large language models (MLLMs) frequently exhibit errors when
solving scientific problems, evaluating the validity of their reasoning
processes is critical for ensuring reliability and uncovering fine-grained
model weaknesses. Since human evaluation is laborious and costly, prompting
MLLMs as automated process judges has become a common practice. However, the
reliability of these model-based judges remains uncertain. To address this, we
introduce ProJudgeBench, the first comprehensive benchmark specifically
designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
comprises 2,400 test cases and 50,118 step-level labels, spanning four
scientific disciplines with diverse difficulty levels and multi-modal content.
In ProJudgeBench, each step is meticulously annotated by human experts for
correctness, error type, and explanation, enabling a systematic evaluation of
judges' capabilities to detect, classify and diagnose errors. Evaluation on
ProJudgeBench reveals a significant performance gap between open-source and
proprietary models. To bridge this gap, we further propose ProJudge-173k, a
large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
strategy that encourages models to explicitly reason through problem-solving
before assessing solutions. Both contributions significantly enhance the
process evaluation capabilities of open-source models. All the resources will
be released to foster future research of reliable multi-modal process
evaluation.
ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model
  with Interleaved Multimodal Generation via Asymmetric Synergy
Authors: name: Jianwen Sun, name: Yukang Feng, name: Chuanhao Li, name: Fanrui Zhang, name: Zizhen Li, name: Jiaxin Ai, name: Sizhuo Zhou, name: Yu Dai, name: Shenglin Zhang, name: Kaipeng Zhang
Published: 2025-03-09 10:15:39

Unified models (UniMs) for multimodal understanding and generation have
recently received much attention in the area of vision and language. Existing
UniMs are designed to simultaneously learn both multimodal understanding and
generation capabilities, demanding substantial computational resources, and
often struggle to generate interleaved text-image. We present ARMOR, a
resource-efficient and pure autoregressive framework that achieves both
understanding and generation by fine-tuning existing multimodal large language
models (MLLMs). Specifically, ARMOR extends existing MLLMs from three
perspectives: (1) For model architecture, an asymmetric encoder-decoder
architecture with a forward-switching mechanism is introduced to unify
embedding space integrating textual and visual modalities for enabling natural
text-image interleaved generation with minimal computational overhead. (2) For
training data, a meticulously curated, high-quality interleaved dataset is
collected for fine-tuning MLLMs. (3) For the training algorithm, we propose a
``what or how to generate" algorithm to empower existing MLLMs with multimodal
generation capabilities while preserving their multimodal understanding
capabilities, through three progressive training stages based on the collected
dataset. Experimental results demonstrate that ARMOR upgrades existing MLLMs to
UniMs with promising image generation capabilities, using limited training
resources. Our code will be released soon at https://armor.github.io.
Large-scale Pre-training for Grounded Video Caption Generation
Authors: name: Evangelos Kazakos, name: Cordelia Schmid, name: Josef Sivic
Published: 2025-03-13 18:21:07

We propose a novel approach for captioning and object grounding in video,
where the objects in the caption are grounded in the video via temporally dense
bounding boxes. We introduce the following contributions. First, we present a
large-scale automatic annotation method that aggregates captions grounded with
bounding boxes across individual frames into temporally dense and consistent
bounding box annotations. We apply this approach on the HowTo100M dataset to
construct a large-scale pre-training dataset, named HowToGround1M. We also
introduce a Grounded Video Caption Generation model, dubbed GROVE, and
pre-train the model on HowToGround1M. Second, we introduce a new dataset,
called iGround, of 3500 videos with manually annotated captions and dense
spatio-temporally grounded bounding boxes. This allows us to measure progress
on this challenging problem, as well as to fine-tune our model on this
small-scale but high-quality data. Third, we demonstrate that our approach
achieves state-of-the-art results on the proposed iGround dataset compared to a
number of baselines, as well as on the VidSTG and ActivityNet-Entities
datasets. We perform extensive ablations that demonstrate the importance of
pre-training using our automatically annotated HowToGround1M dataset followed
by fine-tuning on the manually annotated iGround dataset and validate the key
technical contributions of our model.
Learning Few-Step Diffusion Models by Trajectory Distribution Matching
Authors: name: Yihong Luo, name: Tianyang Hu, name: Jiacheng Sun, name: Yujun Cai, name: Jing Tang
Published: 2025-03-09 15:53:49

Accelerating diffusion model sampling is crucial for efficient AIGC
deployment. While diffusion distillation methods -- based on distribution
matching and trajectory matching -- reduce sampling to as few as one step, they
fall short on complex tasks like text-to-image generation. Few-step generation
offers a better balance between speed and quality, but existing approaches face
a persistent trade-off: distribution matching lacks flexibility for multi-step
sampling, while trajectory matching often yields suboptimal image quality. To
bridge this gap, we propose learning few-step diffusion models by Trajectory
Distribution Matching (TDM), a unified distillation paradigm that combines the
strengths of distribution and trajectory matching. Our method introduces a
data-free score distillation objective, aligning the student's trajectory with
the teacher's at the distribution level. Further, we develop a
sampling-steps-aware objective that decouples learning targets across different
steps, enabling more adjustable sampling. This approach supports both
deterministic sampling for superior image quality and flexible multi-step
adaptation, achieving state-of-the-art performance with remarkable efficiency.
Our model, TDM, outperforms existing methods on various backbones, such as SDXL
and PixArt-alpha, delivering superior quality and significantly reduced
training costs. In particular, our method distills PixArt-alpha into a
4-step generator that outperforms its teacher on real user preference at 1024
resolution. This is accomplished with 500 iterations and 2 A800 hours -- a mere
0.01% of the teacher's training cost. In addition, our proposed TDM can be
extended to accelerate text-to-video diffusion. Notably, TDM can outperform its
teacher model (CogVideoX-2B) by using only 4 NFE on VBench, improving the total
score from 80.91 to 81.65. Project page: https://tdm-t2x.github.io/
TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of
  Tools
Authors: name: Shanghua Gao, name: Richard Zhu, name: Zhenglun Kong, name: Ayush Noori, name: Xiaorui Su, name: Curtis Ginder, name: Theodoros Tsiligkaridis, name: Marinka Zitnik
Published: 2025-03-14 00:28:15

Precision therapeutics require multimodal adaptive models that generate
personalized treatment recommendations. We introduce TxAgent, an AI agent that
leverages multi-step reasoning and real-time biomedical knowledge retrieval
across a toolbox of 211 tools to analyze drug interactions, contraindications,
and patient-specific treatment strategies. TxAgent evaluates how drugs interact
at molecular, pharmacokinetic, and clinical levels, identifies
contraindications based on patient comorbidities and concurrent medications,
and tailors treatment strategies to individual patient characteristics. It
retrieves and synthesizes evidence from multiple biomedical sources, assesses
interactions between drugs and patient conditions, and refines treatment
recommendations through iterative reasoning. It selects tools based on task
objectives and executes structured function calls to solve therapeutic tasks
that require clinical reasoning and cross-source validation. The ToolUniverse
consolidates 211 tools from trusted sources, including all US FDA-approved
drugs since 1939 and validated clinical insights from Open Targets. TxAgent
outperforms leading LLMs, tool-use models, and reasoning agents across five new
benchmarks: DrugPC, BrandPC, GenericPC, TreatmentPC, and DescriptionPC,
covering 3,168 drug reasoning tasks and 456 personalized treatment scenarios.
It achieves 92.1% accuracy in open-ended drug reasoning tasks, surpassing
GPT-4o and outperforming DeepSeek-R1 (671B) in structured multi-step reasoning.
TxAgent generalizes across drug name variants and descriptions. By integrating
multi-step inference, real-time knowledge grounding, and tool-assisted
decision-making, TxAgent ensures that treatment recommendations align with
established clinical guidelines and real-world evidence, reducing the risk of
adverse events and improving therapeutic decision-making.
Adversarial Data Collection: Human-Collaborative Perturbations for
  Efficient and Robust Robotic Imitation Learning
Authors: name: Siyuan Huang, name: Yue Liao, name: Siyuan Feng, name: Shu Jiang, name: Si Liu, name: Hongsheng Li, name: Maoqing Yao, name: Guanghui Ren
Published: 2025-03-14 17:59:07

The pursuit of data efficiency, where quality outweighs quantity, has emerged
as a cornerstone in robotic manipulation, especially given the high costs
associated with real-world data collection. We propose that maximizing the
informational density of individual demonstrations can dramatically reduce
reliance on large-scale datasets while improving task performance. To this end,
we introduce Adversarial Data Collection, a Human-in-the-Loop (HiL) framework
that redefines robotic data acquisition through real-time, bidirectional
human-environment interactions. Unlike conventional pipelines that passively
record static demonstrations, ADC adopts a collaborative perturbation paradigm:
during a single episode, an adversarial operator dynamically alters object
states, environmental conditions, and linguistic commands, while the
tele-operator adaptively adjusts actions to overcome these evolving challenges.
This process compresses diverse failure-recovery behaviors, compositional task
variations, and environmental perturbations into minimal demonstrations. Our
experiments demonstrate that ADC-trained models achieve superior compositional
generalization to unseen task instructions, enhanced robustness to perceptual
perturbations, and emergent error recovery capabilities. Strikingly, models
trained with merely 20% of the demonstration volume collected through ADC
significantly outperform traditional approaches using full datasets. These
advances bridge the gap between data-centric learning paradigms and practical
robotic deployment, demonstrating that strategic data acquisition, not merely
post-hoc processing, is critical for scalable, real-world robot learning.
Additionally, we are curating a large-scale ADC-Robotics dataset comprising
real-world manipulation tasks with adversarial perturbations. This benchmark
will be open-sourced to facilitate advancements in robotic imitation learning.
Technologies on Effectiveness and Efficiency: A Survey of State Spaces
  Models
Authors: name: Xingtai Lv, name: Youbang Sun, name: Kaiyan Zhang, name: Shang Qu, name: Xuekai Zhu, name: Yuchen Fan, name: Yi Wu, name: Ermo Hua, name: Xinwei Long, name: Ning Ding, name: Bowen Zhou
Published: 2025-03-14 09:20:31

State Space Models (SSMs) have emerged as a promising alternative to the
popular transformer-based models and have been increasingly gaining attention.
Compared to transformers, SSMs excel at tasks with sequential data or longer
contexts, demonstrating comparable performances with significant efficiency
gains. In this survey, we provide a coherent and systematic overview for SSMs,
including their theoretical motivations, mathematical formulations, comparison
with existing model classes, and various applications. We divide the SSM series
into three main sections, providing a detailed introduction to the original
SSM, the structured SSM represented by S4, and the selective SSM typified by
Mamba. We put an emphasis on technicality, and highlight the various key
techniques introduced to address the effectiveness and efficiency of SSMs. We
hope this manuscript serves as an introduction for researchers to explore the
theoretical foundations of SSMs.
FlowTok: Flowing Seamlessly Across Text and Image Tokens
Authors: name: Ju He, name: Qihang Yu, name: Qihao Liu, name: Liang-Chieh Chen
Published: 2025-03-13 18:06:13

Bridging different modalities lies at the heart of cross-modality generation.
While conventional approaches treat the text modality as a conditioning signal
that gradually guides the denoising process from Gaussian noise to the target
image modality, we explore a much simpler paradigm-directly evolving between
text and image modalities through flow matching. This requires projecting both
modalities into a shared latent space, which poses a significant challenge due
to their inherently different representations: text is highly semantic and
encoded as 1D tokens, whereas images are spatially redundant and represented as
2D latent embeddings. To address this, we introduce FlowTok, a minimal
framework that seamlessly flows across text and images by encoding images into
a compact 1D token representation. Compared to prior methods, this design
reduces the latent space size by 3.3x at an image resolution of 256,
eliminating the need for complex conditioning mechanisms or noise scheduling.
Moreover, FlowTok naturally extends to image-to-text generation under the same
formulation. With its streamlined architecture centered around compact 1D
tokens, FlowTok is highly memory-efficient, requires significantly fewer
training resources, and achieves much faster sampling speeds-all while
delivering performance comparable to state-of-the-art models. Code will be
available at https://github.com/bytedance/1d-tokenizer.
Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision
  Transformers?
Authors: name: Subhajit Maity, name: Killian Hitsman, name: Xin Li, name: Aritra Dutta
Published: 2025-03-13 17:59:52

Kolmogorov-Arnold networks (KANs) are a remarkable innovation consisting of
learnable activation functions with the potential to capture more complex
relationships from data. Although KANs are useful in finding symbolic
representations and continual learning of one-dimensional functions, their
effectiveness in diverse machine learning (ML) tasks, such as vision, remains
questionable. Presently, KANs are deployed by replacing multilayer perceptrons
(MLPs) in deep network architectures, including advanced architectures such as
vision Transformers (ViTs). In this paper, we are the first to design a general
learnable Kolmogorov-Arnold Attention (KArAt) for vanilla ViTs that can operate
on any choice of basis. However, the computing and memory costs of training
them motivated us to propose a more modular version, and we designed particular
learnable attention, called Fourier-KArAt. Fourier-KArAt and its variants
either outperform their ViT counterparts or show comparable performance on
CIFAR-10, CIFAR-100, and ImageNet-1K datasets. We dissect these architectures'
performance and generalization capacity by analyzing their loss landscapes,
weight distributions, optimizer path, attention visualization, and spectral
behavior, and contrast them with vanilla ViTs. The goal of this paper is not to
produce parameter- and compute-efficient attention, but to encourage the
community to explore KANs in conjunction with more advanced architectures that
require a careful understanding of learnable activations. Our open-source code
and implementation details are available on: https://subhajitmaity.me/KArAt
GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories
  Generation in End-to-End Autonomous Driving
Authors: name: Zebin Xing, name: Xingyu Zhang, name: Yang Hu, name: Bo Jiang, name: Tong He, name: Qian Zhang, name: Xiaoxiao Long, name: Wei Yin
Published: 2025-03-07 18:52:08

We propose GoalFlow, an end-to-end autonomous driving method for generating
high-quality multimodal trajectories. In autonomous driving scenarios, there is
rarely a single suitable trajectory. Recent methods have increasingly focused
on modeling multimodal trajectory distributions. However, they suffer from
trajectory selection complexity and reduced trajectory quality due to high
trajectory divergence and inconsistencies between guidance and scene
information. To address these issues, we introduce GoalFlow, a novel method
that effectively constrains the generative process to produce high-quality,
multimodal trajectories. To resolve the trajectory divergence problem inherent
in diffusion-based methods, GoalFlow constrains the generated trajectories by
introducing a goal point. GoalFlow establishes a novel scoring mechanism that
selects the most appropriate goal point from the candidate points based on
scene information. Furthermore, GoalFlow employs an efficient generative
method, Flow Matching, to generate multimodal trajectories, and incorporates a
refined scoring mechanism to select the optimal trajectory from the candidates.
Our experimental results, validated on the NavsimDauner2024_navsim,
demonstrate that GoalFlow achieves state-of-the-art performance, delivering
robust multimodal trajectories for autonomous driving. GoalFlow achieved PDMS
of 90.3, significantly surpassing other methods. Compared with other
diffusion-policy-based methods, our approach requires only a single denoising
step to obtain excellent performance. The code is available at
https://github.com/YvanYin/GoalFlow.
ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
Authors: name: Jianhong Bai, name: Menghan Xia, name: Xiao Fu, name: Xintao Wang, name: Lianrui Mu, name: Jinwen Cao, name: Zuozhu Liu, name: Haoji Hu, name: Xiang Bai, name: Pengfei Wan, name: Di Zhang
Published: 2025-03-14 17:59:31

Camera control has been actively studied in text or image conditioned video
generation tasks. However, altering camera trajectories of a given video
remains under-explored, despite its importance in the field of video creation.
It is non-trivial due to the extra constraints of maintaining multiple-frame
appearance and dynamic synchronization. To address this, we present
ReCamMaster, a camera-controlled generative video re-rendering framework that
reproduces the dynamic scene of an input video at novel camera trajectories.
The core innovation lies in harnessing the generative capabilities of
pre-trained text-to-video models through a simple yet powerful video
conditioning mechanism -- its capability often overlooked in current research.
To overcome the scarcity of qualified training data, we construct a
comprehensive multi-camera synchronized video dataset using Unreal Engine 5,
which is carefully curated to follow real-world filming characteristics,
covering diverse scenes and camera movements. It helps the model generalize to
in-the-wild videos. Lastly, we further improve the robustness to diverse inputs
through a meticulously designed training strategy. Extensive experiments tell
that our method substantially outperforms existing state-of-the-art approaches
and strong baselines. Our method also finds promising applications in video
stabilization, super-resolution, and outpainting. Project page:
https://jianhongbai.github.io/ReCamMaster/
Cockatiel: Ensembling Synthetic and Human Preferenced Training for
  Detailed Video Caption
Authors: name: Luozheng Qin, name: Zhiyu Tan, name: Mengping Yang, name: Xiaomeng Yang, name: Hao Li
Published: 2025-03-12 11:25:04

Video Detailed Captioning (VDC) is a crucial task for vision-language
bridging, enabling fine-grained descriptions of complex video content. In this
paper, we first comprehensively benchmark current state-of-the-art approaches
and systematically identified two critical limitations: biased capability
towards specific captioning aspect and misalignment with human preferences. To
address these deficiencies, we propose Cockatiel, a novel three-stage training
pipeline that ensembles synthetic and human-aligned training for improving VDC
performance. In the first stage, we derive a scorer from a meticulously
annotated dataset to select synthetic captions high-performing on certain
fine-grained video-caption alignment and human-preferred while disregarding
others. Then, we train Cockatiel-13B, using this curated dataset to infuse it
with assembled model strengths and human preferences. Finally, we further
distill Cockatiel-8B from Cockatiel-13B for the ease of usage. Extensive
quantitative and qualitative experiments reflect the effectiveness of our
method, as we not only set new state-of-the-art performance on VDCSCORE in a
dimension-balanced way but also surpass leading alternatives on human
preference by a large margin as depicted by the human evaluation results.
PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference
  Time by Leveraging Sparsity
Authors: name: Kwanyoung Kim, name: Byeongsu Sim
Published: 2025-03-10 07:23:19

Diffusion models have shown impressive results in generating high-quality
conditional samples using guidance techniques such as Classifier-Free Guidance
(CFG). However, existing methods often require additional training or neural
function evaluations (NFEs), making them incompatible with guidance-distilled
models. Also, they rely on heuristic approaches that need identifying target
layers. In this work, we propose a novel and efficient method, termed PLADIS,
which boosts pre-trained models (U-Net/Transformer) by leveraging sparse
attention. Specifically, we extrapolate query-key correlations using softmax
and its sparse counterpart in the cross-attention layer during inference,
without requiring extra training or NFEs. By leveraging the noise robustness of
sparse attention, our PLADIS unleashes the latent potential of text-to-image
diffusion models, enabling them to excel in areas where they once struggled
with newfound effectiveness. It integrates seamlessly with guidance techniques,
including guidance-distilled models. Extensive experiments show notable
improvements in text alignment and human preference, offering a highly
efficient and universally applicable solution.