Orionnnn/hub-stats huggingface dataset:
MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic
  Evaluation of Audio General Intelligence
Authors: name: Sonal Kumar, name: Šimon Sedláček, name: Vaibhavi Lokegaonkar, name: Fernando López, name: Wenyi Yu, name: Nishit Anand, name: Hyeonggon Ryu, name: Lichang Chen, name: Maxim Plička, name: Miroslav Hlaváček, name: William Fineas Ellingwood, name: Sathvik Udupa, name: Siyuan Hou, name: Allison Ferner, name: Sara Barahona, name: Cecilia Bolaños, name: Satish Rahi, name: Laura Herrera-Alarcón, name: Satvik Dixit, name: Siddhi Patil, name: Soham Deshmukh, name: Lasha Koroshinadze, name: Yao Liu, name: Leibny Paola Garcia Perera, name: Eleni Zanou, name: Themos Stafylakis, name: Joon Son Chung, name: David Harwath, name: Chao Zhang, name: Dinesh Manocha, name: Alicia Lozano-Diez, name: Santosh Kesiraju, name: Sreyan Ghosh, name: Ramani Duraiswami
Published: 2025-08-19 16:33:49

Audio comprehension-including speech, non-speech sounds, and music-is
essential for achieving human-level intelligence. Consequently, AI agents must
demonstrate holistic audio understanding to qualify as generally intelligent.
However, evaluating auditory intelligence comprehensively remains challenging.
To address this gap, we introduce MMAU-Pro, the most comprehensive and
rigorously curated benchmark for assessing audio intelligence in AI systems.
MMAU-Pro contains 5,305 instances, where each instance has one or more audios
paired with human expert-generated question-answer pairs, spanning speech,
sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro
evaluates auditory intelligence across 49 unique skills and multiple complex
dimensions, including long-form audio comprehension, spatial audio reasoning,
multi-audio understanding, among others. All questions are meticulously
designed to require deliberate multi-hop reasoning, including both
multiple-choice and open-ended response formats. Importantly, audio data is
sourced directly ``from the wild" rather than from existing datasets with known
distributions. We evaluate 22 leading open-source and proprietary multimodal AI
models, revealing significant limitations: even state-of-the-art models such as
Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy,
respectively, approaching random performance in multiple categories. Our
extensive analysis highlights specific shortcomings and provides novel
insights, offering actionable perspectives for the community to enhance future
AI systems' progression toward audio general intelligence. The benchmark and
code is available at https://sonalkum.github.io/mmau-pro.
Prompt Orchestration Markup Language
Authors: name: Yuge Zhang, name: Nan Chen, name: Jiahang Xu, name: Yuqing Yang
Published: 2025-08-19 15:37:29

Large Language Models (LLMs) require sophisticated prompting, yet current
practices face challenges in structure, data integration, format sensitivity,
and tooling. Existing methods lack comprehensive solutions for organizing
complex prompts involving diverse data types (documents, tables, images) or
managing presentation variations systematically. To address these gaps, we
introduce POML (Prompt Orchestration Markup Language). POML employs
component-based markup for logical structure (roles, tasks, examples),
specialized tags for seamless data integration, and a CSS-like styling system
to decouple content from presentation, reducing formatting sensitivity. It
includes templating for dynamic prompts and a comprehensive developer toolkit
(IDE support, SDKs) to improve version control and collaboration. We validate
POML through two case studies demonstrating its impact on complex application
integration (PomLink) and accuracy performance (TableQA), as well as a user
study assessing its effectiveness in real-world development scenarios.
Training-Free Text-Guided Color Editing with Multi-Modal Diffusion
  Transformer
Authors: name: Zixin Yin, name: Xili Dai, name: Ling-Hao Chen, name: Deyu Zhou, name: Jianan Wang, name: Duomin Wang, name: Gang Yu, name: Lionel M. Ni, name: Lei Zhang, name: Heung-Yeung Shum
Published: 2025-08-12 17:57:04

Text-guided color editing in images and videos is a fundamental yet unsolved
problem, requiring fine-grained manipulation of color attributes, including
albedo, light source color, and ambient lighting, while preserving physical
consistency in geometry, material properties, and light-matter interactions.
Existing training-free methods offer broad applicability across editing tasks
but struggle with precise color control and often introduce visual
inconsistency in both edited and non-edited regions. In this work, we present
ColorCtrl, a training-free color editing method that leverages the attention
mechanisms of modern Multi-Modal Diffusion Transformers (MM-DiT). By
disentangling structure and color through targeted manipulation of attention
maps and value tokens, our method enables accurate and consistent color
editing, along with word-level control of attribute intensity. Our method
modifies only the intended regions specified by the prompt, leaving unrelated
areas untouched. Extensive experiments on both SD3 and FLUX.1-dev demonstrate
that ColorCtrl outperforms existing training-free approaches and achieves
state-of-the-art performances in both edit quality and consistency.
Furthermore, our method surpasses strong commercial models such as FLUX.1
Kontext Max and GPT-4o Image Generation in terms of consistency. When extended
to video models like CogVideoX, our approach exhibits greater advantages,
particularly in maintaining temporal coherence and editing stability. Finally,
our method also generalizes to instruction-based editing diffusion models such
as Step1X-Edit and FLUX.1 Kontext dev, further demonstrating its versatility.
Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent
  Distillation and Agentic RL
Authors: name: Weizhen Li, name: Jianbo Lin, name: Zhuosong Jiang, name: Jingyi Cao, name: Xinpeng Liu, name: Jiayu Zhang, name: Zhenqiang Huang, name: Qianben Chen, name: Weichen Sun, name: Qiexiang Wang, name: Hongxuan Lu, name: Tianrui Qin, name: Chenghao Zhu, name: Yi Yao, name: Shuying Fan, name: Xiaowan Li, name: Tiannan Wang, name: Pai Liu, name: King Zhu, name: He Zhu, name: Dingfeng Shi, name: Piaohong Wang, name: Yeyi Guan, name: Xiangru Tang, name: Minghao Liu, name: Yuchen Eleanor Jiang, name: Jian Yang, name: Jiaheng Liu, name: Ge Zhang, name: Wangchunshu Zhou
Published: 2025-08-06 17:01:02

Recent advances in large language models (LLMs) and multi-agent systems have
demonstrated remarkable capabilities in complex problem-solving tasks such as
deep research, vibe coding, and mathematical reasoning. However, most existing
multi-agent systems are built upon manual prompt/workflow engineering with
sophisticated agent frameworks, making them computationally inefficient, less
capable, and can not benefit from data-centric learning. In this work, we
introduce Chain-of-Agents (CoA), a novel paradigm of LLM reasoning that enables
native end-to-end complex problem-solving in the same way as a multi-agent
system (i.e., multi-turn problem solving with multiple tools and multiple
agents) within one model. In chain-of-agents problem-solving, the model
dynamically activates different tool agents and role-playing agents to simulate
multi-agent collaboration in an end-to-end fashion. To elicit end-to-end
chain-of-agents problem-solving abilities in LLMs, we introduce a multi-agent
distillation framework to distill state-of-the-art multi-agent systems into
chain-of-agents trajectories for agentic supervised fine-tuning. We then use
agentic reinforcement learning on verifiable agentic tasks to further improve
the models' capabilities on chain-of-agents problem solving. We call the
resulting models Agent Foundation Models (AFMs). Our empirical studies
demonstrate that AFM establishes new state-of-the-art performance across
diverse benchmarks in both web agent and code agent settings. We make the
entire research, including the model weights, code for training and evaluation,
and the training data, fully open-sourced, which offers a solid starting point
for future research on agent models and agentic RL.
OmniTry: Virtual Try-On Anything without Masks
Authors: name: Yutong Feng, name: Linlin Zhang, name: Hengyuan Cao, name: Yiming Chen, name: Xiaoduan Feng, name: Jian Cao, name: Yuxiong Wu, name: Bin Wang
Published: 2025-08-19 08:47:31

Virtual Try-ON (VTON) is a practical and widely-applied task, for which most
of existing works focus on clothes. This paper presents OmniTry, a unified
framework that extends VTON beyond garment to encompass any wearable objects,
e.g., jewelries and accessories, with mask-free setting for more practical
application. When extending to various types of objects, data curation is
challenging for obtaining paired images, i.e., the object image and the
corresponding try-on result. To tackle this problem, we propose a two-staged
pipeline: For the first stage, we leverage large-scale unpaired images, i.e.,
portraits with any wearable items, to train the model for mask-free
localization. Specifically, we repurpose the inpainting model to automatically
draw objects in suitable positions given an empty mask. For the second stage,
the model is further fine-tuned with paired images to transfer the consistency
of object appearance. We observed that the model after the first stage shows
quick convergence even with few paired samples. OmniTry is evaluated on a
comprehensive benchmark consisting of 12 common classes of wearable objects,
with both in-shop and in-the-wild images. Experimental results suggest that
OmniTry shows better performance on both object localization and
ID-preservation compared with existing methods. The code, model weights, and
evaluation benchmark of OmniTry will be made publicly available at
https://omnitry.github.io/.
RotBench: Evaluating Multimodal Large Language Models on Identifying
  Image Rotation
Authors: name: Tianyi Niu, name: Jaemin Cho, name: Elias Stengel-Eskin, name: Mohit Bansal
Published: 2025-08-19 15:58:25

We investigate to what extent Multimodal Large Language Models (MLLMs) can
accurately identify the orientation of input images rotated 0{\deg}, 90{\deg},
180{\deg}, and 270{\deg}. This task demands robust visual reasoning
capabilities to detect rotational cues and contextualize spatial relationships
within images, regardless of their orientation. To evaluate MLLMs on these
abilities, we introduce RotBench -- a 350-image manually-filtered benchmark
comprising lifestyle, portrait, and landscape images. Despite the relatively
simple nature of this task, we show that several state-of-the-art open and
proprietary MLLMs, including GPT-5, o3, and Gemini-2.5-Pro, do not reliably
identify rotation in input images. Providing models with auxiliary information
-- including captions, depth maps, and more -- or using chain-of-thought
prompting offers only small and inconsistent improvements. Our results indicate
that most models are able to reliably identify right-side-up (0{\deg}) images,
while certain models are able to identify upside-down (180{\deg}) images. None
can reliably distinguish between 90{\deg} and 270{\deg}. Simultaneously showing
the image rotated in different orientations leads to moderate performance gains
for reasoning models, while a modified setup using voting improves the
performance of weaker models. We further show that fine-tuning does not improve
models' ability to distinguish 90{\deg} and 270{\deg} rotations, despite
substantially improving the identification of 180{\deg} images. Together, these
results reveal a significant gap between MLLMs' spatial reasoning capabilities
and human perception in identifying rotation.
Unlearning Comparator: A Visual Analytics System for Comparative
  Evaluation of Machine Unlearning Methods
Authors: name: Jaeung Lee, name: Suhyeon Yu, name: Yurim Jang, name: Simon S. Woo, name: Jaemin Jo
Published: 2025-08-18 08:53:53

Machine Unlearning (MU) aims to remove target training data from a trained
model so that the removed data no longer influences the model's behavior,
fulfilling "right to be forgotten" obligations under data privacy laws. Yet, we
observe that researchers in this rapidly emerging field face challenges in
analyzing and understanding the behavior of different MU methods, especially in
terms of three fundamental principles in MU: accuracy, efficiency, and privacy.
Consequently, they often rely on aggregate metrics and ad-hoc evaluations,
making it difficult to accurately assess the trade-offs between methods. To
fill this gap, we introduce a visual analytics system, Unlearning Comparator,
designed to facilitate the systematic evaluation of MU methods. Our system
supports two important tasks in the evaluation process: model comparison and
attack simulation. First, it allows the user to compare the behaviors of two
models, such as a model generated by a certain method and a retrained baseline,
at class-, instance-, and layer-levels to better understand the changes made
after unlearning. Second, our system simulates membership inference attacks
(MIAs) to evaluate the privacy of a method, where an attacker attempts to
determine whether specific data samples were part of the original training set.
We evaluate our system through a case study visually analyzing prominent MU
methods and demonstrate that it helps the user not only understand model
behaviors but also gain insights that can inform the improvement of MU methods.
Reinforcement Learning with Rubric Anchors
Authors: name: Zenan Huang, name: Yihong Zhuang, name: Guoshan Lu, name: Zeyu Qin, name: Haokai Xu, name: Tianyu Zhao, name: Ru Peng, name: Jiaqi Hu, name: Zhanming Shen, name: Xiaomeng Hu, name: Xijun Gu, name: Peiyi Tu, name: Jiaxin Liu, name: Wenyu Chen, name: Yuzhuo Fu, name: Zhiting Fan, name: Yanmei Gu, name: Yuanyuan Wang, name: Zhengkai Yang, name: Jianguo Li, name: Junbo Zhao
Published: 2025-08-18 10:06:08

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a
powerful paradigm for enhancing Large Language Models (LLMs), exemplified by
the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable
signals-such as passing unit tests in code generation or matching correct
answers in mathematical reasoning. While effective, this requirement largely
confines RLVR to domains with automatically checkable outcomes. To overcome
this, we extend the RLVR paradigm to open-ended tasks by integrating
rubric-based rewards, where carefully designed rubrics serve as structured,
model-interpretable criteria for automatic scoring of subjective outputs. We
construct, to our knowledge, the largest rubric reward system to date, with
over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration.
Implementing rubric-based RL is challenging; we tackle these issues with a
clear framework and present an open-sourced Qwen-30B-A3B model with notable
gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended
benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by
+2.4%, while preserving general and reasoning abilities. 2) Our method provides
fine-grained stylistic control, using rubrics as anchors to mitigate the
"AI-like" tone and produce more human-like, expressive responses. We share key
lessons in rubric construction, data selection, and training, and discuss
limitations and future releases.
G-CUT3R: Guided 3D Reconstruction with Camera and Depth Prior
  Integration
Authors: name: Ramil Khafizov, name: Artem Komarichev, name: Ruslan Rakhimov, name: Peter Wonka, name: Evgeny Burnaev
Published: 2025-08-15 10:25:58

We introduce G-CUT3R, a novel feed-forward approach for guided 3D scene
reconstruction that enhances the CUT3R model by integrating prior information.
Unlike existing feed-forward methods that rely solely on input images, our
method leverages auxiliary data, such as depth, camera calibrations, or camera
positions, commonly available in real-world scenarios. We propose a lightweight
modification to CUT3R, incorporating a dedicated encoder for each modality to
extract features, which are fused with RGB image tokens via zero convolution.
This flexible design enables seamless integration of any combination of prior
information during inference. Evaluated across multiple benchmarks, including
3D reconstruction and other multi-view tasks, our approach demonstrates
significant performance improvements, showing its ability to effectively
utilize available priors while maintaining compatibility with varying input
modalities.
When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness
  Methods for LLMs
Authors: name: Mikhail Seleznyov, name: Mikhail Chaichuk, name: Gleb Ershov, name: Alexander Panchenko, name: Elena Tutubalina, name: Oleg Somov
Published: 2025-08-15 10:32:50

Large Language Models (LLMs) are highly sensitive to subtle, non-semantic
variations in prompt phrasing and formatting. In this work, we present the
first systematic evaluation of 5 methods for improving prompt robustness within
a unified experimental framework. We benchmark these techniques on 8 models
from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions
dataset. Our evaluation covers robustness methods from both fine-tuned and
in-context learning paradigms, and tests their generalization against multiple
types of distribution shifts. Finally, we extend our analysis to GPT-4.1 and
DeepSeek V3 to assess frontier models' current robustness to format
perturbations. Our findings offer actionable insights into the relative
effectiveness of these robustness methods, enabling practitioners to make
informed decisions when aiming for stable and reliable LLM performance in
real-world applications. Code:
https://github.com/AIRI-Institute/when-punctuation-matters.
Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning
  Models to Ask for Information
Authors: name: Youcheng Huang, name: Bowen Qin, name: Chen Huang, name: Duanyu Feng, name: Xi Yang, name: Wenqiang Lei
Published: 2025-08-15 06:42:00

Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving
abilities in mathematics, as evaluated by existing benchmarks exclusively on
well-defined problems. However, such evaluation setup constitutes a critical
gap, since a genuine intelligent agent should not only solve problems (as a
math quiz solver), but also be able~to ask for information when the problems
lack sufficient information, enabling proactivity in responding users'
requests. To bridge such gap, we proposes a new dataset consisting of two types
of incomplete problems with diverse contexts. Based on the dataset, our
systematical evaluation of LRMs reveals their inability in proactively asking
for information. In addition, we uncover the behaviors related to overthinking
and hallucination of LRMs, and highlight the potential and challenges of
supervised fine-tuning in learning such ability. We hope to provide new
insights in developing LRMs with genuine intelligence, rather than just solving
problems.
HeroBench: A Benchmark for Long-Horizon Planning and Structured
  Reasoning in Virtual Worlds
Authors: name: Petr Anokhin, name: Roman Khalikov, name: Stefan Rebrikov, name: Viktor Volkov, name: Artyom Sorokin, name: Vincent Bissonnette
Published: 2025-08-18 09:59:02

Large language models (LLMs) have shown remarkable capabilities in isolated
step-by-step reasoning tasks such as mathematics and programming, but their
proficiency in long-horizon planning, where solutions require extended,
structured sequences of interdependent actions, remains underexplored. Existing
benchmarks typically assess LLMs through abstract or low-dimensional
algorithmic tasks, failing to capture the complexity of realistic planning
environments. We introduce HeroBench, a novel benchmark designed specifically
to evaluate long-horizon planning and structured reasoning within complex
RPG-inspired virtual worlds. HeroBench provides a rigorously constructed
dataset of tasks covering a wide range of difficulties, a simulated environment
to execute and validate agent plans, and detailed analytical tools for
evaluating model performance. Tasks challenge models to formulate strategic
plans, efficiently gather resources, master necessary skills, craft equipment,
and defeat adversaries, reflecting practical scenarios' layered dependencies
and constraints. Our extensive evaluation of 25 state-of-the-art LLMs, spanning
both open-source and proprietary models, including the GPT-5 family, reveals
substantial performance disparities rarely observed in conventional reasoning
benchmarks. Detailed error analysis further uncovers specific weaknesses in
current models' abilities to generate robust high-level plans and reliably
execute structured actions. HeroBench thus not only significantly advances the
evaluation of LLM reasoning but also provides a flexible, scalable foundation
for future research into advanced, autonomous planning in virtual environments.
Precise Action-to-Video Generation Through Visual Action Prompts
Authors: name: Yuang Wang, name: Chao Wen, name: Haoyu Guo, name: Sida Peng, name: Minghan Qin, name: Hujun Bao, name: Xiaowei Zhou, name: Ruizhen Hu
Published: 2025-08-18 17:12:28

We present visual action prompts, a unified action representation for
action-to-video generation of complex high-DoF interactions while maintaining
transferable visual dynamics across domains. Action-driven video generation
faces a precision-generality trade-off: existing methods using text, primitive
actions, or coarse masks offer generality but lack precision, while
agent-centric action signals provide precision at the cost of cross-domain
transferability. To balance action precision and dynamic transferability, we
propose to "render" actions into precise visual prompts as domain-agnostic
representations that preserve both geometric precision and cross-domain
adaptability for complex actions; specifically, we choose visual skeletons for
their generality and accessibility. We propose robust pipelines to construct
skeletons from two interaction-rich data sources - human-object interactions
(HOI) and dexterous robotic manipulation - enabling cross-domain training of
action-driven generative models. By integrating visual skeletons into
pretrained video generation models via lightweight fine-tuning, we enable
precise action control of complex interaction while preserving the learning of
cross-domain dynamics. Experiments on EgoVid, RT-1 and DROID demonstrate the
effectiveness of our proposed approach. Project page:
https://zju3dv.github.io/VAP/.
Has GPT-5 Achieved Spatial Intelligence? An Empirical Study
Authors: name: Zhongang Cai, name: Yubo Wang, name: Qingping Sun, name: Ruisi Wang, name: Chenyang Gu, name: Wanqi Yin, name: Zhiqian Lin, name: Zhitao Yang, name: Chen Wei, name: Xuanke Shi, name: Kewang Deng, name: Xiaoyang Han, name: Zukai Chen, name: Jiaqi Li, name: Xiangyu Fan, name: Hanming Deng, name: Lewei Lu, name: Bo Li, name: Ziwei Liu, name: Quan Wang, name: Dahua Lin, name: Lei Yang
Published: 2025-08-18 17:55:17

Multi-modal models have achieved remarkable progress in recent years.
Nevertheless, they continue to exhibit notable limitations in spatial
understanding and reasoning, which are fundamental capabilities to achieving
artificial general intelligence. With the recent release of GPT-5, allegedly
the most powerful AI model to date, it is timely to examine where the leading
models stand on the path toward spatial intelligence. First, we propose a
comprehensive taxonomy of spatial tasks that unifies existing benchmarks and
discuss the challenges in ensuring fair evaluation. We then evaluate
state-of-the-art proprietary and open-source models on eight key benchmarks, at
a cost exceeding one billion total tokens. Our empirical study reveals that (1)
GPT-5 demonstrates unprecedented strength in spatial intelligence, yet (2)
still falls short of human performance across a broad spectrum of tasks.
Moreover, we (3) identify the more challenging spatial intelligence problems
for multi-modal models, and (4) proprietary models do not exhibit a decisive
advantage when facing the most difficult problems. In addition, we conduct a
qualitative evaluation across a diverse set of scenarios that are intuitive for
humans yet fail even the most advanced multi-modal models.