Orionnnn/hub-stats huggingface dataset:
InfoAgent: Advancing Autonomous Information-Seeking Agents
Authors: name: Gongrui Zhang, name: Jialiang Zhu, name: Ruiqi Yang, name: Kai Qiu, name: Miaosen Zhang, name: Zhirong Wu, name: Qi Dai, name: Bei Liu, name: Chong Luo, name: Zhengyuan Yang, name: Linjie Li, name: Lijuan Wang, name: Weizhu Chen, name: Yuan Zhang, name: Xin Li, name: Zhaoyi Liu, name: Xin Geng, name: Baining Guo
Published: 2025-09-29 17:59:57

Building Large Language Model agents that expand their capabilities by
interacting with external tools represents a new frontier in AI research and
applications. In this paper, we introduce InfoAgent, a deep research agent
powered by an innovative data synthesis pipeline and orchestrated web search
tools. To construct challenging, hard-to-find queries,we build entity trees and
apply sub-tree sampling with entity fuzzification to systematically increase
question difficulty. Unlike prior work that relies heavily on commercial search
tools, we develop a dedicated self-hosted search infrastructure, enhancing
transparency of agent environments and facilitating further advancement of
agent capacity. We evaluate the effectiveness of our data pipeline by measuring
the average number of tool calls required to correctly answer a question, and
also show that our agent yields better performance when equipped with our
tools. Our InfoAgent is post-trained from Qwen3-14B using a two-stage
recipe: cold-start supervised finetuning to instill long-horizon search
behaviors, followed by reinforcement learning which significantly improves
reasoning-driven tool use. With our methods, InfoAgent achieves 15.3\% accuracy
on BrowseComp, 29.2\% on BrowseComp-ZH, and 40.4\% on Xbench-DS, outperforming
prior open-source deep research agents such as WebSailor-72B and DeepDive-32B.
Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional
  Video Generation
Authors: name: Agneet Chatterjee, name: Rahim Entezari, name: Maksym Zhuravinskyi, name: Maksim Lapin, name: Reshinth Adithyan, name: Amit Raj, name: Chitta Baral, name: Yezhou Yang, name: Varun Jampani
Published: 2025-09-30 17:22:18

Recent advances in video generation have enabled high-fidelity video
synthesis from user provided prompts. However, existing models and benchmarks
fail to capture the complexity and requirements of professional video
generation. Towards that goal, we introduce Stable Cinemetrics, a structured
evaluation framework that formalizes filmmaking controls into four
disentangled, hierarchical taxonomies: Setup, Event, Lighting, and Camera.
Together, these taxonomies define 76 fine-grained control nodes grounded in
industry practices. Using these taxonomies, we construct a benchmark of prompts
aligned with professional use cases and develop an automated pipeline for
prompt categorization and question generation, enabling independent evaluation
of each control dimension. We conduct a large-scale human study spanning 10+
models and 20K videos, annotated by a pool of 80+ film professionals. Our
analysis, both coarse and fine-grained reveal that even the strongest current
models exhibit significant gaps, particularly in Events and Camera-related
controls. To enable scalable evaluation, we train an automatic evaluator, a
vision-language model aligned with expert annotations that outperforms existing
zero-shot baselines. SCINE is the first approach to situate professional video
generation within the landscape of video generative models, introducing
taxonomies centered around cinematic controls and supporting them with
structured evaluation pipelines and detailed analyses to guide future research.
Humanline: Online Alignment as Perceptual Loss
Authors: name: Sijia Liu, name: Niklas Muennighoff, name: Kawin Ethayarajh
Published: 2025-09-29 02:41:16

Online alignment (e.g., GRPO) is generally more performant than offline
alignment (e.g., DPO) -- but why? Drawing on prospect theory from behavioral
economics, we propose a human-centric explanation. We prove that online
on-policy sampling better approximates the human-perceived distribution of what
the model can produce, and PPO/GRPO-style clipping -- originally introduced to
just stabilize training -- recovers a perceptual bias in how humans perceive
probability. In this sense, PPO/GRPO act as perceptual losses already. Our
theory further suggests that the online/offline dichotomy is itself incidental
to maximizing human utility, since we can achieve the same effect by
selectively training on any data in a manner that mimics human perception,
rather than restricting ourselves to online on-policy data. Doing so would
allow us to post-train more quickly, cheaply, and flexibly without sacrificing
performance. To this end, we propose a design pattern that explicitly
incorporates perceptual distortions of probability into objectives like
DPO/KTO/GRPO, creating humanline variants of them. Surprisingly, we find that
these humanline variants, even when trained with offline off-policy data, can
match the performance of their online counterparts on both verifiable and
unverifiable tasks.
Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with
  LLMs
Authors: name: Chenxing Wei, name: Hong Wang, name: Ying He, name: Fei Yu, name: Yao Shu
Published: 2025-09-27 07:46:15

Large Language Models (LLMs) employ multi-turn interaction as a fundamental
paradigm for completing complex tasks. However, their performance often
degrades in extended interactions, as they are typically trained on static,
single-turn data, which hinders their ability to adapt to real-time user
feedback. To address this limitation, we first propose a new paradigm:
Test-Time Policy Adaptation for Multi-Turn Interactions (T2PAM), which utilizes
user feedback from the ongoing interaction as a reward signal to estimate a
latent optimal policy aligned with user preferences, then updates a small
subset of parameters to steer the model toward this policy, ultimately enabling
efficient in-conversation self-correction. We then introduce Optimum-Referenced
One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes T2PAM.
ROSA guides the model parameters toward a theoretical optimal policy in a
single, efficient update step, avoiding costly iterative gradient-based
optimization and minimizing computational overhead. We provide a rigorous
theoretical analysis guaranteeing that the policy of ROSA converges to the
preference of user as the number of interactions increases. Extensive
experiments on challenging benchmark demonstrate that ROSA achieves significant
improvements in both task effectiveness and efficiency.
More Thought, Less Accuracy? On the Dual Nature of Reasoning in
  Vision-Language Models
Authors: name: Xinyu Tian, name: Shu Zou, name: Zhaoyuan Yang, name: Mengqi He, name: Fabian Waschkowski, name: Lukas Wesemann, name: Peter Tu, name: Jing Zhang
Published: 2025-09-30 06:37:47

Reasoning has emerged as a pivotal capability in Large Language Models
(LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy
Optimization (GRPO), these models are able to solve complex tasks such as
mathematics and code generation. Building on these advances, recent research
has sought to extend reasoning to Vision-Language Models (VLMs), yielding
promising results across diverse visual tasks. Despite this progress, our study
uncovers the dual nature of multimodal reasoning: while it substantially
enhances logical inference and facilitates performance on challenging problems,
it may gradually impair perceptual grounding, leading to recognition failures
on otherwise basic visual questions. Through further analysis, we attribute
this phenomenon to visual forgetting, wherein prolonged reasoning causes the
model to increasingly disregard visual input. To address this, we propose
Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that
explicitly steers the reasoning process toward visually grounded trajectories.
Our result model, VAPO-Thinker-7B, significantly strengthens the model's
reliance on visual information and achieves new state-of-the-art results on a
wide range of established benchmarks. Project page:
https://xytian1008.github.io/VAPO/
MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP
  Use
Authors: name: Zijian Wu, name: Xiangyan Liu, name: Xinyuan Zhang, name: Lingjun Chen, name: Fanqing Meng, name: Lingxiao Du, name: Yiran Zhao, name: Fanshi Zhang, name: Yaoqi Ye, name: Jiawei Wang, name: Zirui Wang, name: Jinjie Ni, name: Yufan Yang, name: Arvin Xu, name: Michael Qizhe Shieh
Published: 2025-09-28 17:53:27

MCP standardizes how LLMs interact with external systems, forming the
foundation for general agents. However, existing MCP benchmarks remain narrow
in scope: they focus on read-heavy tasks or tasks with limited interaction
depth, and fail to capture the complexity and realism of real-world workflows.
To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP
use in a more realistic and comprehensive manner. It consists of 127
high-quality tasks collaboratively created by domain experts and AI agents.
Each task begins with a curated initial state and includes a programmatic
script for automatic verification. These tasks demand richer and more diverse
interactions with the environment, involving a broad range of create, read,
update, and delete (CRUD) operations. We conduct a comprehensive evaluation of
cutting-edge LLMs using a minimal agent framework that operates in a
tool-calling loop. Empirical results show that the best-performing model,
gpt-5-medium, reaches only 52.56\% pass@1 and 33.86\% pass^4, while other
widely regarded strong models, including claude-sonnet-4 and o3, fall below
30\% pass@1 and 15\% pass^4. On average, LLMs require 16.2 execution
turns and 17.4 tool calls per task, significantly surpassing those in
previous MCP benchmarks and highlighting the stress-testing nature of MCPMark.
OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost
  Always!
Authors: name: Jingdi Lei, name: Varun Gumma, name: Rishabh Bhardwaj, name: Seok Min Lim, name: Chuan Li, name: Amir Zadeh, name: Soujanya Poria
Published: 2025-09-30 16:39:17

Large Language Model (LLM) safety is one of the most pressing challenges for
enabling wide-scale deployment. While most studies and global discussions focus
on generic harms, such as models assisting users in harming themselves or
others, enterprises face a more fundamental concern: whether LLM-based agents
are safe for their intended use case. To address this, we introduce operational
safety, defined as an LLM's ability to appropriately accept or refuse user
queries when tasked with a specific purpose. We further propose OffTopicEval,
an evaluation suite and benchmark for measuring operational safety both in
general and within specific agentic use cases. Our evaluations on six model
families comprising 20 open-weight LLMs reveal that while performance varies
across models, all of them remain highly operationally unsafe. Even the
strongest models -- Qwen-3 (235B) with 77.77\% and Mistral (24B) with 79.96\%
-- fall far short of reliable operational safety, while GPT models plateau in
the 62--73\% range, Phi achieves only mid-level scores (48--70\%), and Gemma
and Llama-3 collapse to 39.53\% and 23.84\%, respectively. While operational
safety is a core model alignment issue, to suppress these failures, we propose
prompt-based steering methods: query grounding (Q-ground) and system-prompt
grounding (P-ground), which substantially improve OOD refusal. Q-ground
provides consistent gains of up to 23\%, while P-ground delivers even larger
boosts, raising Llama-3.3 (70B) by 41\% and Qwen-3 (30B) by 27\%. These results
highlight both the urgent need for operational safety interventions and the
promise of prompt-based steering as a first step toward more reliable LLM-based
agents.
IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
Authors: name: Jiayi Guo, name: Chuanhao Yan, name: Xingqian Xu, name: Yulin Wang, name: Kai Wang, name: Gao Huang, name: Humphrey Shi
Published: 2025-09-30 13:27:03

Ensuring precise multimodal alignment between diffusion-generated images and
input prompts has been a long-standing challenge. Earlier works finetune
diffusion weight using high-quality preference data, which tends to be limited
and difficult to scale up. Recent editing-based methods further refine local
regions of generated images but may compromise overall image quality. In this
work, we propose Implicit Multimodal Guidance (IMG), a novel
re-generation-based multimodal alignment framework that requires no extra data
or editing operations. Specifically, given a generated image and its prompt,
IMG a) utilizes a multimodal large language model (MLLM) to identify
misalignments; b) introduces an Implicit Aligner that manipulates diffusion
conditioning features to reduce misalignments and enable re-generation; and c)
formulates the re-alignment goal into a trainable objective, namely Iteratively
Updated Preference Objective. Extensive qualitative and quantitative
evaluations on SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing
alignment methods. Furthermore, IMG acts as a flexible plug-and-play adapter,
seamlessly enhancing prior finetuning-based alignment methods. Our code will be
available at https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment.
Knowledge Homophily in Large Language Models
Authors: name: Utkarsh Sahu, name: Zhisheng Qi, name: Mahantesh Halappanavar, name: Nedim Lipka, name: Ryan A. Rossi, name: Franck Dernoncourt, name: Yu Zhang, name: Yao Ma, name: Yu Wang
Published: 2025-09-28 09:40:27

Large Language Models (LLMs) have been increasingly studied as neural
knowledge bases for supporting knowledge-intensive applications such as
question answering and fact checking. However, the structural organization of
their knowledge remains unexplored. Inspired by cognitive neuroscience
findings, such as semantic clustering and priming, where knowing one fact
increases the likelihood of recalling related facts, we investigate an
analogous knowledge homophily pattern in LLMs. To this end, we map LLM
knowledge into a graph representation through knowledge checking at both the
triplet and entity levels. After that, we analyze the knowledgeability
relationship between an entity and its neighbors, discovering that LLMs tend to
possess a similar level of knowledge about entities positioned closer in the
graph. Motivated by this homophily principle, we propose a Graph Neural Network
(GNN) regression model to estimate entity-level knowledgeability scores for
triplets by leveraging their neighborhood scores. The predicted
knowledgeability enables us to prioritize checking less well-known triplets,
thereby maximizing knowledge coverage under the same labeling budget. This not
only improves the efficiency of active labeling for fine-tuning to inject
knowledge into LLMs but also enhances multi-hop path retrieval in
reasoning-intensive question answering.
TAU: A Benchmark for Cultural Sound Understanding Beyond Semantics
Authors: name: Yi-Cheng Lin, name: Yu-Hua Chen, name: Jia-Kai Dong, name: Yueh-Hsuan Huang, name: Szu-Chi Chen, name: Yu-Chen Chen, name: Chih-Yao Chen, name: Yu-Jung Lin, name: Yu-Ling Chen, name: Zih-Yu Chen, name: I-Ning Tsai, name: Hsiu-Hsuan Wang, name: Ho-Lam Chung, name: Ke-Han Lu, name: Hung-yi Lee
Published: 2025-09-30 14:40:45

Large audio-language models are advancing rapidly, yet most evaluations
emphasize speech or globally sourced sounds, overlooking culturally distinctive
cues. This gap raises a critical question: can current models generalize to
localized, non-semantic audio that communities instantly recognize but
outsiders do not? To address this, we present TAU (Taiwan Audio Understanding),
a benchmark of everyday Taiwanese "soundmarks." TAU is built through a pipeline
combining curated sources, human editing, and LLM-assisted question generation,
producing 702 clips and 1,794 multiple-choice items that cannot be solved by
transcripts alone. Experiments show that state-of-the-art LALMs, including
Gemini 2.5 and Qwen2-Audio, perform far below local humans. TAU demonstrates
the need for localized benchmarks to reveal cultural blind spots, guide more
equitable multimodal evaluation, and ensure models serve communities beyond the
global mainstream.
Who's Your Judge? On the Detectability of LLM-Generated Judgments
Authors: name: Dawei Li, name: Zhen Tan, name: Chengshuai Zhao, name: Bohan Jiang, name: Baixiang Huang, name: Pingchuan Ma, name: Abdullah Alnaibari, name: Kai Shu, name: Huan Liu
Published: 2025-09-29 17:54:57

Large Language Model (LLM)-based judgments leverage powerful LLMs to
efficiently evaluate candidate content and provide judgment scores. However,
the inherent biases and vulnerabilities of LLM-generated judgments raise
concerns, underscoring the urgent need for distinguishing them in sensitive
scenarios like academic peer reviewing. In this work, we propose and formalize
the task of judgment detection and systematically investigate the detectability
of LLM-generated judgments. Unlike LLM-generated text detection, judgment
detection relies solely on judgment scores and candidates, reflecting
real-world scenarios where textual feedback is often unavailable in the
detection process. Our preliminary analysis shows that existing LLM-generated
text detection methods perform poorly given their incapability to capture the
interaction between judgment scores and candidate content -- an aspect crucial
for effective judgment detection. Inspired by this, we introduce
J-Detector, a lightweight and transparent neural detector augmented
with explicitly extracted linguistic and LLM-enhanced features to link LLM
judges' biases with candidates' properties for accurate detection. Experiments
across diverse datasets demonstrate the effectiveness of J-Detector
and show how its interpretability enables quantifying biases in LLM judges.
Finally, we analyze key factors affecting the detectability of LLM-generated
judgments and validate the practical utility of judgment detection in
real-world scenarios.
OceanGym: A Benchmark Environment for Underwater Embodied Agents
Authors: name: Yida Xue, name: Mingjun Mao, name: Xiangyuan Ru, name: Yuqi Zhu, name: Baochang Ren, name: Shuofei Qiao, name: Mengru Wang, name: Shumin Deng, name: Xinyu An, name: Ningyu Zhang, name: Ying Chen, name: Huajun Chen
Published: 2025-09-30 17:09:32

We introduce OceanGym, the first comprehensive benchmark for ocean underwater
embodied agents, designed to advance AI in one of the most demanding real-world
environments. Unlike terrestrial or aerial domains, underwater settings present
extreme perceptual and decision-making challenges, including low visibility,
dynamic ocean currents, making effective agent deployment exceptionally
difficult. OceanGym encompasses eight realistic task domains and a unified
agent framework driven by Multi-modal Large Language Models (MLLMs), which
integrates perception, memory, and sequential decision-making. Agents are
required to comprehend optical and sonar data, autonomously explore complex
environments, and accomplish long-horizon objectives under these harsh
conditions. Extensive experiments reveal substantial gaps between
state-of-the-art MLLM-driven agents and human experts, highlighting the
persistent difficulty of perception, planning, and adaptability in ocean
underwater environments. By providing a high-fidelity, rigorously designed
platform, OceanGym establishes a testbed for developing robust embodied AI and
transferring these capabilities to real-world autonomous ocean underwater
vehicles, marking a decisive step toward intelligent agents capable of
operating in one of Earth's last unexplored frontiers. The code and data are
available at https://github.com/OceanGPT/OceanGym.
Regression Language Models for Code
Authors: name: Yash Akhauri, name: Xingyou Song, name: Arissa Wongpanich, name: Bryan Lewandowski, name: Mohamed S. Abdelfattah
Published: 2025-09-30 16:25:23

We study code-to-metric regression: predicting numeric outcomes of code
executions, a challenging task due to the open-ended nature of programming
languages. While prior methods have resorted to heavy and domain-specific
feature engineering, we show that a single unified Regression Language Model
(RLM) can simultaneously predict directly from text, (i) the memory footprint
of code across multiple high-level languages such as Python and C++, (ii) the
latency of Triton GPU kernels, and (iii) the accuracy and speed of trained
neural networks represented in ONNX. In particular, a relatively small 300M
parameter RLM initialized from T5Gemma, obtains > 0.9 Spearman-rank on
competitive programming submissions from APPS, and a single unified model
achieves > 0.5 average Spearman-rank across 17 separate languages from CodeNet.
Furthermore, the RLM can obtain the highest average Kendall-Tau of 0.46 on five
classic NAS design spaces previously dominated by graph neural networks, and
simultaneously predict architecture latencies on numerous hardware platforms.
MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
Authors: name: Chenhui Zhu, name: Yilu Wu, name: Shuai Wang, name: Gangshan Wu, name: Limin Wang
Published: 2025-09-30 15:26:04

Image-to-video generation has made remarkable progress with the advancements
in diffusion models, yet generating videos with realistic motion remains highly
challenging. This difficulty arises from the complexity of accurately modeling
motion, which involves capturing physical constraints, object interactions, and
domain-specific dynamics that are not easily generalized across diverse
scenarios. To address this, we propose MotionRAG, a retrieval-augmented
framework that enhances motion realism by adapting motion priors from relevant
reference videos through Context-Aware Motion Adaptation (CAMA). The key
technical innovations include: (i) a retrieval-based pipeline extracting
high-level motion features using video encoder and specialized resamplers to
distill semantic motion representations; (ii) an in-context learning approach
for motion adaptation implemented through a causal transformer architecture;
(iii) an attention-based motion injection adapter that seamlessly integrates
transferred motion features into pretrained video diffusion models. Extensive
experiments demonstrate that our method achieves significant improvements
across multiple domains and various base models, all with negligible
computational overhead during inference. Furthermore, our modular design
enables zero-shot generalization to new domains by simply updating the
retrieval database without retraining any components. This research enhances
the core capability of video generation systems by enabling the effective
retrieval and transfer of motion priors, facilitating the synthesis of
realistic motion dynamics.