Orionnnn/hub-stats huggingface dataset:
When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with
  Benign Inputs
Authors: name: Bodam Kim, name: Hiskias Dingeto, name: Taeyoun Kwon, name: Dasol Choi, name: DongGeon Lee, name: Haon Park, name: JaeHoon Lee, name: Jongho Shin
Published: 2025-08-05 12:14:01

As large language models become increasingly integrated into daily life,
audio has emerged as a key interface for human-AI interaction. However, this
convenience also introduces new vulnerabilities, making audio a potential
attack surface for adversaries. Our research introduces WhisperInject, a
two-stage adversarial audio attack framework that can manipulate
state-of-the-art audio language models to generate harmful content. Our method
uses imperceptible perturbations in audio inputs that remain benign to human
listeners. The first stage uses a novel reward-based optimization method,
Reinforcement Learning with Projected Gradient Descent (RL-PGD), to guide the
target model to circumvent its own safety protocols and generate harmful native
responses. This native harmful response then serves as the target for Stage 2,
Payload Injection, where we use Projected Gradient Descent (PGD) to optimize
subtle perturbations that are embedded into benign audio carriers, such as
weather queries or greeting messages. Validated under the rigorous
StrongREJECT, LlamaGuard, as well as Human Evaluation safety evaluation
framework, our experiments demonstrate a success rate exceeding 86% across
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Phi-4-Multimodal. Our work demonstrates a
new class of practical, audio-native threats, moving beyond theoretical
exploits to reveal a feasible and covert method for manipulating AI behavior.
VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation
  for Multilingual Long Document Understanding
Authors: name: Jian Chen, name: Ming Li, name: Jihyung Kil, name: Chenguang Wang, name: Tong Yu, name: Ryan Rossi, name: Tianyi Zhou, name: Changyou Chen, name: Ruiyi Zhang
Published: 2025-08-10 21:44:43

Most organizational data in this world are stored as documents, and visual
retrieval plays a crucial role in unlocking the collective intelligence from
all these documents. However, existing benchmarks focus on English-only
document retrieval or only consider multilingual question-answering on a
single-page image. To bridge this gap, we introduce VisR-Bench, a multilingual
benchmark designed for question-driven multimodal retrieval in long documents.
Our benchmark comprises over 35K high-quality QA pairs across 1.2K documents,
enabling fine-grained evaluation of multimodal retrieval. VisR-Bench spans
sixteen languages with three question types (figures, text, and tables),
offering diverse linguistic and question coverage. Unlike prior datasets, we
include queries without explicit answers, preventing models from relying on
superficial keyword matching. We evaluate various retrieval models, including
text-based methods, multimodal encoders, and MLLMs, providing insights into
their strengths and limitations. Our results show that while MLLMs
significantly outperform text-based and multimodal encoder models, they still
struggle with structured tables and low-resource languages, highlighting key
challenges in multilingual visual retrieval.
UserBench: An Interactive Gym Environment for User-Centric Agents
Authors: name: Cheng Qian, name: Zuxin Liu, name: Akshara Prabhakar, name: Zhiwei Liu, name: Jianguo Zhang, name: Haolin Chen, name: Heng Ji, name: Weiran Yao, name: Shelby Heinecke, name: Silvio Savarese, name: Caiming Xiong, name: Huan Wang
Published: 2025-07-29 17:34:12

Large Language Models (LLMs)-based agents have made impressive progress in
reasoning and tool use, enabling them to solve complex tasks. However, their
ability to proactively collaborate with users, especially when goals are vague,
evolving, or indirectly expressed, remains underexplored. To address this gap,
we introduce UserBench, a user-centric benchmark designed to evaluate agents in
multi-turn, preference-driven interactions. UserBench features simulated users
who start with underspecified goals and reveal preferences incrementally,
requiring agents to proactively clarify intent and make grounded decisions with
tools. Our evaluation of leading open- and closed-source LLMs reveals a
significant disconnect between task completion and user alignment. For
instance, models provide answers that fully align with all user intents only
20% of the time on average, and even the most advanced models uncover fewer
than 30% of all user preferences through active interaction. These results
highlight the challenges of building agents that are not just capable task
executors, but true collaborative partners. UserBench offers an interactive
environment to measure and advance this critical capability.
Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via
  Past-Future
Authors: name: Yidong Wang, name: Xin Wang, name: Cunxiang Wang, name: Junfeng Fang, name: Qiufeng Wang, name: Jianing Chu, name: Xuran Meng, name: Shuxun Yang, name: Libo Qin, name: Yue Zhang, name: Wei Ye, name: Shikun Zhang
Published: 2025-08-08 05:25:54

Self-Rewarding Language Models propose an architecture in which the Large
Language Models(LLMs) both generates responses and evaluates its own outputs
via LLM-as-a-Judge prompting, dynamically improving its generative capabilities
through iterative Direct Preference Optimization (DPO). However, our analysis
reveals a critical limitation in existing Self-Rewarding paradigms: the
synchronized improvement of chosen and rejected responses progressively narrows
the representational difference between contrasting samples, undermining
effective preference learning. We propose Temporal Self-Rewarding
Language Models that strategically coordinate past, present, and future model
generations to sustain learning signals. Our dual-phase framework introduces:
(1) Anchored Rejection - fixing rejected responses using the past
initial model's outputs and (2) Future-Guided Chosen - dynamically
curating chosen samples using next-generation model predictions. Extensive
experiments across three model families (Llama, Qwen, Mistral) and different
model sizes (Llama3B/8B/70B) demonstrate significant improvements when trained
with our method compared to Self-Rewarding using same computation resources.
For example, Llama3.1-8B reaches a 29.44 win rate on AlpacaEval 2.0 with our
method, outperforming the Self-Rewarding baseline (19.69) by 9.75. Notably, our
method also demonstrates superior out-of-distribution generalization across
mathematical reasoning (GSM8K), knowledge-based QA (ARC, TruthfulQA), and code
generation (HumanEval) tasks, even though we do not specifically collect such
training data.
Less Is More: Training-Free Sparse Attention with Global Locality for
  Efficient Reasoning
Authors: name: Lijie Yang, name: Zhihao Zhang, name: Arti Jain, name: Shijie Cao, name: Baihong Yuan, name: Yiwei Chen, name: Zhihao Jia, name: Ravi Netravali
Published: 2025-08-09 21:10:33

Large reasoning models achieve strong performance through test-time scaling
but incur substantial computational overhead, particularly from excessive token
generation when processing short input prompts. While sparse attention
mechanisms can reduce latency and memory usage, existing approaches suffer from
significant accuracy degradation due to accumulated errors during
long-generation reasoning. These methods generally require either high token
retention rates or expensive retraining. We introduce LessIsMore, a
training-free sparse attention mechanism for reasoning tasks, which leverages
global attention patterns rather than relying on traditional head-specific
local optimizations. LessIsMore aggregates token selections from local
attention heads with recent contextual information, enabling unified cross-head
token ranking for future decoding layers. This unified selection improves
generalization and efficiency by avoiding the need to maintain separate token
subsets per head. Evaluation across diverse reasoning tasks and benchmarks
shows that LessIsMore preserves -- and in some cases improves -- accuracy while
achieving a 1.1times average decoding speed-up compared to full attention.
Moreover, LessIsMore attends to 2times fewer tokens without accuracy loss,
achieving a 1.13times end-to-end speed-up compared to existing sparse
attention methods.
ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability
Authors: name: Wenhan Liu, name: Xinyu Ma, name: Weiwei Sun, name: Yutao Zhu, name: Yuchen Li, name: Dawei Yin, name: Zhicheng Dou
Published: 2025-08-09 17:26:18

Large Language Model (LLM) based listwise ranking has shown superior
performance in many passage ranking tasks. With the development of Large
Reasoning Models, many studies have demonstrated that step-by-step reasoning
during test-time helps improve listwise ranking performance. However, due to
the scarcity of reasoning-intensive training data, existing rerankers perform
poorly in many complex ranking scenarios and the ranking ability of
reasoning-intensive rerankers remains largely underdeveloped. In this paper, we
first propose an automated reasoning-intensive training data synthesis
framework, which sources training queries and passages from diverse domains and
applies DeepSeek-R1 to generate high-quality training labels. A
self-consistency data filtering mechanism is designed to ensure the data
quality. To empower the listwise reranker with strong reasoning ability, we
further propose a two-stage post-training approach, which includes a cold-start
supervised fine-tuning (SFT) stage for reasoning pattern learning and a
reinforcement learning (RL) stage for further ranking ability enhancement.
During the RL stage, based on the nature of listwise ranking, we design a
multi-view ranking reward, which is more effective than a ranking metric-based
reward. Extensive experiments demonstrate that our trained reasoning-intensive
reranker ReasonRank outperforms existing baselines significantly and
also achieves much lower latency than pointwise reranker Rank1. Through
further experiments, our ReasonRank has achieved state-of-the-art (SOTA)
performance 40.6 on the BRIGHT
leaderboard\footnote{https://brightbenchmark.github.io/.} Our codes are
available at https://github.com/8421BCD/ReasonRank.
Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving
  Clipping Policy Optimization
Authors: name: Zhenpeng Su, name: Leiyu Pan, name: Xue Bai, name: Dening Liu, name: Guanting Dong, name: Jiaming Huang, name: Wenping Hu, name: Guorui Zhou
Published: 2025-08-11 05:17:51

We present Klear-Reasoner, a model with long reasoning capabilities that
demonstrates careful deliberation during problem solving, achieving outstanding
performance across multiple benchmarks. Although there are already many
excellent works related to inference models in the current community, there are
still many problems with reproducing high-performance inference models due to
incomplete disclosure of training details. This report provides an in-depth
analysis of the reasoning model, covering the entire post-training workflow
from data preparation and long Chain-of-Thought supervised fine-tuning (long
CoT SFT) to reinforcement learning (RL), along with detailed ablation studies
for each experimental component. For SFT data, our experiments show that a
small number of high-quality data sources are more effective than a large
number of diverse data sources, and that difficult samples can achieve better
results without accuracy filtering. In addition, we investigate two key issues
with current clipping mechanisms in RL: Clipping suppresses critical
exploration signals and ignores suboptimal trajectories. To address these
challenges, we propose Gradient-Preserving clipping Policy Optimization (GPPO)
that gently backpropagates gradients from clipped tokens. GPPO not only
enhances the model's exploration capacity but also improves its efficiency in
learning from negative samples. Klear-Reasoner exhibits exceptional reasoning
abilities in mathematics and programming, scoring 90.5\% on AIME 2024, 83.2\%
on AIME 2025, 66.0\% on LiveCodeBench V5 and 58.1\% on LiveCodeBench V6.
Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided
  Region Control
Authors: name: Zeqian Long, name: Mingzhe Zheng, name: Kunyu Feng, name: Xinhua Zhang, name: Hongyu Liu, name: Harry Yang, name: Linfeng Zhang, name: Qifeng Chen, name: Yue Ma
Published: 2025-08-11 16:10:00

While recent flow-based image editing models demonstrate general-purpose
capabilities across diverse tasks, they often struggle to specialize in
challenging scenarios -- particularly those involving large-scale shape
transformations. When performing such structural edits, these methods either
fail to achieve the intended shape change or inadvertently alter non-target
regions, resulting in degraded background quality. We propose
Follow-Your-Shape, a training-free and mask-free framework that supports
precise and controllable editing of object shapes while strictly preserving
non-target content. Motivated by the divergence between inversion and editing
trajectories, we compute a Trajectory Divergence Map (TDM) by comparing
token-wise velocity differences between the inversion and denoising paths. The
TDM enables precise localization of editable regions and guides a Scheduled KV
Injection mechanism that ensures stable and faithful editing. To facilitate a
rigorous evaluation, we introduce ReShapeBench, a new benchmark comprising 120
new images and enriched prompt pairs specifically curated for shape-aware
editing. Experiments demonstrate that our method achieves superior editability
and visual fidelity, particularly in tasks requiring large-scale shape
replacement.
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of
  Deep-Research Agent
Authors: name: Zijian Chen, name: Xueguang Ma, name: Shengyao Zhuang, name: Ping Nie, name: Kai Zou, name: Andrew Liu, name: Joshua Green, name: Kshama Patel, name: Ruoxi Meng, name: Mingyi Su, name: Sahel Sharifymoghaddam, name: Yanxi Li, name: Haoran Hong, name: Xinyu Shi, name: Xuye Liu, name: Nandan Thakur, name: Crystina Zhang, name: Luyu Gao, name: Wenhu Chen, name: Jimmy Lin
Published: 2025-08-08 17:55:11

Deep-Research agents, which integrate large language models (LLMs) with
search tools, have shown success in improving the effectiveness of handling
complex queries that require iterative search planning and reasoning over
search results. Evaluations on current benchmarks like BrowseComp relies on
black-box live web search APIs, have notable limitations in (1) fairness:
dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep
research methods; (2) transparency: lack of control over the document corpus
makes it difficult to isolate retriever contributions. In other words, the
current evaluations may compare a complete deep research system at a given
time, but they do not foster well-controlled experiments to provide insights
into the capability of underlying deep research LLMs. To address these
challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp,
employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus
includes human-verified supporting documents and mined challenging negatives,
enabling controlled experimentation. The benchmark is shown to be effective in
distinguishing the performance of deep research systems. For instance, the
open-source model Search-R1, when paired with the BM25 retriever, achieves
3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with
the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with
fewer search calls. This benchmark allows comprehensive evaluation and
disentangled analysis of deep research agents and retrieval methods, fostering
insights into retrieval effectiveness, citation accuracy, and context
engineering in Deep-Research system.
WideSearch: Benchmarking Agentic Broad Info-Seeking
Authors: name: Ryan Wong, name: Jiawei Wang, name: Junjie Zhao, name: Li Chen, name: Yan Gao, name: Long Zhang, name: Xuan Zhou, name: Zuo Wang, name: Kai Xiang, name: Ge Zhang, name: Wenhao Huang, name: Yang Wang, name: Ke Wang
Published: 2025-08-11 14:03:09

From professional research to everyday planning, many tasks are bottlenecked
by wide-scale information seeking, which is more repetitive than cognitively
complex. With the rapid development of Large Language Models (LLMs), automated
search agents powered by LLMs offer a promising solution to liberate humans
from this tedious work. However, the capability of these agents to perform such
"wide-context" collection reliably and completely remains largely unevaluated
due to a lack of suitable benchmarks. To bridge this gap, we introduce
WideSearch, a new benchmark engineered to evaluate agent reliability on these
large-scale collection tasks. The benchmark features 200 manually curated
questions (100 in English, 100 in Chinese) from over 15 diverse domains,
grounded in real user queries. Each task requires agents to collect large-scale
atomic information, which could be verified one by one objectively, and arrange
it into a well-organized output. A rigorous five-stage quality control pipeline
ensures the difficulty, completeness, and verifiability of the dataset. We
benchmark over 10 state-of-the-art agentic search systems, including
single-agent, multi-agent frameworks, and end-to-end commercial systems. Most
systems achieve overall success rates near 0\%, with the best performer
reaching just 5\%. However, given sufficient time, cross-validation by multiple
human testers can achieve a near 100\% success rate. These results demonstrate
that present search agents have critical deficiencies in large-scale
information seeking, underscoring urgent areas for future research and
development in agentic search. Our dataset, evaluation pipeline, and benchmark
results have been publicly released at https://widesearch-seed.github.io/
MolmoAct: Action Reasoning Models that can Reason in Space
Authors: name: Jason Lee, name: Jiafei Duan, name: Haoquan Fang, name: Yuquan Deng, name: Shuo Liu, name: Boyang Li, name: Bohan Fang, name: Jieyu Zhang, name: Yi Ru Wang, name: Sangho Lee, name: Winson Han, name: Wilbert Pumacay, name: Angelica Wu, name: Rose Hendrix, name: Karen Farley, name: Eli VanderBilt, name: Ali Farhadi, name: Dieter Fox, name: Ranjay Krishna
Published: 2025-08-11 12:32:45

Reasoning is central to purposeful action, yet most robotic foundation models
map perception and instructions directly to control, which limits adaptability,
generalization, and semantic grounding. We introduce Action Reasoning Models
(ARMs), a class of vision-language-action models that integrate perception,
planning, and control through a structured three-stage pipeline. Our model,
MolmoAct, encodes observations and instructions into depth-aware perception
tokens, generates mid-level spatial plans as editable trajectory traces, and
predicts precise low-level actions, enabling explainable and steerable
behavior. MolmoAct-7B-D achieves strong performance across simulation and
real-world settings: 70.5% zero-shot accuracy on SimplerEnv Visual Matching
tasks, surpassing closed-source Pi-0 and GR00T N1; 86.6% average success on
LIBERO, including an additional 6.3% gain over ThinkAct on long-horizon tasks;
and in real-world fine-tuning, an additional 10% (single-arm) and an additional
22.7% (bimanual) task progression over Pi-0-FAST. It also outperforms baselines
by an additional 23.3% on out-of-distribution generalization and achieves top
human-preference scores for open-ended instruction following and trajectory
steering. Furthermore, we release, for the first time, the MolmoAct Dataset --
a mid-training robot dataset comprising over 10,000 high quality robot
trajectories across diverse scenarios and tasks. Training with this dataset
yields an average 5.5% improvement in general performance over the base model.
We release all model weights, training code, our collected dataset, and our
action reasoning dataset, establishing MolmoAct as both a state-of-the-art
robotics foundation model and an open blueprint for building ARMs that
transform perception into purposeful action through structured reasoning.
Blogpost: https://allenai.org/blog/molmoact
Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant
  Safeguards into Open-Weight LLMs
Authors: name: Kyle O'Brien, name: Stephen Casper, name: Quentin Anthony, name: Tomek Korbak, name: Robert Kirk, name: Xander Davies, name: Ishan Mishra, name: Geoffrey Irving, name: Yarin Gal, name: Stella Biderman
Published: 2025-08-08 17:59:47

Open-weight AI systems offer unique benefits, including enhanced
transparency, open research, and decentralized access. However, they are
vulnerable to tampering attacks which can efficiently elicit harmful behaviors
by modifying weights or activations. Currently, there is not yet a robust
science of open-weight model risk management. Existing safety fine-tuning
methods and other post-training techniques have struggled to make LLMs
resistant to more than a few dozen steps of adversarial fine-tuning. In this
paper, we investigate whether filtering text about dual-use topics from
training data can prevent unwanted capabilities and serve as a more
tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable
data filtering and show that it offers a tractable and effective method for
minimizing biothreat proxy knowledge in LLMs. We pretrain multiple
6.9B-parameter models from scratch and find that they exhibit substantial
resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M
tokens of biothreat-related text -- outperforming existing post-training
baselines by over an order of magnitude -- with no observed degradation to
unrelated capabilities. However, while filtered models lack internalized
dangerous knowledge, we find that they can still leverage such information when
it is provided in context (e.g., via search tool augmentation), demonstrating a
need for a defense-in-depth approach. Overall, these findings help to establish
pretraining data curation as a promising layer of defense for open-weight AI
systems.
OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks
Authors: name: Zixuan Wang, name: Dingming Li, name: Hongxing Li, name: Shuo Chen, name: Yuchen Yan, name: Wenqi Zhang, name: Yongliang Shen, name: Weiming Lu, name: Jun Xiao, name: Yueting Zhuang
Published: 2025-08-07 17:54:15

Large language models excel at abstract reasoning but their capacity for
embodied agent reasoning remains largely unexplored. We present OmniEAR, a
comprehensive framework for evaluating how language models reason about
physical interactions, tool usage, and multi-agent coordination in embodied
tasks. Unlike existing benchmarks that provide predefined tool sets or explicit
collaboration directives, OmniEAR requires agents to dynamically acquire
capabilities and autonomously determine coordination strategies based on task
demands. Through text-based environment representation, we model continuous
physical properties and complex spatial relationships across 1,500 scenarios
spanning household and industrial domains. Our systematic evaluation reveals
severe performance degradation when models must reason from constraints: while
achieving 85-96% success with explicit instructions, performance drops to
56-85% for tool reasoning and 63-85% for implicit collaboration, with compound
tasks showing over 50% failure rates. Surprisingly, complete environmental
information degrades coordination performance, indicating models cannot filter
task-relevant constraints. Fine-tuning improves single-agent tasks dramatically
(0.6% to 76.3%) but yields minimal multi-agent gains (1.5% to 5.5%), exposing
fundamental architectural limitations. These findings demonstrate that embodied
reasoning poses fundamentally different challenges than current models can
address, establishing OmniEAR as a rigorous benchmark for evaluating and
advancing embodied AI systems. Our code and data are included in the
supplementary materials and will be open-sourced upon acceptance.