Orionnnn/hub-stats huggingface dataset:
DyePack: Provably Flagging Test Set Contamination in LLMs Using
  Backdoors
Authors: name: Yize Cheng, name: Wenxiao Wang, name: Mazda Moayeri, name: Soheil Feizi
Published: 2025-05-29 02:22:14

Open benchmarks are essential for evaluating and advancing large language
models, offering reproducibility and transparency. However, their accessibility
makes them likely targets of test set contamination. In this work, we introduce
DyePack, a framework that leverages backdoor attacks to identify models that
used benchmark test sets during training, without requiring access to the loss,
logits, or any internal details of the model. Like how banks mix dye packs with
their money to mark robbers, DyePack mixes backdoor samples with the test data
to flag models that trained on it. We propose a principled design incorporating
multiple backdoors with stochastic targets, enabling exact false positive rate
(FPR) computation when flagging every model. This provably prevents false
accusations while providing strong evidence for every detected case of
contamination. We evaluate DyePack on five models across three datasets,
covering both multiple-choice and open-ended generation tasks. For
multiple-choice questions, it successfully detects all contaminated models with
guaranteed FPRs as low as 0.000073% on MMLU-Pro and 0.000017% on Big-Bench-Hard
using eight backdoors. For open-ended generation tasks, it generalizes well and
identifies all contaminated models on Alpaca with a guaranteed false positive
rate of just 0.127% using six backdoors.
VAU-R1: Advancing Video Anomaly Understanding via Reinforcement
  Fine-Tuning
Authors: name: Liyun Zhu, name: Qixiang Chen, name: Xi Shen, name: Xiaodong Cun
Published: 2025-05-29 14:48:10

Video Anomaly Understanding (VAU) is essential for applications such as smart
cities, security surveillance, and disaster alert systems, yet remains
challenging due to its demand for fine-grained spatio-temporal perception and
robust reasoning under ambiguity. Despite advances in anomaly detection,
existing methods often lack interpretability and struggle to capture the causal
and contextual aspects of abnormal events. This limitation is further
compounded by the absence of comprehensive benchmarks for evaluating reasoning
ability in anomaly scenarios. To address both challenges, we introduce VAU-R1,
a data-efficient framework built upon Multimodal Large Language Models (MLLMs),
which enhances anomaly reasoning through Reinforcement Fine-Tuning (RFT).
Besides, we propose VAU-Bench, the first Chain-of-Thought benchmark tailored
for video anomaly reasoning, featuring multiple-choice QA, detailed rationales,
temporal annotations, and descriptive captions. Empirical results show that
VAU-R1 significantly improves question answering accuracy, temporal grounding,
and reasoning coherence across diverse contexts. Together, our method and
benchmark establish a strong foundation for interpretable and reasoning-aware
video anomaly understanding. Our code is available at
https://github.com/GVCLab/VAU-R1.
TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional
  Diversified Red-Teaming Data Synthesis
Authors: name: Xiaorui Wu, name: Xiaofeng Mao, name: Fei Li, name: Xin Zhang, name: Xuanhong Li, name: Chong Teng, name: Donghong Ji, name: Zhuang Li
Published: 2025-05-30 15:02:21

Large Language Models (LLMs) excel in various natural language processing
tasks but remain vulnerable to generating harmful content or being exploited
for malicious purposes. Although safety alignment datasets have been introduced
to mitigate such risks through supervised fine-tuning (SFT), these datasets
often lack comprehensive risk coverage. Most existing datasets focus primarily
on lexical diversity while neglecting other critical dimensions. To address
this limitation, we propose a novel analysis framework to systematically
measure the risk coverage of alignment datasets across three essential
dimensions: Lexical Diversity, Malicious Intent, and Jailbreak Tactics. We
further introduce TRIDENT, an automated pipeline that leverages persona-based,
zero-shot LLM generation to produce diverse and comprehensive instructions
spanning these dimensions. Each harmful instruction is paired with an ethically
aligned response, resulting in two datasets: TRIDENT-Core, comprising 26,311
examples, and TRIDENT-Edge, with 18,773 examples. Fine-tuning Llama 3.1-8B on
TRIDENT-Edge demonstrates substantial improvements, achieving an average 14.29%
reduction in Harm Score, and a 20% decrease in Attack Success Rate compared to
the best-performing baseline model fine-tuned on the WildBreak dataset.
The Automated but Risky Game: Modeling Agent-to-Agent Negotiations and
  Transactions in Consumer Markets
Authors: name: Shenzhe Zhu, name: Jiao Sun, name: Yi Nian, name: Tobin South, name: Alex Pentland, name: Jiaxin Pei
Published: 2025-05-29 17:41:39

AI agents are increasingly used in consumer-facing applications to assist
with tasks such as product search, negotiation, and transaction execution. In
this paper, we explore a future scenario where both consumers and merchants
authorize AI agents to fully automate negotiations and transactions. We aim to
answer two key questions: (1) Do different LLM agents vary in their ability to
secure favorable deals for users? (2) What risks arise from fully automating
deal-making with AI agents in consumer markets? To address these questions, we
develop an experimental framework that evaluates the performance of various LLM
agents in real-world negotiation and transaction settings. Our findings reveal
that AI-mediated deal-making is an inherently imbalanced game -- different
agents achieve significantly different outcomes for their users. Moreover,
behavioral anomalies in LLMs can result in financial losses for both consumers
and merchants, such as overspending or accepting unreasonable deals. These
results underscore that while automation can improve efficiency, it also
introduces substantial risks. Users should exercise caution when delegating
business decisions to AI agents.
EXP-Bench: Can AI Conduct AI Research Experiments?
Authors: name: Patrick Tser Jern Kon, name: Jiachen Liu, name: Xinyi Zhu, name: Qiuyi Ding, name: Jingjia Peng, name: Jiarong Xing, name: Yibo Huang, name: Yiming Qiu, name: Jayanth Srinivasa, name: Myungjin Lee, name: Mosharaf Chowdhury, name: Matei Zaharia, name: Ang Chen
Published: 2025-05-30 16:46:29

Automating AI research holds immense potential for accelerating scientific
progress, yet current AI agents struggle with the complexities of rigorous,
end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed
to systematically evaluate AI agents on complete research experiments sourced
from influential AI publications. Given a research question and incomplete
starter code, EXP-Bench challenges AI agents to formulate hypotheses, design
and implement experimental procedures, execute them, and analyze results. To
enable the creation of such intricate and authentic tasks with high-fidelity,
we design a semi-autonomous pipeline to extract and structure crucial
experimental details from these research papers and their associated
open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks
from 51 top-tier AI research papers. Evaluations of leading LLM-based agents,
such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial
capabilities: while scores on individual experimental aspects such as design or
implementation correctness occasionally reach 20-35%, the success rate for
complete, executable experiments was a mere 0.5%. By identifying these
bottlenecks and providing realistic step-by-step experiment procedures,
EXP-Bench serves as a vital tool for future AI agents to improve their ability
to conduct AI research experiments. EXP-Bench is open-sourced at
https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
SiLVR: A Simple Language-based Video Reasoning Framework
Authors: name: Ce Zhang, name: Yan-Bo Lin, name: Ziyang Wang, name: Mohit Bansal, name: Gedas Bertasius
Published: 2025-05-30 17:59:19

Recent advances in test-time optimization have led to remarkable reasoning
capabilities in Large Language Models (LLMs), enabling them to solve highly
complex problems in math and coding. However, the reasoning capabilities of
multimodal LLMs (MLLMs) still significantly lag, especially for complex
video-language tasks. To address this issue, we present SiLVR, a Simple
Language-based Video Reasoning framework that decomposes complex video
understanding into two stages. In the first stage, SiLVR transforms raw video
into language-based representations using multisensory inputs, such as short
clip captions and audio/speech subtitles. In the second stage, language
descriptions are fed into a powerful reasoning LLM to solve complex
video-language understanding tasks. To handle long-context multisensory inputs,
we use an adaptive token reduction scheme, which dynamically determines the
temporal granularity with which to sample the tokens. Our simple, modular, and
training-free video reasoning framework achieves the best-reported results on
Video-MME (long), Video-MMMU (comprehension), Video-MMLU, CGBench, and EgoLife.
Furthermore, our empirical study focused on video reasoning capabilities shows
that, despite not being explicitly trained on video, strong reasoning LLMs can
effectively aggregate multisensory input information from video, speech, and
audio for complex temporal, causal, long-context, and knowledge acquisition
reasoning tasks in video. Code is available at https://github.com/CeeZh/SILVR.
OMNIGUARD: An Efficient Approach for AI Safety Moderation Across
  Modalities
Authors: name: Sahil Verma, name: Keegan Hines, name: Jeff Bilmes, name: Charlotte Siska, name: Luke Zettlemoyer, name: Hila Gonen, name: Chandan Singh
Published: 2025-05-29 05:25:27

The emerging capabilities of large language models (LLMs) have sparked
concerns about their immediate potential for harmful misuse. The core approach
to mitigate these concerns is the detection of harmful queries to the model.
Current detection approaches are fallible, and are particularly susceptible to
attacks that exploit mismatched generalization of model capabilities (e.g.,
prompts in low-resource languages or prompts provided in non-text modalities
such as image and audio). To tackle this challenge, we propose OMNIGUARD, an
approach for detecting harmful prompts across languages and modalities. Our
approach (i) identifies internal representations of an LLM/MLLM that are
aligned across languages or modalities and then (ii) uses them to build a
language-agnostic or modality-agnostic classifier for detecting harmful
prompts. OMNIGUARD improves harmful prompt classification accuracy by 11.57\%
over the strongest baseline in a multilingual setting, by 20.44\% for
image-based prompts, and sets a new SOTA for audio-based prompts. By
repurposing embeddings computed during generation, OMNIGUARD is also very
efficient (approx 120 times faster than the next fastest baseline). Code
and data are available at: https://github.com/vsahil/OmniGuard.
DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models
Authors: name: Chenbin Pan, name: Wenbin He, name: Zhengzhong Tu, name: Liu Ren
Published: 2025-05-29 21:58:06

The recent explosive interest in the reasoning capabilities of large language
models, such as DeepSeek-R1, has demonstrated remarkable success through
reinforcement learning-based fine-tuning frameworks, exemplified by methods
like Group Relative Policy Optimization (GRPO). However, such reasoning
abilities remain underexplored and notably absent in vision foundation models,
including representation models like the DINO series. In this work, we propose
DINO-R1, the first such attempt to incentivize visual in-context
reasoning capabilities of vision foundation models using reinforcement
learning. Specifically, DINO-R1 introduces Group Relative Query
Optimization (GRQO), a novel reinforcement-style training strategy explicitly
designed for query-based representation models, which computes query-level
rewards based on group-normalized alignment quality. We also apply
KL-regularization to stabilize the objectness distribution to reduce the
training instability. This joint optimization enables dense and expressive
supervision across queries while mitigating overfitting and distributional
drift. Building upon Grounding-DINO, we train a series of DINO-R1 family models
that integrate a visual prompt encoder and a visual-guided query selection
mechanism. Extensive experiments on COCO, LVIS, and ODinW demonstrate that
DINO-R1 significantly outperforms supervised fine-tuning baselines, achieving
strong generalization in both open-vocabulary and closed-set visual prompting
scenarios.
MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement
  Learning
Authors: name: Yiqing Liang, name: Jielin Qiu, name: Wenhao Ding, name: Zuxin Liu, name: James Tompkin, name: Mengdi Xu, name: Mengzhou Xia, name: Zhengzhong Tu, name: Laixi Shi, name: Jiacheng Zhu
Published: 2025-05-30 17:59:38

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a powerful paradigm for post-training large language models (LLMs), achieving
state-of-the-art performance on tasks with structured, verifiable answers.
Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but
is complicated by the broader, heterogeneous nature of vision-language tasks
that demand nuanced visual, logical, and spatial capabilities. As such,
training MLLMs using RLVR on multiple datasets could be beneficial but creates
challenges with conflicting objectives from interaction among diverse datasets,
highlighting the need for optimal dataset mixture strategies to improve
generalization and reasoning. We introduce a systematic post-training framework
for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation
and benchmark implementation. Specifically, (1) We developed a multimodal RLVR
framework for multi-dataset post-training by curating a dataset that contains
different verifiable vision-language problems and enabling multi-domain online
RL learning with different verifiable rewards; (2) We proposed a data mixture
strategy that learns to predict the RL fine-tuning outcome from the data
mixture distribution, and consequently optimizes the best mixture.
Comprehensive experiments showcase that multi-domain RLVR training, when
combined with mixture prediction strategies, can significantly boost MLLM
general reasoning capacities. Our best mixture improves the post-trained
model's accuracy on out-of-distribution benchmarks by an average of 5.24%
compared to the same model post-trained with uniform data mixture, and by a
total of 20.74% compared to the pre-finetuning baseline.
ChARM: Character-based Act-adaptive Reward Modeling for Advanced
  Role-Playing Language Agents
Authors: name: Feiteng Fang, name: Ting-En Lin, name: Yuchuan Wu, name: Xiong Liu, name: Xiang Huang, name: Dingwei Chen, name: Jing Ye, name: Haonan Zhang, name: Liang Zhu, name: Hamid Alinejad-Rokny, name: Min Yang, name: Fei Huang, name: Yongbin Li
Published: 2025-05-29 18:15:18

Role-Playing Language Agents (RPLAs) aim to simulate characters for realistic
and engaging human-computer interactions. However, traditional reward models
often struggle with scalability and adapting to subjective conversational
preferences. We propose ChARM, a Character-based Act-adaptive Reward Model,
addressing these challenges through two innovations: (1) an act-adaptive margin
that significantly enhances learning efficiency and generalizability, and (2) a
self-evolution mechanism leveraging large-scale unlabeled data to improve
training coverage. Additionally, we introduce RoleplayPref, the first
large-scale preference dataset specifically for RPLAs, featuring 1,108
characters, 13 subcategories, and 16,888 bilingual dialogues, alongside
RoleplayEval, a dedicated evaluation benchmark. Experimental results show a 13%
improvement over the conventional Bradley-Terry model in preference rankings.
Furthermore, applying ChARM-generated rewards to preference learning techniques
(e.g., direct preference optimization) achieves state-of-the-art results on
CharacterEval and RoleplayEval. Code and dataset are available at
https://github.com/calubkk/ChARM.
ReasonGen-R1: CoT for Autoregressive Image generation models through SFT
  and RL
Authors: name: Yu Zhang, name: Yunqi Li, name: Yifan Yang, name: Rui Wang, name: Yuqing Yang, name: Dai Qi, name: Jianmin Bao, name: Dongdong Chen, name: Chong Luo, name: Lili Qiu
Published: 2025-05-30 17:59:48

Although chain-of-thought reasoning and reinforcement learning (RL) have
driven breakthroughs in NLP, their integration into generative vision models
remains underexplored. We introduce ReasonGen-R1, a two-stage framework that
first imbues an autoregressive image generator with explicit text-based
"thinking" skills via supervised fine-tuning on a newly generated reasoning
dataset of written rationales, and then refines its outputs using Group
Relative Policy Optimization. To enable the model to reason through text before
generating images, We automatically generate and release a corpus of model
crafted rationales paired with visual prompts, enabling controlled planning of
object layouts, styles, and scene compositions. Our GRPO algorithm uses reward
signals from a pretrained vision language model to assess overall visual
quality, optimizing the policy in each update. Evaluations on GenEval, DPG, and
the T2I benchmark demonstrate that ReasonGen-R1 consistently outperforms strong
baselines and prior state-of-the-art models. More: aka.ms/reasongen.
Context is Gold to find the Gold Passage: Evaluating and Training
  Contextual Document Embeddings
Authors: name: Max Conti, name: Manuel Faysse, name: Gautier Viaud, name: Antoine Bosselut, name: Céline Hudelot, name: Pierre Colombo
Published: 2025-05-30 16:43:28

A limitation of modern document retrieval embedding methods is that they
typically encode passages (chunks) from the same documents independently, often
overlooking crucial contextual information from the rest of the document that
could greatly improve individual chunk representations.
  In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a
benchmark designed to evaluate retrieval models on their ability to leverage
document-wide context. Our results show that state-of-the-art embedding models
struggle in retrieval scenarios where context is required. To address this
limitation, we propose InSeNT (In-sequence Negative Training), a novel
contrastive post-training approach which combined with late chunking pooling
enhances contextual representation learning while preserving computational
efficiency. Our method significantly improves retrieval quality on ConTEB
without sacrificing base model performance. We further find chunks embedded
with our method are more robust to suboptimal chunking strategies and larger
retrieval corpus sizes. We open-source all artifacts at
https://github.com/illuin-tech/contextual-embeddings.
Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
Authors: name: M. Reza Ebrahimi, name: Roland Memisevic
Published: 2025-05-27 20:38:19

The role of hidden units in recurrent neural networks is typically seen as
modeling memory, with research focusing on enhancing information retention
through gating mechanisms. A less explored perspective views hidden units as
active participants in the computation performed by the network, rather than
passive memory stores. In this work, we revisit bi-linear operations, which
involve multiplicative interactions between hidden units and input embeddings.
We demonstrate theoretically and empirically that they constitute a natural
inductive bias for representing the evolution of hidden states in state
tracking tasks. These are the simplest type of task that require hidden units
to actively contribute to the behavior of the network. We also show that
bi-linear state updates form a natural hierarchy corresponding to state
tracking tasks of increasing complexity, with popular linear recurrent networks
such as Mamba residing at the lowest-complexity center of that hierarchy.
The State of Multilingual LLM Safety Research: From Measuring the
  Language Gap to Mitigating It
Authors: name: Zheng-Xin Yong, name: Beyza Ermis, name: Marzieh Fadaee, name: Stephen H. Bach, name: Julia Kreutzer
Published: 2025-05-30 01:32:44

This paper presents a comprehensive analysis of the linguistic diversity of
LLM safety research, highlighting the English-centric nature of the field.
Through a systematic review of nearly 300 publications from 2020--2024 across
major NLP conferences and workshops at *ACL, we identify a significant and
growing language gap in LLM safety research, with even high-resource
non-English languages receiving minimal attention. We further observe that
non-English languages are rarely studied as a standalone language and that
English safety research exhibits poor language documentation practice. To
motivate future research into multilingual safety, we make several
recommendations based on our survey, and we then pose three concrete future
directions on safety evaluation, training data generation, and crosslingual
safety generalization. Based on our survey and proposed directions, the field
can develop more robust, inclusive AI safety practices for diverse global
populations.
Evaluating and Steering Modality Preferences in Multimodal Large
  Language Model
Authors: name: Yu Zhang, name: Jinlong Ma, name: Yongshuai Hou, name: Xuefeng Bai, name: Kehai Chen, name: Yang Xiang, name: Jun Yu, name: Min Zhang
Published: 2025-05-27 10:07:59

Multimodal large language models (MLLMs) have achieved remarkable performance
on complex tasks with multimodal context. However, it is still understudied
whether they exhibit modality preference when processing multimodal contexts.
To study this question, we first build a MC\textsuperscript{2}
benchmark under controlled evidence conflict scenarios to systematically
evaluate modality preference, which is the tendency to favor one modality over
another when making decisions based on multimodal conflicting evidence. Our
extensive evaluation reveals that all 18 tested MLLMs generally demonstrate
clear modality bias, and modality preference can be influenced by external
interventions. An in-depth analysis reveals that the preference direction can
be captured within the latent representations of MLLMs. Built on this, we
propose a probing and steering method based on representation engineering to
explicitly control modality preference without additional fine-tuning or
carefully crafted prompts. Our method effectively amplifies modality preference
toward a desired direction and applies to downstream tasks such as
hallucination mitigation and multimodal machine translation, yielding promising
improvements.
Fine-Tune an SLM or Prompt an LLM? The Case of Generating Low-Code
  Workflows
Authors: name: Orlando Marquez Ayala, name: Patrice Bechard, name: Emily Chen, name: Maggie Baird, name: Jingfei Chen
Published: 2025-05-30 03:59:35

Large Language Models (LLMs) such as GPT-4o can handle a wide range of
complex tasks with the right prompt. As per token costs are reduced, the
advantages of fine-tuning Small Language Models (SLMs) for real-world
applications -- faster inference, lower costs -- may no longer be clear. In
this work, we present evidence that, for domain-specific tasks that require
structured outputs, SLMs still have a quality advantage. We compare fine-tuning
an SLM against prompting LLMs on the task of generating low-code workflows in
JSON form. We observe that while a good prompt can yield reasonable results,
fine-tuning improves quality by 10% on average. We also perform systematic
error analysis to reveal model limitations.