Orionnnn/hub-stats huggingface dataset:
CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction
  Following
Authors: name: Yinghao Ma, name: Siyou Li, name: Juntao Yu, name: Emmanouil Benetos, name: Akira Maezawa
Published: 2025-06-14 00:18:44

Recent advances in audio-text large language models (LLMs) have opened new
possibilities for music understanding and generation. However, existing
benchmarks are limited in scope, often relying on simplified tasks or
multi-choice evaluations that fail to reflect the complexity of real-world
music analysis. We reinterpret a broad range of traditional MIR annotations as
instruction-following formats and introduce CMI-Bench, a comprehensive music
instruction following benchmark designed to evaluate audio-text LLMs on a
diverse set of music information retrieval (MIR) tasks. These include genre
classification, emotion regression, emotion tagging, instrument classification,
pitch estimation, key detection, lyrics transcription, melody extraction, vocal
technique recognition, instrument performance technique detection, music
tagging, music captioning, and (down)beat tracking: reflecting core challenges
in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized
evaluation metrics consistent with previous state-of-the-art MIR models,
ensuring direct comparability with supervised approaches. We provide an
evaluation toolkit supporting all open-source audio-textual LLMs, including
LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant
performance gaps between LLMs and supervised models, along with their culture,
chronological and gender bias, highlighting the potential and limitations of
current models in addressing MIR tasks. CMI-Bench establishes a unified
foundation for evaluating music instruction following, driving progress in
music-aware LLMs.
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction
  and Planning
Authors: name: Mido Assran, name: Adrien Bardes, name: David Fan, name: Quentin Garrido, name: Russell Howes, name: Mojtaba, name: Komeili, name: Matthew Muckley, name: Ammar Rizvi, name: Claire Roberts, name: Koustuv Sinha, name: Artem Zholus, name: Sergio Arnaud, name: Abha Gejji, name: Ada Martin, name: Francois Robert Hogan, name: Daniel Dugas, name: Piotr Bojanowski, name: Vasil Khalidov, name: Patrick Labatut, name: Francisco Massa, name: Marc Szafraniec, name: Kapil Krishnakumar, name: Yong Li, name: Xiaodong Ma, name: Sarath Chandar, name: Franziska Meier, name: Yann LeCun, name: Michael Rabbat, name: Nicolas Ballas
Published: 2025-06-11 17:57:09

A major challenge for modern AI is to learn to understand the world and learn
to act largely by observation. This paper explores a self-supervised approach
that combines internet-scale video data with a small amount of interaction data
(robot trajectories), to develop models capable of understanding, predicting,
and planning in the physical world. We first pre-train an action-free
joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset
comprising over 1 million hours of internet video. V-JEPA 2 achieves strong
performance on motion understanding (77.3 top-1 accuracy on Something-Something
v2) and state-of-the-art performance on human action anticipation (39.7
recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models.
Additionally, after aligning V-JEPA 2 with a large language model, we
demonstrate state-of-the-art performance on multiple video question-answering
tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on
TempCompass). Finally, we show how self-supervised learning can be applied to
robotic planning tasks by post-training a latent action-conditioned world
model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the
Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different
labs and enable picking and placing of objects using planning with image goals.
Notably, this is achieved without collecting any data from the robots in these
environments, and without any task-specific training or reward. This work
demonstrates how self-supervised learning from web-scale data and a small
amount of robot interaction data can yield a world model capable of planning in
the physical world.
DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
Authors: name: Maximilian Du, name: Shuran Song
Published: 2025-06-16 19:00:54

Deploying large, complex policies in the real world requires the ability to
steer them to fit the needs of a situation. Most common steering approaches,
like goal-conditioning, require training the robot policy with a distribution
of test-time objectives in mind. To overcome this limitation, we present
DynaGuide, a steering method for diffusion policies using guidance from an
external dynamics model during the diffusion denoising process. DynaGuide
separates the dynamics model from the base policy, which gives it multiple
advantages, including the ability to steer towards multiple objectives, enhance
underrepresented base policy behaviors, and maintain robustness on low-quality
objectives. The separate guidance signal also allows DynaGuide to work with
off-the-shelf pretrained diffusion policies. We demonstrate the performance and
features of DynaGuide against other steering approaches in a series of
simulated and real experiments, showing an average steering success of 70% on a
set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x
when steered with low-quality objectives. We also successfully steer an
off-the-shelf real robot policy to express preference for particular objects
and even create novel behavior. Videos and more can be found on the project
website: https://dynaguide.github.io
VisText-Mosquito: A Multimodal Dataset and Benchmark for AI-Based
  Mosquito Breeding Site Detection and Reasoning
Authors: name: Md. Adnanul Islam, name: Md. Faiyaz Abdullah Sayeedi, name: Md. Asaduzzaman Shuvo, name: Muhammad Ziaur Rahman, name: Shahanur Rahman Bappy, name: Raiyan Rahman, name: Swakkhar Shatabda
Published: 2025-06-17 15:24:30

Mosquito-borne diseases pose a major global health risk, requiring early
detection and proactive control of breeding sites to prevent outbreaks. In this
paper, we present VisText-Mosquito, a multimodal dataset that integrates visual
and textual data to support automated detection, segmentation, and reasoning
for mosquito breeding site analysis. The dataset includes 1,828 annotated
images for object detection, 142 images for water surface segmentation, and
natural language reasoning texts linked to each image. The YOLOv9s model
achieves the highest precision of 0.92926 and mAP@50 of 0.92891 for object
detection, while YOLOv11n-Seg reaches a segmentation precision of 0.91587 and
mAP@50 of 0.79795. For reasoning generation, our fine-tuned BLIP model achieves
a final loss of 0.0028, with a BLEU score of 54.7, BERTScore of 0.91, and
ROUGE-L of 0.87. This dataset and model framework emphasize the theme
"Prevention is Better than Cure", showcasing how AI-based detection can
proactively address mosquito-borne disease risks. The dataset and
implementation code are publicly available at GitHub:
https://github.com/adnanul-islam-jisun/VisText-Mosquito
Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via
  Reinforcement Learning
Authors: name: Haozhen Zhang, name: Tao Feng, name: Jiaxuan You
Published: 2025-06-10 17:56:45

The rapid emergence of diverse large language models (LLMs) has spurred the
development of LLM routers that assign user queries to the most suitable model.
However, existing LLM routers typically perform a single-round, one-to-one
mapping (i.e., assigning each query to a single model in isolation),
which limits their capability to tackle complex tasks that demand the
complementary strengths of multiple LLMs. In this paper, we present
Router-R1, a reinforcement learning (RL)-based framework that
formulates multi-LLM routing and aggregation as a sequential decision process.
Router-R1 instantiates the router itself as a capable LLM, leveraging its
reasoning ability to interleave "think" actions (internal deliberation) with
"route" actions (dynamic model invocation), and integrates each response into
its evolving context. To guide learning, we employ a lightweight rule-based
reward comprising format rewards, final outcome rewards, and a novel cost
reward for performance and cost trade-off optimization, opening a pathway
toward optimizing performance-cost tradeoffs via RL. Router-R1 also conditions
only on simple model descriptors such as pricing, latency, and example
performance, enabling strong generalization to unseen model selection.
Experiments on seven general and multi-hop QA benchmarks show that Router-R1
outperforms over several strong baselines, achieving superior performance while
maintaining robust generalization and cost management.Code is available at
https://github.com/ulab-uiuc/Router-R1.
Universal Jailbreak Suffixes Are Strong Attention Hijackers
Authors: name: Matan Ben-Tov, name: Mor Geva, name: Mahmood Sharif
Published: 2025-06-15 15:20:37

We study suffix-based jailbreaksx2013a powerful family of attacks
against large language models (LLMs) that optimize adversarial suffixes to
circumvent safety alignment. Focusing on the widely used foundational GCG
attack (Zou et al., 2023), we observe that suffixes vary in efficacy: some
markedly more universalx2013generalizing to many unseen harmful
instructionsx2013than others. We first show that GCG's
effectiveness is driven by a shallow, critical mechanism, built on the
information flow from the adversarial suffix to the final chat template tokens
before generation. Quantifying the dominance of this mechanism during
generation, we find GCG irregularly and aggressively hijacks the
contextualization process. Crucially, we tie hijacking to the universality
phenomenon, with more universal suffixes being stronger hijackers.
Subsequently, we show that these insights have practical implications: GCG
universality can be efficiently enhanced (up to times5 in some cases) at no
additional computational cost, and can also be surgically mitigated, at least
halving attack success with minimal utility loss. We release our code and data
at http://github.com/matanbt/interp-jailbreak.
Mixture-of-Experts Meets In-Context Reinforcement Learning
Authors: name: Wenhao Wu, name: Fuhong Liu, name: Haoru Li, name: Zican Hu, name: Daoyi Dong, name: Chunlin Chen, name: Zhi Wang
Published: 2025-06-05 06:29:14

In-context reinforcement learning (ICRL) has emerged as a promising paradigm
for adapting RL agents to downstream tasks through prompt conditioning.
However, two notable challenges remain in fully harnessing in-context learning
within RL domains: the intrinsic multi-modality of the state-action-reward data
and the diverse, heterogeneous nature of decision tasks. To tackle these
challenges, we propose T2MIR (Token- and Task-wise
MoE for In-context RL), an innovative framework that
introduces architectural advances of mixture-of-experts (MoE) into
transformer-based decision models. T2MIR substitutes the feedforward layer with
two parallel layers: a token-wise MoE that captures distinct semantics of input
tokens across multiple modalities, and a task-wise MoE that routes diverse
tasks to specialized experts for managing a broad task distribution with
alleviated gradient conflicts. To enhance task-wise routing, we introduce a
contrastive learning method that maximizes the mutual information between the
task and its router representation, enabling more precise capture of
task-relevant information. The outputs of two MoE components are concatenated
and fed into the next layer. Comprehensive experiments show that T2MIR
significantly facilitates in-context learning capacity and outperforms various
types of baselines. We bring the potential and promise of MoE to ICRL, offering
a simple and scalable architectural enhancement to advance ICRL one step closer
toward achievements in language and vision communities. Our code is available
at https://github.com/NJU-RL/T2MIR.
EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction
Authors: name: Hsi-Che Lin, name: Yu-Chu Yu, name: Kai-Po Chang, name: Yu-Chiang Frank Wang
Published: 2025-06-13 17:59:58

Open-source foundation models have seen rapid adoption and development,
enabling powerful general-purpose capabilities across diverse domains. However,
fine-tuning large foundation models for domain-specific or personalized tasks
remains prohibitively expensive for most users due to the significant memory
overhead beyond that of inference. We introduce EMLoC, an Emulator-based
Memory-efficient fine-tuning framework with LoRA Correction, which enables
model fine-tuning within the same memory budget required for inference. EMLoC
constructs a task-specific light-weight emulator using activation-aware
singular value decomposition (SVD) on a small downstream calibration set.
Fine-tuning then is performed on this lightweight emulator via LoRA. To tackle
the misalignment between the original model and the compressed emulator, we
propose a novel compensation algorithm to correct the fine-tuned LoRA module,
which thus can be merged into the original model for inference. EMLoC supports
flexible compression ratios and standard training pipelines, making it
adaptable to a wide range of applications. Extensive experiments demonstrate
that EMLoC outperforms other baselines across multiple datasets and modalities.
Moreover, without quantization, EMLoC enables fine-tuning of a 38B model on a
single 24GB consumer GPU-bringing efficient and practical model adaptation to
individual users.
xbench: Tracking Agents Productivity Scaling with Profession-Aligned
  Real-World Evaluations
Authors: name: Kaiyuan Chen, name: Yixin Ren, name: Yang Liu, name: Xiaobo Hu, name: Haotong Tian, name: Tianbao Xie, name: Fangfu Liu, name: Haoye Zhang, name: Hongzhang Liu, name: Yuan Gong, name: Chen Sun, name: Han Hou, name: Hui Yang, name: James Pan, name: Jianan Lou, name: Jiayi Mao, name: Jizheng Liu, name: Jinpeng Li, name: Kangyi Liu, name: Kenkun Liu, name: Rui Wang, name: Run Li, name: Tong Niu, name: Wenlong Zhang, name: Wenqi Yan, name: Xuanzheng Wang, name: Yuchen Zhang, name: Yi-Hsin Hung, name: Yuan Jiang, name: Zexuan Liu, name: Zihan Yin, name: Zijian Ma, name: Zhiwen Mo
Published: 2025-06-16 16:16:14

We introduce xbench, a dynamic, profession-aligned evaluation suite designed
to bridge the gap between AI agent capabilities and real-world productivity.
While existing benchmarks often focus on isolated technical skills, they may
not accurately reflect the economic value agents deliver in professional
settings. To address this, xbench targets commercially significant domains with
evaluation tasks defined by industry professionals. Our framework creates
metrics that strongly correlate with productivity value, enables prediction of
Technology-Market Fit (TMF), and facilitates tracking of product capabilities
over time. As our initial implementations, we present two benchmarks:
Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world
headhunting business scenarios to evaluate agents' abilities in company
mapping, information retrieval, and talent sourcing. For Marketing, we assess
agents' ability to match influencers with advertiser needs, evaluating their
performance across 50 advertiser requirements using a curated pool of 836
candidate influencers. We present initial evaluation results for leading
contemporary agents, establishing a baseline for these professional domains.
Our continuously updated evalsets and evaluations are available at
https://xbench.org.
Scaling Test-time Compute for LLM Agents
Authors: name: King Zhu, name: Hanhao Li, name: Siwei Wu, name: Tianshun Xing, name: Dehua Ma, name: Xiangru Tang, name: Minghao Liu, name: Jian Yang, name: Jiaheng Liu, name: Yuchen Eleanor Jiang, name: Changwang Zhang, name: Chenghua Lin, name: Jun Wang, name: Ge Zhang, name: Wangchunshu Zhou
Published: 2025-06-15 17:59:47

Scaling test time compute has shown remarkable success in improving the
reasoning abilities of large language models (LLMs). In this work, we conduct
the first systematic exploration of applying test-time scaling methods to
language agents and investigate the extent to which it improves their
effectiveness. Specifically, we explore different test-time scaling strategies,
including: (1) parallel sampling algorithms; (2) sequential revision
strategies; (3) verifiers and merging methods; (4)strategies for diversifying
rollouts.We carefully analyze and ablate the impact of different design
strategies on applying test-time scaling on language agents, and have follow
findings: 1. Scaling test time compute could improve the performance of agents.
2. Knowing when to reflect is important for agents. 3. Among different
verification and result merging approaches, the list-wise method performs best.
4. Increasing diversified rollouts exerts a positive effect on the agent's task
performance.
Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic
  Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise
  Pooled Representations
Authors: name: Abhilekh Borah, name: Chhavi Sharma, name: Danush Khanna, name: Utkarsh Bhatt, name: Gurpreet Singh, name: Hasnat Md Abdullah, name: Raghav Kaushik Ravi, name: Vinija Jain, name: Jyoti Patel, name: Shubham Singh, name: Vasu Sharma, name: Arpita Vats, name: Rahul Raja, name: Aman Chadha, name: Amitava Das
Published: 2025-06-16 18:22:28

Alignment is no longer a luxury, it is a necessity. As large language models
(LLMs) enter high-stakes domains like education, healthcare, governance, and
law, their behavior must reliably reflect human-aligned values and safety
constraints. Yet current evaluations rely heavily on behavioral proxies such as
refusal rates, G-Eval scores, and toxicity classifiers, all of which have
critical blind spots. Aligned models are often vulnerable to jailbreaking,
stochasticity of generation, and alignment faking.
  To address this issue, we introduce the Alignment Quality Index (AQI). This
novel geometric and prompt-invariant metric empirically assesses LLM alignment
by analyzing the separation of safe and unsafe activations in latent space. By
combining measures such as the Davies-Bouldin Score (DBS), Dunn Index (DI),
Xie-Beni Index (XBI), and Calinski-Harabasz Index (CHI) across various
formulations, AQI captures clustering quality to detect hidden misalignments
and jailbreak risks, even when outputs appear compliant. AQI also serves as an
early warning signal for alignment faking, offering a robust, decoding
invariant tool for behavior agnostic safety auditing.
  Additionally, we propose the LITMUS dataset to facilitate robust evaluation
under these challenging conditions. Empirical tests on LITMUS across different
models trained under DPO, GRPO, and RLHF conditions demonstrate AQI's
correlation with external judges and ability to reveal vulnerabilities missed
by refusal metrics. We make our implementation publicly available to foster
future research in this area.
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language
  Models in Tool-Calling Error Scenarios
Authors: name: Shiting Huang, name: Zhen Fang, name: Zehui Chen, name: Siyu Yuan, name: Junjie Ye, name: Yu Zeng, name: Lin Chen, name: Qi Mao, name: Feng Zhao
Published: 2025-06-11 17:59:18

The ability of large language models (LLMs) to utilize external tools has
enabled them to tackle an increasingly diverse range of tasks. However, as the
tasks become more complex and long-horizon, the intricate tool utilization
process may trigger various unexpected errors. Therefore, how to effectively
handle such errors, including identifying, diagnosing, and recovering from
them, has emerged as a key research direction for advancing tool learning. In
this work, we first extensively analyze the types of errors encountered during
the function-calling process on several competitive tool evaluation benchmarks.
Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation
benchmark specialized for tool learning. Building upon a novel evolutionary
strategy for dataset construction, CRITICTOOL holds diverse tool-use errors
with varying complexities, which better reflects real-world scenarios. We
conduct extensive experiments on CRITICTOOL, and validate the generalization
and effectiveness of our constructed benchmark strategy. We also provide an
in-depth analysis of the tool reflection ability on various LLMs, offering a
new perspective on the field of tool learning in LLMs. The code is available at
https://github.com/Shellorley0513/CriticTool{https://github.com/Shellorley0513/CriticTool}.
Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning
  for LLMs
Authors: name: Ring Team, name: Bin Hu, name: Cai Chen, name: Deng Zhao, name: Ding Liu, name: Dingnan Jin, name: Feng Zhu, name: Hao Dai, name: Hongzhi Luan, name: Jia Guo, name: Jiaming Liu, name: Jiewei Wu, name: Jun Mei, name: Jun Zhou, name: Junbo Zhao, name: Junwu Xiong, name: Kaihong Zhang, name: Kuan Xu, name: Lei Liang, name: Liang Jiang, name: Liangcheng Fu, name: Longfei Zheng, name: Qiang Gao, name: Qing Cui, name: Quan Wan, name: Shaomian Zheng, name: Shuaicheng Li, name: Tongkai Yang, name: Wang Ren, name: Xiaodong Yan, name: Xiaopei Wan, name: Xiaoyun Feng, name: Xin Zhao, name: Xinxing Yang, name: Xinyu Kong, name: Xuemin Yang, name: Yang Li, name: Yingting Wu, name: Yongkang Liu, name: Zhankai Xu, name: Zhenduo Zhang, name: Zhenglei Zhou, name: Zhenyu Huang, name: Zhiqiang Zhang, name: Zihao Wang, name: Zujie Wen
Published: 2025-06-17 17:12:34

We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model
optimized via reinforcement learning (RL) to achieve efficient and robust
reasoning capabilities. Built upon the publicly available Ling-lite model, a
16.8 billion parameter model with 2.75 billion activated parameters, our
approach matches the performance of state-of-the-art (SOTA) small-scale
reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench,
GPQA-Diamond) while activating only one-third of the parameters required by
comparable models. To accomplish this, we introduce a joint training pipeline
integrating distillation with RL, revealing undocumented challenges in MoE RL
training. First, we identify optimization instability during RL training, and
we propose Constrained Contextual Computation Policy Optimization(C3PO), a
novel approach that enhances training stability and improves computational
throughput via algorithm-system co-design methodology. Second, we empirically
demonstrate that selecting distillation checkpoints based on entropy loss for
RL training, rather than validation metrics, yields superior
performance-efficiency trade-offs in subsequent RL training. Finally, we
develop a two-stage training paradigm to harmonize multi-domain data
integration, addressing domain conflicts that arise in training with mixed
dataset. We will release the model, dataset, and code.