Orionnnn/hub-stats huggingface dataset:
Seeing from Another Perspective: Evaluating Multi-View Understanding in
  MLLMs
Authors: name: Chun-Hsiao Yeh, name: Chenyu Wang, name: Shengbang Tong, name: Ta-Ying Cheng, name: Rouyu Wang, name: Tianzhe Chu, name: Yuexiang Zhai, name: Yubei Chen, name: Shenghua Gao, name: Yi Ma
Published: 2025-04-21 17:59:53

Multi-view understanding, the ability to reconcile visual information across
diverse viewpoints for effective navigation, manipulation, and 3D scene
comprehension, is a fundamental challenge in Multi-Modal Large Language Models
(MLLMs) to be used as embodied agents. While recent MLLMs have shown impressive
advances in high-level reasoning and planning, they frequently fall short when
confronted with multi-view geometric consistency and cross-view correspondence.
To comprehensively evaluate the challenges of MLLMs in multi-view scene
reasoning, we propose All-Angles Bench, a benchmark of over 2,100 human
carefully annotated multi-view question-answer pairs across 90 diverse
real-world scenes. Our six tasks (counting, attribute identification, relative
distance, relative direction, object manipulation, and camera pose estimation)
specifically test model's geometric correspondence and the capacity to align
information consistently across views. Our extensive experiments, benchmark on
27 representative MLLMs including Gemini-2.0-Flash, Claude-3.7-Sonnet, and
GPT-4o against human evaluators reveals a substantial performance gap,
indicating that current MLLMs remain far from human-level proficiency. Through
in-depth analysis, we show that MLLMs are particularly underperforming under
two aspects: (1) cross-view correspondence for partially occluded views and (2)
establishing the coarse camera poses. These findings highlight the necessity of
domain-specific refinements or modules that embed stronger multi-view
awareness. We believe that our All-Angles Bench offers valuable insights and
contribute to bridging the gap between MLLMs and human-level multi-view
understanding. The project and benchmark are publicly available at
https://danielchyeh.github.io/All-Angles-Bench/.
LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient
  Training of Code LLMs
Authors: name: Yunhui Xia, name: Wei Shen, name: Yan Wang, name: Jason Klein Liu, name: Huifeng Sun, name: Siyue Wu, name: Jian Hu, name: Xiaolong Xu
Published: 2025-04-20 15:28:16

We introduce LeetCodeDataset, a high-quality benchmark for evaluating and
training code-generation models, addressing two key challenges in LLM research:
the lack of reasoning-focused coding benchmarks and self-contained training
testbeds. By curating LeetCode Python problems with rich metadata, broad
coverage, 100+ test cases per problem, and temporal splits (pre/post July
2024), our dataset enables contamination-free evaluation and efficient
supervised fine-tuning (SFT). Experiments show reasoning models significantly
outperform non-reasoning counterparts, while SFT with only 2.6K model-generated
solutions achieves performance comparable to 110K-sample counterparts. The
dataset and evaluation framework are available on Hugging Face and Github.
ToolRL: Reward is All Tool Learning Needs
Authors: name: Cheng Qian, name: Emre Can Acikgoz, name: Qi He, name: Hongru Wang, name: Xiusi Chen, name: Dilek Hakkani-Tür, name: Gokhan Tur, name: Heng Ji
Published: 2025-04-16 21:45:32

Current Large Language Models (LLMs) often undergo supervised fine-tuning
(SFT) to acquire tool use capabilities. However, SFT struggles to generalize to
unfamiliar or complex tool use scenarios. Recent advancements in reinforcement
learning (RL), particularly with R1-like models, have demonstrated promising
reasoning and generalization abilities. Yet, reward design for tool use
presents unique challenges: multiple tools may be invoked with diverse
parameters, and coarse-grained reward signals, such as answer matching, fail to
offer the finegrained feedback required for effective learning. In this work,
we present the first comprehensive study on reward design for tool selection
and application tasks within the RL paradigm. We systematically explore a wide
range of reward strategies, analyzing their types, scales, granularity, and
temporal dynamics. Building on these insights, we propose a principled reward
design tailored for tool use tasks and apply it to train LLMs using Group
Relative Policy Optimization (GRPO). Empirical evaluations across diverse
benchmarks demonstrate that our approach yields robust, scalable, and stable
training, achieving a 17% improvement over base models and a 15% gain over SFT
models. These results highlight the critical role of thoughtful reward design
in enhancing the tool use capabilities and generalization performance of LLMs.
All the codes are released to facilitate future research.
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration
  Benchmark
Authors: name: Guangyi Liu, name: Pengxiang Zhao, name: Liang Liu, name: Zhiming Chen, name: Yuxiang Chai, name: Shuai Ren, name: Hao Wang, name: Shibo He, name: Wenchao Meng
Published: 2025-04-18 17:13:34

Mobile GUI agents show promise in automating tasks but face generalization
challenges in diverse real-world scenarios. Traditional approaches using
pre-training or fine-tuning with massive datasets struggle with the diversity
of mobile applications and user-specific tasks. We propose enhancing mobile GUI
agent capabilities through human demonstrations, focusing on improving
performance in unseen scenarios rather than pursuing universal generalization
through larger datasets. To realize this paradigm, we introduce LearnGUI, the
first comprehensive dataset specifically designed for studying
demonstration-based learning in mobile GUI agents, comprising 2,252 offline
tasks and 101 online tasks with high-quality human demonstrations. We further
develop LearnAct, a sophisticated multi-agent framework that automatically
extracts knowledge from demonstrations to enhance task completion. This
framework integrates three specialized agents: DemoParser for knowledge
extraction, KnowSeeker for relevant knowledge retrieval, and ActExecutor for
demonstration-enhanced task execution. Our experimental results show
significant performance gains in both offline and online evaluations. In
offline assessments, a single demonstration improves model performance,
increasing Gemini-1.5-Pro's accuracy from 19.3% to 51.7%. In online
evaluations, our framework enhances UI-TARS-7B-SFT's task success rate from
18.1% to 32.8%. LearnAct framework and LearnGUI benchmark establish
demonstration-based learning as a promising direction for more adaptable,
personalized, and deployable mobile GUI agents.
X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents
Authors: name: Salman Rahman, name: Liwei Jiang, name: James Shiffer, name: Genglin Liu, name: Sheriff Issaka, name: Md Rizwan Parvez, name: Hamid Palangi, name: Kai-Wei Chang, name: Yejin Choi, name: Saadia Gabriel
Published: 2025-04-15 16:11:28

Multi-turn interactions with language models (LMs) pose critical safety
risks, as harmful intent can be strategically spread across exchanges. Yet, the
vast majority of prior work has focused on single-turn safety, while
adaptability and diversity remain among the key challenges of multi-turn
red-teaming. To address these challenges, we present X-Teaming, a scalable
framework that systematically explores how seemingly harmless interactions
escalate into harmful outcomes and generates corresponding attack scenarios.
X-Teaming employs collaborative agents for planning, attack optimization, and
verification, achieving state-of-the-art multi-turn jailbreak effectiveness and
diversity with success rates up to 98.1% across representative leading
open-weight and closed-source models. In particular, X-Teaming achieves a 96.2%
attack success rate against the latest Claude 3.7 Sonnet model, which has been
considered nearly immune to single-turn attacks. Building on X-Teaming, we
introduce XGuard-Train, an open-source multi-turn safety training dataset that
is 20x larger than the previous best resource, comprising 30K interactive
jailbreaks, designed to enable robust multi-turn safety alignment for LMs. Our
work offers essential tools and insights for mitigating sophisticated
conversational attacks, advancing the multi-turn safety of LMs.
Learning to Reason under Off-Policy Guidance
Authors: name: Jianhao Yan, name: Yafu Li, name: Zican Hu, name: Zhi Wang, name: Ganqu Cui, name: Xiaoye Qu, name: Yu Cheng, name: Yue Zhang
Published: 2025-04-21 08:09:13

Recent advances in large reasoning models (LRMs) demonstrate that
sophisticated behaviors such as multi-step reasoning and self-reflection can
emerge via reinforcement learning (RL) with simple rule-based rewards. However,
existing zero-RL approaches are inherently ``on-policy'', limiting learning to
a model's own outputs and failing to acquire reasoning abilities beyond its
initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY
guidance), a framework that augments zero-RL with off-policy reasoning traces.
LUFFY dynamically balances imitation and exploration by combining off-policy
demonstrations with on-policy rollouts during training. Notably, we propose
policy shaping via regularized importance sampling to avoid superficial and
rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an
over +7.0 average gain across six math benchmarks and an advantage of over +6.2
points in out-of-distribution tasks. It also substantially surpasses
imitation-based supervised fine-tuning (SFT), particularly in generalization.
Analysis shows LUFFY not only imitates effectively but also explores beyond
demonstrations, offering a scalable path to train generalizable reasoning
models with off-policy guidance.
SphereDiff: Tuning-free Omnidirectional Panoramic Image and Video
  Generation via Spherical Latent Representation
Authors: name: Minho Park, name: Taewoong Kang, name: Jooyeol Yun, name: Sungwon Hwang, name: Jaegul Choo
Published: 2025-04-19 19:59:11

The increasing demand for AR/VR applications has highlighted the need for
high-quality 360-degree panoramic content. However, generating high-quality
360-degree panoramic images and videos remains a challenging task due to the
severe distortions introduced by equirectangular projection (ERP). Existing
approaches either fine-tune pretrained diffusion models on limited ERP datasets
or attempt tuning-free methods that still rely on ERP latent representations,
leading to discontinuities near the poles. In this paper, we introduce
SphereDiff, a novel approach for seamless 360-degree panoramic image and video
generation using state-of-the-art diffusion models without additional tuning.
We define a spherical latent representation that ensures uniform distribution
across all perspectives, mitigating the distortions inherent in ERP. We extend
MultiDiffusion to spherical latent space and propose a spherical latent
sampling method to enable direct use of pretrained diffusion models. Moreover,
we introduce distortion-aware weighted averaging to further improve the
generation quality in the projection process. Our method outperforms existing
approaches in generating 360-degree panoramic content while maintaining high
fidelity, making it a robust solution for immersive AR/VR applications. The
code is available here. https://github.com/pmh9960/SphereDiff
RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary
  Quality-Diversity Search
Authors: name: Quy-Anh Dang, name: Chris Ngo, name: Truong-Son Hy
Published: 2025-04-21 12:04:57

Large Language Models (LLMs) exhibit remarkable capabilities but are
susceptible to adversarial prompts that exploit vulnerabilities to produce
unsafe or biased outputs. Existing red-teaming methods often face scalability
challenges, resource-intensive requirements, or limited diversity in attack
strategies. We propose RainbowPlus, a novel red-teaming framework rooted in
evolutionary computation, enhancing adversarial prompt generation through an
adaptive quality-diversity (QD) search that extends classical evolutionary
algorithms like MAP-Elites with innovations tailored for language models. By
employing a multi-element archive to store diverse high-quality prompts and a
comprehensive fitness function to evaluate multiple prompts concurrently,
RainbowPlus overcomes the constraints of single-prompt archives and pairwise
comparisons in prior QD methods like Rainbow Teaming. Experiments comparing
RainbowPlus to QD methods across six benchmark datasets and four open-source
LLMs demonstrate superior attack success rate (ASR) and diversity
(Diverse-Score approx 0.84), generating up to 100 times more unique prompts
(e.g., 10,418 vs. 100 for Ministral-8B-Instruct-2410). Against nine
state-of-the-art methods on the HarmBench dataset with twelve LLMs (ten
open-source, two closed-source), RainbowPlus achieves an average ASR of 81.1%,
surpassing AutoDAN-Turbo by 3.9%, and is 9 times faster (1.45 vs. 13.50 hours).
Our open-source implementation fosters further advancements in LLM safety,
offering a scalable tool for vulnerability assessment. Code and resources are
publicly available at https://github.com/knoveleng/rainbowplus, supporting
reproducibility and future research in LLM red-teaming.
UFO2: The Desktop AgentOS
Authors: name: Chaoyun Zhang, name: He Huang, name: Chiming Ni, name: Jian Mu, name: Si Qin, name: Shilin He, name: Lu Wang, name: Fangkai Yang, name: Pu Zhao, name: Chao Du, name: Liqun Li, name: Yu Kang, name: Zhao Jiang, name: Suzhen Zheng, name: Rujia Wang, name: Jiaxu Qian, name: Minghua Ma, name: Jian-Guang Lou, name: Qingwei Lin, name: Saravan Rajmohan, name: Dongmei Zhang
Published: 2025-04-20 13:04:43

Recent Computer-Using Agents (CUAs), powered by multimodal large language
models (LLMs), offer a promising direction for automating complex desktop
workflows through natural language. However, most existing CUAs remain
conceptual prototypes, hindered by shallow OS integration, fragile
screenshot-based interaction, and disruptive execution.
  We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
into practical, system-level automation. UFO2 features a centralized HostAgent
for task decomposition and coordination, alongside a collection of
application-specialized AppAgent equipped with native APIs, domain-specific
knowledge, and a unified GUI--API action layer. This architecture enables
robust task execution while preserving modularity and extensibility. A hybrid
control detection pipeline fuses Windows UI Automation (UIA) with vision-based
parsing to support diverse interface styles. Runtime efficiency is further
enhanced through speculative multi-action planning, reducing per-step LLM
overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
within an isolated virtual desktop, allowing agents and users to operate
concurrently without interference.
  We evaluate UFO2 across over 20 real-world Windows applications,
demonstrating substantial improvements in robustness and execution accuracy
over prior CUAs. Our results show that deep OS integration unlocks a scalable
path toward reliable, user-aligned desktop automation.
NEMOTRON-CROSSTHINK: Scaling Self-Learning beyond Math Reasoning
Authors: name: Syeda Nahida Akter, name: Shrimai Prabhumoye, name: Matvei Novikov, name: Seungju Han, name: Ying Lin, name: Evelina Bakhturi, name: Eric Nyberg, name: Yejin Choi, name: Mostofa Patwary, name: Mohammad Shoeybi, name: Bryan Catanzaro
Published: 2025-04-15 21:37:13

Large Language Models (LLMs) have shown strong reasoning capabilities,
particularly when enhanced through Reinforcement Learning (RL). While prior
work has successfully applied RL to mathematical reasoning -- where rules and
correctness are well-defined -- generalizing these methods to broader reasoning
domains remains challenging due to limited data, the lack of verifiable reward
structures, and diverse task requirements. In this work, we propose
NEMOTRON-CROSSTHINK, a framework that systematically incorporates multi-domain
corpora, including both synthetic and real-world question-answer pairs, into RL
training to improve generalization across diverse reasoning tasks.
NEMOTRON-CROSSTHINK addresses key challenges by (1) incorporating data from
varied sources spanning STEM, humanities, social sciences, etc.; (2) applying
structured templates (e.g., multiple-choice and open-ended) to control
answer-space complexity; (3) filtering for verifiable answers; and (4)
optimizing data blending strategies that utilizes data from multiple sources
effectively. Our approach enables scalable and verifiable reward modeling
beyond mathematics and demonstrates improved accuracies on both math (MATH-500:
+30.1%, AMC23:+27.5%) and non-math reasoning benchmarks (MMLU-PRO: +12.8%,
GPQA-DIAMOND: +11.3%, AGIEVAL: +15.1%, SUPERGPQA: +3.8%). Moreover,
NEMOTRON-CROSSTHINK exhibits significantly improved response efficiency --
using 28% fewer tokens for correct answers -- highlighting more focused and
effective reasoning. Through NEMOTRON-CROSSTHINK, we demonstrate that
integrating multi-domain, multi-format data in RL leads to more accurate,
efficient, and generalizable LLMs.
Revisiting Uncertainty Quantification Evaluation in Language Models:
  Spurious Interactions with Response Length Bias Results
Authors: name: Andrea Santilli, name: Adam Golinski, name: Michael Kirchhof, name: Federico Danieli, name: Arno Blaas, name: Miao Xiong, name: Luca Zappella, name: Sinead Williamson
Published: 2025-04-18 13:13:42

Uncertainty Quantification (UQ) in Language Models (LMs) is crucial for
improving their safety and reliability. Evaluations often use performance
metrics like AUROC to assess how well UQ methods (e.g., negative sequence
probabilities) correlate with task correctness functions (e.g., ROUGE-L). In
this paper, we show that commonly used correctness functions bias UQ
evaluations by inflating the performance of certain UQ methods. We evaluate 7
correctness functions -- from lexical-based and embedding-based metrics to
LLM-as-a-judge approaches -- across 4 datasets x 4 models x 6 UQ methods. Our
analysis reveals that length biases in the errors of these correctness
functions distort UQ assessments by interacting with length biases in UQ
methods. We identify LLM-as-a-judge approaches as among the least length-biased
choices and hence a potential solution to mitigate these biases.
Cost-of-Pass: An Economic Framework for Evaluating Language Models
Authors: name: Mehmet Hamza Erol, name: Batu El, name: Mirac Suzgun, name: Mert Yuksekgonul, name: James Zou
Published: 2025-04-17 21:58:29

The widespread adoption of AI systems in the economy hinges on their ability
to generate economic value that outweighs their inference costs. Evaluating
this tradeoff requires metrics that account for both performance and costs. We
propose a framework grounded in production theory for evaluating language
models by combining accuracy and inference cost. We introduce "cost-of-pass",
the expected monetary cost of generating a correct solution. We then define the
"frontier cost-of-pass" as the minimum cost-of-pass achievable across available
models or the "human-expert, using the approximate cost of hiring an expert.
Our analysis reveals distinct economic insights. First, lightweight models are
most cost-effective for basic quantitative tasks, large models for
knowledge-intensive ones, and reasoning models for complex quantitative
problems, despite higher per-token costs. Second, tracking this frontier
cost-of-pass over the past year reveals significant progress, particularly for
complex quantitative tasks where the cost has roughly halved every few months.
Third, to trace key innovations driving this progress, we examine
counterfactual frontiers: estimates of cost-efficiency without specific model
classes. We find that innovations in lightweight, large, and reasoning models
have been essential for pushing the frontier in basic quantitative,
knowledge-intensive, and complex quantitative tasks, respectively. Finally, we
assess the cost-reductions afforded by common inference-time techniques like
majority voting and self-refinement, finding that their marginal accuracy gains
rarely justify their costs. Our findings underscore that complementary
model-level innovations are the primary drivers of cost-efficiency, and our
economic framework provides a principled tool for measuring this progress and
guiding deployment.
Self-alignment of Large Video Language Models with Refined Regularized
  Preference Optimization
Authors: name: Pritam Sarkar, name: Ali Etemad
Published: 2025-04-16 13:43:56

Despite recent advances in Large Video Language Models (LVLMs), they still
struggle with fine-grained temporal understanding, hallucinate, and often make
simple mistakes on even simple video question-answering tasks, all of which
pose significant challenges to their safe and reliable deployment in real-world
applications. To address these limitations, we propose a self-alignment
framework that enables LVLMs to learn from their own errors. Our proposed
framework first obtains a training set of preferred and non-preferred response
pairs, where non-preferred responses are generated by incorporating common
error patterns that often occur due to inadequate spatio-temporal
understanding, spurious correlations between co-occurring concepts, and
over-reliance on linguistic cues while neglecting the vision modality, among
others. To facilitate self-alignment of LVLMs with the constructed preferred
and non-preferred response pairs, we introduce Refined Regularized Preference
Optimization (RRPO), a novel preference optimization method that utilizes
sub-sequence-level refined rewards and token-wise KL regularization to address
the limitations of Direct Preference Optimization (DPO). We demonstrate that
RRPO achieves more precise alignment and more stable training compared to DPO.
Our experiments and analysis validate the effectiveness of our approach across
diverse video tasks, including video hallucination, short- and long-video
understanding, and fine-grained temporal reasoning.
Filter2Noise: Interpretable Self-Supervised Single-Image Denoising for
  Low-Dose CT with Attention-Guided Bilateral Filtering
Authors: name: Yipeng Sun, name: Linda-Sophie Schneider, name: Mingxuan Gu, name: Siyuan Mei, name: Chengze Ye, name: Fabian Wagner, name: Siming Bayer, name: Andreas Maier
Published: 2025-04-18 07:15:27

Effective denoising is crucial in low-dose CT to enhance subtle structures
and low-contrast lesions while preventing diagnostic errors. Supervised methods
struggle with limited paired datasets, and self-supervised approaches often
require multiple noisy images and rely on deep networks like U-Net, offering
little insight into the denoising mechanism. To address these challenges, we
propose an interpretable self-supervised single-image denoising framework --
Filter2Noise (F2N). Our approach introduces an Attention-Guided Bilateral
Filter that adapted to each noisy input through a lightweight module that
predicts spatially varying filter parameters, which can be visualized and
adjusted post-training for user-controlled denoising in specific regions of
interest. To enable single-image training, we introduce a novel downsampling
shuffle strategy with a new self-supervised loss function that extends the
concept of Noise2Noise to a single image and addresses spatially correlated
noise. On the Mayo Clinic 2016 low-dose CT dataset, F2N outperforms the leading
self-supervised single-image method (ZS-N2N) by 4.59 dB PSNR while improving
transparency, user control, and parametric efficiency. These features provide
key advantages for medical applications that require precise and interpretable
noise reduction. Our code is demonstrated at
https://github.com/sypsyp97/Filter2Noise.git .