Orionnnn/hub-stats huggingface dataset:
RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Authors: name: Xinnuo Xu, name: Rachel Lawrence, name: Kshitij Dubey, name: Atharva Pandey, name: Risa Ueno, name: Fabian Falck, name: Aditya V. Nori, name: Rahul Sharma, name: Amit Sharma, name: Javier Gonzalez
Published: 2025-06-18 13:35:47

Recent Large Language Models (LLMs) have reported high accuracy on reasoning
benchmarks. However, it is still unclear whether the observed results arise
from true reasoning or from statistical recall of the training set. Inspired by
the ladder of causation (Pearl, 2009) and its three levels (associations,
interventions and counterfactuals), this paper introduces RE-IMAGINE, a
framework to characterize a hierarchy of reasoning ability in LLMs, alongside
an automated pipeline to generate problem variations at different levels of the
hierarchy. By altering problems in an intermediate symbolic representation,
RE-IMAGINE generates arbitrarily many problems that are not solvable using
memorization alone. Moreover, the framework is general and can work across
reasoning domains, including math, code, and logic. We demonstrate our
framework on four widely-used benchmarks to evaluate several families of LLMs,
and observe reductions in performance when the models are queried with problem
variations. These assessments indicate a degree of reliance on statistical
recall for past performance, and open the door to further research targeting
skills across the reasoning hierarchy.
EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech
  Emotion Detection
Authors: name: Christoph Schuhmann, name: Robert Kaczmarczyk, name: Gollam Rabby, name: Felix Friedrich, name: Maurice Kraus, name: Kourosh Nadi, name: Huu Nguyen, name: Kristian Kersting, name: Sören Auer
Published: 2025-06-11 15:06:59

The advancement of text-to-speech and audio generation models necessitates
robust benchmarks for evaluating the emotional understanding capabilities of AI
systems. Current speech emotion recognition (SER) datasets often exhibit
limitations in emotional granularity, privacy concerns, or reliance on acted
portrayals. This paper introduces EmoNet-Voice, a new resource for speech
emotion detection, which includes EmoNet-Voice Big, a large-scale pre-training
dataset (featuring over 4,500 hours of speech across 11 voices, 40 emotions,
and 4 languages), and EmoNet-Voice Bench, a novel benchmark dataset with human
expert annotations. EmoNet-Voice is designed to evaluate SER models on a
fine-grained spectrum of 40 emotion categories with different levels of
intensities. Leveraging state-of-the-art voice generation, we curated synthetic
audio snippets simulating actors portraying scenes designed to evoke specific
emotions. Crucially, we conducted rigorous validation by psychology experts who
assigned perceived intensity labels. This synthetic, privacy-preserving
approach allows for the inclusion of sensitive emotional states often absent in
existing datasets. Lastly, we introduce Empathic Insight Voice models that set
a new standard in speech emotion recognition with high agreement with human
experts. Our evaluations across the current model landscape exhibit valuable
findings, such as high-arousal emotions like anger being much easier to detect
than low-arousal states like concentration.
Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain
  Perspective
Authors: name: Zhoujun Cheng, name: Shibo Hao, name: Tianyang Liu, name: Fan Zhou, name: Yutao Xie, name: Feng Yao, name: Yuexin Bian, name: Yonghao Zhuang, name: Nilabjo Dey, name: Yuheng Zha, name: Yi Gu, name: Kun Zhou, name: Yuqi Wang, name: Yuan Li, name: Richard Fan, name: Jianshu She, name: Chengqian Gao, name: Abulhair Saparov, name: Haonan Li, name: Taylor W. Killian, name: Mikhail Yurochkin, name: Zhengzhong Liu, name: Eric P. Xing, name: Zhiting Hu
Published: 2025-06-17 20:24:00

Reinforcement learning (RL) has emerged as a promising approach to improve
large language model (LLM) reasoning, yet most open efforts focus narrowly on
math and code, limiting our understanding of its broader applicability to
general reasoning. A key challenge lies in the lack of reliable, scalable RL
reward signals across diverse reasoning domains. We introduce Guru, a curated
RL reasoning corpus of 92K verifiable examples spanning six reasoning
domains--Math, Code, Science, Logic, Simulation, and Tabular--each built
through domain-specific reward design, deduplication, and filtering to ensure
reliability and effectiveness for RL training. Based on Guru, we systematically
revisit established findings in RL for LLM reasoning and observe significant
variation across domains. For example, while prior work suggests that RL
primarily elicits existing knowledge from pretrained models, our results reveal
a more nuanced pattern: domains frequently seen during pretraining (Math, Code,
Science) easily benefit from cross-domain RL training, while domains with
limited pretraining exposure (Logic, Simulation, and Tabular) require in-domain
training to achieve meaningful performance gains, suggesting that RL is likely
to facilitate genuine skill acquisition. Finally, we present Guru-7B and
Guru-32B, two models that achieve state-of-the-art performance among open
models RL-trained with publicly available data, outperforming best baselines by
7.9% and 6.7% on our 17-task evaluation suite across six reasoning domains. We
also show that our models effectively improve the Pass@k performance of their
base models, particularly on complex tasks less likely to appear in pretraining
data. We release data, models, training and evaluation code to facilitate
general-purpose reasoning at: https://github.com/LLM360/Reasoning360
SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning
Authors: name: Anuradha Chopra, name: Abhinaba Roy, name: Dorien Herremans
Published: 2025-06-18 05:51:36

Detailed captions that accurately reflect the characteristics of a music
piece can enrich music databases and drive forward research in music AI. This
paper introduces a multi-task music captioning model, SonicVerse, that
integrates caption generation with auxiliary music feature detection tasks such
as key detection, vocals detection, and more, so as to directly capture both
low-level acoustic details as well as high-level musical attributes. The key
contribution is a projection-based architecture that transforms audio input
into language tokens, while simultaneously detecting music features through
dedicated auxiliary heads. The outputs of these heads are also projected into
language tokens, to enhance the captioning input. This framework not only
produces rich, descriptive captions for short music fragments but also directly
enables the generation of detailed time-informed descriptions for longer music
pieces, by chaining the outputs using a large-language model. To train the
model, we extended the MusicBench dataset by annotating it with music features
using MIRFLEX, a modular music feature extractor, resulting in paired audio,
captions and music feature data. Experimental results show that incorporating
features in this way improves the quality and detail of the generated captions.
Improved Iterative Refinement for Chart-to-Code Generation via
  Structured Instruction
Authors: name: Chengzhi Xu, name: Yuyang Wang, name: Lai Wei, name: Lichao Sun, name: Weiran Huang
Published: 2025-06-15 14:10:16

Recently, multimodal large language models (MLLMs) have attracted increasing
research attention due to their powerful visual understanding capabilities.
While they have achieved impressive results on various vision tasks, their
performance on chart-to-code generation remains suboptimal. This task requires
MLLMs to generate executable code that can reproduce a given chart, demanding
not only precise visual understanding but also accurate translation of visual
elements into structured code. Directly prompting MLLMs to perform this complex
task often yields unsatisfactory results. To address this challenge, we propose
{ChartIR}, an iterative refinement method based on structured instruction.
First, we distinguish two tasks: visual understanding and code translation. To
accomplish the visual understanding component, we design two types of
structured instructions: description and difference. The description
instruction captures the visual elements of the reference chart, while the
difference instruction characterizes the discrepancies between the reference
chart and the generated chart. These instructions effectively transform visual
features into language representations, thereby facilitating the subsequent
code translation process. Second, we decompose the overall chart generation
pipeline into two stages: initial code generation and iterative refinement,
enabling progressive enhancement of the final output. Experimental results show
that, compared to other method, our method achieves superior performance on
both the open-source model Qwen2-VL and the closed-source model GPT-4o.
All is Not Lost: LLM Recovery without Checkpoints
Authors: name: Nikolay Blagoev, name: Oğuzhan Ersoy, name: Lydia Yiyu Chen
Published: 2025-06-18 13:48:33

Training LLMs on decentralized and wimpy computation nodes, e.g., multiple
on-spot instances, lowers the training cost and enables model democratization.
The inevitable challenge here is the churn of nodes due to failures and the
operator's scheduling policies, leading to losing a stage - a part of the
model. The conventional approaches to recover from failures are to either use
checkpointing, where periodically a copy of the entire model is sent to an
additional storage, or redundant computation. These approaches yield
significant communication and/or computation overhead even in non-failure cases
and scale poorly in settings with large models. In this paper, we propose,
CheckFree, an efficient recovery method where a failing stage is substituted by
a weighted average of the closest neighboring stages. In contrast to the state
of the art, CheckFree requires no additional computation or storage. However,
because of the nature of averaging neighbouring stages, it can only recover
failures of intermediate stages. We further extend our method to CheckFree+
with out-of-order pipeline execution to tolerate crashes of the first and last
stages. Thanks to out-of-order pipelining, behaviour of those stages is
mimicked by their neighboring ones, which allows CheckFree+ to recover them by
simply copying the weights from the immediate neighbour. To be able to recover
the (de)embedding layers, CheckFree+ copies those layers to the neighboring
stages, which requires relatively small storage overhead. We extensively
evaluate our method on LLaMa models of model sizes from 124M to 1.5B with
varying failure frequencies. In the case of low and medium failure rates
(5-10%), CheckFree and CheckFree+ outperform both checkpointing and redundant
computation in terms of convergence in wall-clock time by over 12%. Both of our
proposals can be run via our code available at:
https://github.com/gensyn-ai/CheckFree.
Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model
Authors: name: Anirud Aggarwal, name: Abhinav Shrivastava, name: Matthew Gwilliam
Published: 2025-06-18 17:59:50

Diffusion-based image generation models excel at producing high-quality
synthetic content, but suffer from slow and computationally expensive
inference. Prior work has attempted to mitigate this by caching and reusing
features within diffusion transformers across inference steps. These methods,
however, often rely on rigid heuristics that result in limited acceleration or
poor generalization across architectures. We propose Evolutionary Caching to
Accelerate Diffusion models (ECAD), a genetic algorithm that learns efficient,
per-model, caching schedules forming a Pareto frontier, using only a small set
of calibration prompts. ECAD requires no modifications to network parameters or
reference images. It offers significant inference speedups, enables
fine-grained control over the quality-latency trade-off, and adapts seamlessly
to different diffusion models. Notably, ECAD's learned schedules can generalize
effectively to resolutions and model variants not seen during calibration. We
evaluate ECAD on PixArt-alpha, PixArt-Sigma, and FLUX-1.dev using multiple
metrics (FID, CLIP, Image Reward) across diverse benchmarks (COCO, MJHQ-30k,
PartiPrompts), demonstrating consistent improvements over previous approaches.
On PixArt-alpha, ECAD identifies a schedule that outperforms the previous
state-of-the-art method by 4.47 COCO FID while increasing inference speedup
from 2.35x to 2.58x. Our results establish ECAD as a scalable and generalizable
approach for accelerating diffusion inference. Our project website is available
at https://aniaggarwal.github.io/ecad and our code is available at
https://github.com/aniaggarwal/ecad.
AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language
  Models
Authors: name: Jaeho Lee, name: Atharv Chowdhary
Published: 2025-06-08 14:08:22

Recent benchmarks have probed factual consistency and rhetorical robustness
in Large Language Models (LLMs). However, a knowledge gap exists regarding how
directional framing of factually true statements influences model agreement, a
common scenario for LLM users. AssertBench addresses this by sampling
evidence-supported facts from FEVEROUS, a fact verification dataset. For each
(evidence-backed) fact, we construct two framing prompts: one where the user
claims the statement is factually correct, and another where the user claims it
is incorrect. We then record the model's agreement and reasoning. The desired
outcome is that the model asserts itself, maintaining consistent truth
evaluation across both framings, rather than switching its evaluation to agree
with the user. AssertBench isolates framing-induced variability from the
model's underlying factual knowledge by stratifying results based on the
model's accuracy on the same claims when presented neutrally. In doing so, this
benchmark aims to measure an LLM's ability to "stick to its guns" when
presented with contradictory user assertions about the same fact. The complete
source code is available at https://github.com/achowd32/assert-bench.
GMT: General Motion Tracking for Humanoid Whole-Body Control
Authors: name: Zixuan Chen, name: Mazeyu Ji, name: Xuxin Cheng, name: Xuanbin Peng, name: Xue Bin Peng, name: Xiaolong Wang
Published: 2025-06-17 17:59:33

The ability to track general whole-body motions in the real world is a useful
way to build general-purpose humanoid robots. However, achieving this can be
challenging due to the temporal and kinematic diversity of the motions, the
policy's capability, and the difficulty of coordination of the upper and lower
bodies. To address these issues, we propose GMT, a general and scalable
motion-tracking framework that trains a single unified policy to enable
humanoid robots to track diverse motions in the real world. GMT is built upon
two core components: an Adaptive Sampling strategy and a Motion
Mixture-of-Experts (MoE) architecture. The Adaptive Sampling automatically
balances easy and difficult motions during training. The MoE ensures better
specialization of different regions of the motion manifold. We show through
extensive experiments in both simulation and the real world the effectiveness
of GMT, achieving state-of-the-art performance across a broad spectrum of
motions using a unified general policy. Videos and additional information can
be found at https://gmt-humanoid.github.io.
PictSure: Pretraining Embeddings Matters for In-Context Learning Image
  Classifiers
Authors: name: Lukas Schiesser, name: Cornelius Wolff, name: Sophie Haas, name: Simon Pukrop
Published: 2025-06-16 08:57:03

Building image classification models remains cumbersome in data-scarce
domains, where collecting large labeled datasets is impractical. In-context
learning (ICL) has emerged as a promising paradigm for few-shot image
classification (FSIC), enabling models to generalize across domains without
gradient-based adaptation. However, prior work has largely overlooked a
critical component of ICL-based FSIC pipelines: the role of image embeddings.
In this work, we present PictSure, an ICL framework that places the embedding
model -- its architecture, pretraining, and training dynamics -- at the center
of analysis. We systematically examine the effects of different visual encoder
types, pretraining objectives, and fine-tuning strategies on downstream FSIC
performance. Our experiments show that the training success and the
out-of-domain performance are highly dependent on how the embedding models are
pretrained. Consequently, PictSure manages to outperform existing ICL-based
FSIC models on out-of-domain benchmarks that differ significantly from the
training distribution, while maintaining comparable results on in-domain tasks.
Code can be found at https://github.com/PictSure/pictsure-library.
ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured
  Proxies
Authors: name: Jinyan Yuan, name: Bangbang Yang, name: Keke Wang, name: Panwang Pan, name: Lin Ma, name: Xuehai Zhang, name: Xiao Liu, name: Zhaopeng Cui, name: Yuewen Ma
Published: 2025-06-17 08:50:05

Automatic creation of 3D scenes for immersive VR presence has been a
significant research focus for decades. However, existing methods often rely on
either high-poly mesh modeling with post-hoc simplification or massive 3D
Gaussians, resulting in a complex pipeline or limited visual realism. In this
paper, we demonstrate that such exhaustive modeling is unnecessary for
achieving compelling immersive experience. We introduce ImmerseGen, a novel
agent-guided framework for compact and photorealistic world modeling.
ImmerseGen represents scenes as hierarchical compositions of lightweight
geometric proxies, i.e., simplified terrain and billboard meshes, and generates
photorealistic appearance by synthesizing RGBA textures onto these proxies.
Specifically, we propose terrain-conditioned texturing for user-centric base
world synthesis, and RGBA asset texturing for midground and foreground scenery.
This reformulation offers several advantages: (i) it simplifies modeling by
enabling agents to guide generative models in producing coherent textures that
integrate seamlessly with the scene; (ii) it bypasses complex geometry creation
and decimation by directly synthesizing photorealistic textures on proxies,
preserving visual quality without degradation; (iii) it enables compact
representations suitable for real-time rendering on mobile VR headsets. To
automate scene creation from text prompts, we introduce VLM-based modeling
agents enhanced with semantic grid-based analysis for improved spatial
reasoning and accurate asset placement. ImmerseGen further enriches scenes with
dynamic effects and ambient audio to support multisensory immersion.
Experiments on scene generation and live VR showcases demonstrate that
ImmerseGen achieves superior photorealism, spatial coherence and rendering
efficiency compared to prior methods. Project webpage:
https://immersegen.github.io.
BUT System for the MLC-SLM Challenge
Authors: name: Alexander Polok, name: Jiangyu Han, name: Dominik Klement, name: Samuele Cornell, name: Jan Černocký, name: Lukáš Burget
Published: 2025-06-16 12:28:35

We present a two-speaker automatic speech recognition (ASR) system that
combines DiCoW -- a diarization-conditioned variant of Whisper -- with
DiariZen, a diarization pipeline built on top of Pyannote. We first evaluate
both systems in out-of-domain (OOD) multilingual scenarios without any
fine-tuning. In this scenario, DiariZen consistently outperforms the baseline
Pyannote diarization model, demonstrating strong generalization. Despite being
fine-tuned on English-only data for target-speaker ASR, DiCoW retains solid
multilingual performance, indicating that encoder modifications preserve
Whisper's multilingual capabilities. We then fine-tune both DiCoW and DiariZen
on the MLC-SLM challenge data. The fine-tuned DiariZen continues to outperform
the fine-tuned Pyannote baseline, while DiCoW sees further gains from domain
adaptation. Our final system achieves a micro-average tcpWER/CER of 16.75% and
ranks second in Task 2 of the MLC-SLM challenge. Lastly, we identify several
labeling inconsistencies in the training data -- such as missing speech
segments and incorrect silence annotations -- which can hinder diarization
fine-tuning. We propose simple mitigation strategies to address these issues
and improve system robustness.
ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning
  in LLMs
Authors: name: Feng He, name: Zijun Chen, name: Xinnian Liang, name: Tingting Ma, name: Yunqi Qiu, name: Shuangzhi Wu, name: Junchi Yan
Published: 2025-06-18 07:44:09

Recent advances in Large Reasoning Models (LRMs) trained with Long
Chain-of-Thought (Long CoT) reasoning have demonstrated remarkable cross-domain
generalization capabilities. However, the underlying mechanisms supporting such
transfer remain poorly understood. We hypothesize that cross-domain
generalization arises from shared abstract reasoning prototypes -- fundamental
reasoning patterns that capture the essence of problems across domains. These
prototypes minimize the nuances of the representation, revealing that seemingly
diverse tasks are grounded in shared reasoning structures.Based on this
hypothesis, we propose ProtoReasoning, a framework that enhances the reasoning
ability of LLMs by leveraging scalable and verifiable prototypical
representations (Prolog for logical reasoning, PDDL for
planning).ProtoReasoning features: (1) an automated prototype construction
pipeline that transforms problems into corresponding prototype representations;
(2) a comprehensive verification system providing reliable feedback through
Prolog/PDDL interpreters; (3) the scalability to synthesize problems
arbitrarily within prototype space while ensuring correctness. Extensive
experiments show that ProtoReasoning achieves 4.7% improvement over baseline
models on logical reasoning (Enigmata-Eval), 6.3% improvement on planning
tasks, 4.0% improvement on general reasoning (MMLU) and 1.0% on mathematics
(AIME24). Significantly, our ablation studies confirm that learning in
prototype space also demonstrates enhanced generalization to structurally
similar problems compared to training solely on natural language
representations, validating our hypothesis that reasoning prototypes serve as
the foundation for generalizable reasoning in large language models.
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Authors: name: Thomas Kuntz, name: Agatha Duzan, name: Hao Zhao, name: Francesco Croce, name: Zico Kolter, name: Nicolas Flammarion, name: Maksym Andriushchenko
Published: 2025-06-17 17:59:31

Computer use agents are LLM-based agents that can directly interact with a
graphical user interface, by processing screenshots or accessibility trees.
While these systems are gaining popularity, their safety has been largely
overlooked, despite the fact that evaluating and understanding their potential
for harmful behavior is essential for widespread adoption. To address this gap,
we introduce OS-Harm, a new benchmark for measuring safety of computer use
agents. OS-Harm is built on top of the OSWorld environment and aims to test
models across three categories of harm: deliberate user misuse, prompt
injection attacks, and model misbehavior. To cover these cases, we create 150
tasks that span several types of safety violations (harassment, copyright
infringement, disinformation, data exfiltration, etc.) and require the agent to
interact with a variety of OS applications (email client, code editor, browser,
etc.). Moreover, we propose an automated judge to evaluate both accuracy and
safety of agents that achieves high agreement with human annotations (0.76 and
0.79 F1 score). We evaluate computer use agents based on a range of frontier
models - such as o4-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro - and provide
insights into their safety. In particular, all models tend to directly comply
with many deliberate misuse queries, are relatively vulnerable to static prompt
injections, and occasionally perform unsafe actions. The OS-Harm benchmark is
available at https://github.com/tml-epfl/os-harm.