Orionnnn/hub-stats huggingface dataset:
Unified Scaling Laws for Compressed Representations
Authors: name: Andrei Panferov, name: Alexandra Volkova, name: Ionut-Vlad Modoranu, name: Vage Egiazarian, name: Mher Safaryan, name: Dan Alistarh
Published: 2025-06-02 16:52:51

Scaling laws have shaped recent advances in machine learning by enabling
predictable scaling of model performance based on model size, computation, and
data volume. Concurrently, the rise in computational cost for AI has motivated
model compression techniques, notably quantization and sparsification, which
have emerged to mitigate the steep computational demands associated with
large-scale training and inference. This paper investigates the interplay
between scaling laws and compression formats, exploring whether a unified
scaling framework can accurately predict model performance when training occurs
over various compressed representations, such as sparse, scalar-quantized,
sparse-quantized or even vector-quantized formats. Our key contributions
include validating a general scaling law formulation and showing that it is
applicable both individually but also composably across compression types.
Based on this, our main finding is demonstrating both theoretically and
empirically that there exists a simple "capacity" metric -- based on the
representation's ability to fit random Gaussian data -- which can robustly
predict parameter efficiency across multiple compressed representations. On the
practical side, we extend our formulation to directly compare the accuracy
potential of different compressed formats, and to derive better algorithms for
training over sparse-quantized formats.
Pitfalls in Evaluating Language Model Forecasters
Authors: name: Daniel Paleka, name: Shashwat Goel, name: Jonas Geiping, name: Florian Tramèr
Published: 2025-05-31 21:49:17

Large language models (LLMs) have recently been applied to forecasting tasks,
with some works claiming these systems match or exceed human performance. In
this paper, we argue that, as a community, we should be careful about such
conclusions as evaluating LLM forecasters presents unique challenges. We
identify two broad categories of issues: (1) difficulty in trusting evaluation
results due to many forms of temporal leakage, and (2) difficulty in
extrapolating from evaluation performance to real-world forecasting. Through
systematic analysis and concrete examples from prior work, we demonstrate how
evaluation flaws can raise concerns about current and future performance
claims. We argue that more rigorous evaluation methodologies are needed to
confidently assess the forecasting abilities of LLMs.
CityLens: Benchmarking Large Language-Vision Models for Urban
  Socioeconomic Sensing
Authors: name: Tianhui Liu, name: Jie Feng, name: Hetian Pang, name: Xin Zhang, name: Tianjian Ouyang, name: Zhiyuan Zhang, name: Yong Li
Published: 2025-05-31 12:25:33

Understanding urban socioeconomic conditions through visual data is a
challenging yet essential task for sustainable urban development and policy
planning. In this work, we introduce CityLens, a comprehensive
benchmark designed to evaluate the capabilities of large language-vision models
(LLVMs) in predicting socioeconomic indicators from satellite and street view
imagery. We construct a multi-modal dataset covering a total of 17 globally
distributed cities, spanning 6 key domains: economy, education, crime,
transport, health, and environment, reflecting the multifaceted nature of urban
life. Based on this dataset, we define 11 prediction tasks and utilize three
evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation,
and Feature-Based Regression. We benchmark 17 state-of-the-art LLVMs across
these tasks. Our results reveal that while LLVMs demonstrate promising
perceptual and reasoning capabilities, they still exhibit limitations in
predicting urban socioeconomic indicators. CityLens provides a unified
framework for diagnosing these limitations and guiding future efforts in using
LLVMs to understand and predict urban socioeconomic patterns. Our codes and
datasets are open-sourced via https://github.com/tsinghua-fib-lab/CityLens.
Stress-testing Machine Generated Text Detection: Shifting Language
  Models Writing Style to Fool Detectors
Authors: name: Andrea Pedrotti, name: Michele Papucci, name: Cristiano Ciaccio, name: Alessio Miaschi, name: Giovanni Puccetti, name: Felice Dell'Orletta, name: Andrea Esuli
Published: 2025-05-30 12:33:30

Recent advancements in Generative AI and Large Language Models (LLMs) have
enabled the creation of highly realistic synthetic content, raising concerns
about the potential for malicious use, such as misinformation and manipulation.
Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the
lack of robust benchmarks that assess generalization to real-world scenarios.
In this work, we present a pipeline to test the resilience of state-of-the-art
MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed
adversarial attacks. To challenge the detectors, we fine-tune language models
using Direct Preference Optimization (DPO) to shift the MGT style toward
human-written text (HWT). This exploits the detectors' reliance on stylistic
clues, making new generations more challenging to detect. Additionally, we
analyze the linguistic shifts induced by the alignment and which features are
used by detectors to detect MGT texts. Our results show that detectors can be
easily fooled with relatively few examples, resulting in a significant drop in
detection performance. This highlights the importance of improving detection
methods and making them robust to unseen in-domain texts.
Synthesis of discrete-continuous quantum circuits with multimodal
  diffusion models
Authors: name: Florian Fürrutter, name: Zohim Chandani, name: Ikko Hamamura, name: Hans J. Briegel, name: Gorka Muñoz-Gil
Published: 2025-06-02 13:35:33

Efficiently compiling quantum operations remains a major bottleneck in
scaling quantum computing. Today's state-of-the-art methods achieve low
compilation error by combining search algorithms with gradient-based parameter
optimization, but they incur long runtimes and require multiple calls to
quantum hardware or expensive classical simulations, making their scaling
prohibitive. Recently, machine-learning models have emerged as an alternative,
though they are currently restricted to discrete gate sets. Here, we introduce
a multimodal denoising diffusion model that simultaneously generates a
circuit's structure and its continuous parameters for compiling a target
unitary. It leverages two independent diffusion processes, one for discrete
gate selection and one for parameter prediction. We benchmark the model over
different experiments, analyzing the method's accuracy across varying qubit
counts, circuit depths, and proportions of parameterized gates. Finally, by
exploiting its rapid circuit generation, we create large datasets of circuits
for particular operations and use these to extract valuable heuristics that can
help us discover new insights into quantum circuit synthesis.
MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech
  Paralinguistic and Affect Labeling
Authors: name: Yifan Cheng, name: Ruoyi Zhang, name: Jiatong Shi
Published: 2025-05-21 17:23:12

Acquiring large-scale emotional speech data with strong consistency remains a
challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated
multimodal pipeline for extracting high-consistency emotional speech from
unlabeled video data. Leveraging face detection and tracking algorithms, we
developed an automatic emotion analysis system using a multimodal large
language model (MLLM). Our results demonstrate that MIKU-PAL can achieve
human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss
kappa score) while being much cheaper and faster than human annotation. With
the high-quality, flexible, and consistent annotation from MIKU-PAL, we can
annotate fine-grained speech emotion categories of up to 26 types, validated by
human annotators with 83% rationality ratings. Based on our proposed system, we
further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2
hours) as a new benchmark for emotional text-to-speech and visual voice
cloning.
SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware
  Reinforcement Learning
Authors: name: Zhongwei Wan, name: Zhihao Dou, name: Che Liu, name: Yu Zhang, name: Dongfei Cui, name: Qinjian Zhao, name: Hui Shen, name: Jing Xiong, name: Yi Xin, name: Yifan Jiang, name: Yangfan He, name: Mi Zhang, name: Shen Yan
Published: 2025-06-02 14:21:44

Multimodal large language models (MLLMs) have shown promising capabilities in
reasoning tasks, yet still struggle with complex problems requiring explicit
self-reflection and self-correction, especially compared to their unimodal
text-based counterparts. Existing reflection methods are simplistic and
struggle to generate meaningful and instructive feedback, as the reasoning
ability and knowledge limits of pre-trained models are largely fixed during
initial training. To overcome these challenges, we propose Multimodal
Self-Reflection enhanced reasoning with Group Relative Policy Optimization
(SRPO), a two-stage reflection-aware reinforcement learning (RL) framework
explicitly designed to enhance multimodal LLM reasoning. In the first stage, we
construct a high-quality, reflection-focused dataset under the guidance of an
advanced MLLM, which generates reflections based on initial responses to help
the policy model learn both reasoning and self-reflection. In the second stage,
we introduce a novel reward mechanism within the GRPO framework that encourages
concise and cognitively meaningful reflection while avoiding redundancy.
Extensive experiments across multiple multimodal reasoning benchmarks,
including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B
and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms
state-of-the-art models, achieving notable improvements in both reasoning
accuracy and reflection quality.
OmniResponse: Online Multimodal Conversational Response Generation in
  Dyadic Interactions
Authors: name: Cheng Luo, name: Jianghui Wang, name: Bing Li, name: Siyang Song, name: Bernard Ghanem
Published: 2025-05-27 20:12:46

In this paper, we introduce Online Multimodal Conversational Response
Generation (OMCRG), a novel task that aims to online generate synchronized
verbal and non-verbal listener feedback, conditioned on the speaker's
multimodal input. OMCRG reflects natural dyadic interactions and poses new
challenges in achieving synchronization between the generated audio and facial
responses of the listener. To address these challenges, we innovatively
introduce text as an intermediate modality to bridge the audio and facial
responses. We hence propose OmniResponse, a Multimodal Large Language Model
(MLLM) that autoregressively generates high-quality multi-modal listener
responses. OmniResponse leverages a pretrained LLM enhanced with two novel
components: Chrono-Text, which temporally anchors generated text tokens, and
TempoVoice, a controllable online TTS module that produces speech synchronized
with facial reactions. To support further OMCRG research, we present
ResponseNet, a new dataset comprising 696 high-quality dyadic interactions
featuring synchronized split-screen videos, multichannel audio, transcripts,
and facial behavior annotations. Comprehensive evaluations conducted on
ResponseNet demonstrate that OmniResponse significantly outperforms baseline
models in terms of semantic speech content, audio-visual synchronization, and
generation quality.
From Guidelines to Practice: A New Paradigm for Arabic Language Model
  Evaluation
Authors: name: Serry Sibaee, name: Omer Nacar, name: Adel Ammar, name: Yasser Al-Habashi, name: Abdulrahman Al-Batati, name: Wadii Boulila
Published: 2025-06-02 17:39:50

This paper addresses critical gaps in Arabic language model evaluation by
establishing comprehensive theoretical guidelines and introducing a novel
evaluation framework. We first analyze existing Arabic evaluation datasets,
identifying significant issues in linguistic accuracy, cultural alignment, and
methodological rigor. To address these limitations in LLMs, we present the
Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490
challenging questions spanning ten major domains (42 sub-domains, see Figure 1.
Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet,
Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant
variations in model performance across different domains, with particular
challenges in areas requiring deep cultural understanding and specialized
knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%,
showing relative strength in mathematical theory in Arabic, Arabic language,
and islamic domains. This work provides both theoretical foundations and
practical insights for improving Arabic language model evaluation, emphasizing
the importance of cultural competence alongside technical capabilities.
LLM in the Loop: Creating the PARADEHATE Dataset for Hate Speech
  Detoxification
Authors: name: Shuzhou Yuan, name: Ercong Nie, name: Lukas Kouba, name: Ashish Yashwanth Kangen, name: Helmut Schmid, name: Hinrich Schutze, name: Michael Farber
Published: 2025-06-02 09:45:05

Detoxification, the task of rewriting harmful language into non-toxic text,
has become increasingly important amid the growing prevalence of toxic content
online. However, high-quality parallel datasets for detoxification, especially
for hate speech, remain scarce due to the cost and sensitivity of human
annotation. In this paper, we propose a novel LLM-in-the-loop pipeline
leveraging GPT-4o-mini for automated detoxification. We first replicate the
ParaDetox pipeline by replacing human annotators with an LLM and show that the
LLM performs comparably to human annotation. Building on this, we construct
PARADEHATE, a large-scale parallel dataset specifically for hatespeech
detoxification. We release PARADEHATE as a benchmark of over 8K hate/non-hate
text pairs and evaluate a wide range of baseline methods. Experimental results
show that models such as BART, fine-tuned on PARADEHATE, achieve better
performance in style accuracy, content preservation, and fluency, demonstrating
the effectiveness of LLM-generated detoxification text as a scalable
alternative to human annotation.
LIFT the Veil for the Truth: Principal Weights Emerge after Rank
  Reduction for Reasoning-Focused Supervised Fine-Tuning
Authors: name: Zihang Liu, name: Tianyu Pang, name: Oleg Balabanov, name: Chaoqun Yang, name: Tianjin Huang, name: Lu Yin, name: Yaoqing Yang, name: Shiwei Liu
Published: 2025-06-01 01:31:50

Recent studies have shown that supervised fine-tuning of LLMs on a small
number of high-quality datasets can yield strong reasoning capabilities.
However, full fine-tuning (Full FT), while powerful, is computationally
expensive and susceptible to overfitting and catastrophic forgetting,
particularly when data is limited. Sparse fine-tuning, which previously
achieved notable success by updating only a small subset of model parameters,
offers a promising trade-off between efficiency and effectiveness. Yet, it has
lagged behind in the LLM era due to the difficulty of identifying parameters
truly critical for reasoning. In this work, we state that weights with the
largest magnitude after low-rank approximation are critical weights for
fine-tuning, which we call Principal Weights. Surprisingly, while
magnitude-based sparse fine-tuning performs poorly as a baseline on LLM
fine-tuning, it becomes highly effective after rank reduction. These insights
motivate our method: Low-rank Informed Sparse Fine-Tuning (LIFT). LIFT only
updates the top 5% Principal Weights throughout training and consistently
achieves better performance on reasoning tasks than Full FT, while maintaining
memory efficiency on par with popular parameter-efficient fine-tuning methods.
In addition to strong performance on target domains such as arithmetic
reasoning, LIFT also retains up to 20% more source-domain knowledge, compared
to Full FT and LoRA. Our code is available at:
https://github.com/zihanghliu/LIFT.
Normalized Attention Guidance: Universal Negative Guidance for Diffusion
  Model
Authors: name: Dar-Yen Chen, name: Hmrishav Bandyopadhyay, name: Kai Zou, name: Yi-Zhe Song
Published: 2025-05-27 13:30:46

Negative guidance -- explicitly suppressing unwanted attributes -- remains a
fundamental challenge in diffusion models, particularly in few-step sampling
regimes. While Classifier-Free Guidance (CFG) works well in standard settings,
it fails under aggressive sampling step compression due to divergent
predictions between positive and negative branches. We present Normalized
Attention Guidance (NAG), an efficient, training-free mechanism that applies
extrapolation in attention space with L1-based normalization and refinement.
NAG restores effective negative guidance where CFG collapses while maintaining
fidelity. Unlike existing approaches, NAG generalizes across architectures
(UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image,
video), functioning as a universal plug-in with minimal computational
overhead. Through extensive experimentation, we demonstrate consistent
improvements in text alignment (CLIP Score), fidelity (FID, PFID), and
human-perceived quality (ImageReward). Our ablation studies validate each
design component, while user studies confirm significant preference for
NAG-guided outputs. As a model-agnostic inference-time approach requiring no
retraining, NAG provides effortless negative guidance for all modern diffusion
frameworks -- pseudocode in the Appendix!
ComposeAnything: Composite Object Priors for Text-to-Image Generation
Authors: name: Zeeshan Khan, name: Shizhe Chen, name: Cordelia Schmid
Published: 2025-05-30 00:13:36

Generating images from text involving complex and novel object arrangements
remains a significant challenge for current text-to-image (T2I) models.
Although prior layout-based methods improve object arrangements using spatial
constraints with 2D layouts, they often struggle to capture 3D positioning and
sacrifice quality and coherence. In this work, we introduce ComposeAnything, a
novel framework for improving compositional image generation without retraining
existing T2I models. Our approach first leverages the chain-of-thought
reasoning abilities of LLMs to produce 2.5D semantic layouts from text,
consisting of 2D object bounding boxes enriched with depth information and
detailed captions. Based on this layout, we generate a spatial and depth aware
coarse composite of objects that captures the intended composition, serving as
a strong and interpretable prior that replaces stochastic noise initialization
in diffusion-based T2I models. This prior guides the denoising process through
object prior reinforcement and spatial-controlled denoising, enabling seamless
generation of compositional objects and coherent backgrounds, while allowing
refinement of inaccurate priors. ComposeAnything outperforms state-of-the-art
methods on the T2I-CompBench and NSR-1K benchmarks for prompts with 2D/3D
spatial arrangements, high object counts, and surreal compositions. Human
evaluations further demonstrate that our model generates high-quality images
with compositions that faithfully reflect the text.
WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web
  Tasks
Authors: name: Atsuyuki Miyai, name: Zaiying Zhao, name: Kazuki Egashira, name: Atsuki Sato, name: Tatsumi Sunada, name: Shota Onohara, name: Hiromasa Yamanishi, name: Mashiro Toyooka, name: Kunato Nishina, name: Ryoma Maeda, name: Kiyoharu Aizawa, name: Toshihiko Yamasaki
Published: 2025-06-02 17:59:45

Powered by a large language model (LLM), a web browsing agent operates web
browsers in a human-like manner and offers a highly transparent path toward
automating a wide range of everyday tasks. As web agents become increasingly
capable and demonstrate proficiency in general browsing tasks, a critical
question emerges: Can they go beyond general browsing to robustly handle tasks
that are tedious and complex, or chores that humans often avoid doing
themselves? In this paper, we introduce WebChoreArena, a new fully reproducible
benchmark comprising 532 carefully curated tasks designed to extend the scope
of WebArena beyond general browsing to more labor-intensive and tedious tasks.
WebChoreArena systematically integrates three key challenges: (i) Massive
Memory tasks requiring accurate retrieval of large amounts of information in
the observations, (ii) Calculation tasks demanding precise mathematical
reasoning, and (iii) Long-Term Memory tasks necessitating long-term memory
across multiple webpages. Built on top of the fully reproducible and widely
adopted four WebArena simulation environments, WebChoreArena ensures strict
reproducibility and enables fair, direct comparisons with the established
WebArena benchmark, offering key insights into agent progress. Our experimental
results demonstrate that as LLMs evolve, represented by GPT-4o, Claude 3.7
Sonnet, and Gemini 2.5 Pro, significant improvements in performance are
observed on WebChoreArena. These findings suggest that WebChoreArena is
well-suited to measure the advancement of state-of-the-art LLMs with greater
clarity. Nevertheless, the results also indicate that even with Gemini 2.5 Pro,
there remains substantial room for improvement compared to WebArena,
highlighting the increased challenges posed by WebChoreArena.
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient
  Robotics
Authors: name: Mustafa Shukor, name: Dana Aubakirova, name: Francesco Capuano, name: Pepijn Kooijmans, name: Steven Palma, name: Adil Zouitine, name: Michel Aractingi, name: Caroline Pascal, name: Martino Russi, name: Andres Marafioti, name: Simon Alibert, name: Matthieu Cord, name: Thomas Wolf, name: Remi Cadene
Published: 2025-06-02 16:30:19

Vision-language models (VLMs) pretrained on large-scale multimodal datasets
encode rich visual and linguistic knowledge, making them a strong foundation
for robotics. Rather than training robotic policies from scratch, recent
approaches adapt VLMs into vision-language-action (VLA) models that enable
natural language-driven perception and control. However, existing VLAs are
typically massive--often with billions of parameters--leading to high training
costs and limited real-world deployability. Moreover, they rely on academic and
industrial datasets, overlooking the growing availability of
community-collected data from affordable robotic platforms. In this work, we
present SmolVLA, a small, efficient, and community-driven VLA that drastically
reduces both training and inference costs, while retaining competitive
performance. SmolVLA is designed to be trained on a single GPU and deployed on
consumer-grade GPUs or even CPUs. To further improve responsiveness, we
introduce an asynchronous inference stack decoupling perception and action
prediction from action execution, allowing higher control rates with chunked
action generation. Despite its compact size, SmolVLA achieves performance
comparable to VLAs that are 10x larger. We evaluate SmolVLA on a range of both
simulated as well as real-world robotic benchmarks and release all code,
pretrained models, and training data.