Orionnnn/hub-stats huggingface dataset:
Train Sparse Autoencoders Efficiently by Utilizing Features Correlation
Authors: name: Vadim Kurochkin, name: Yaroslav Aksenov, name: Daniil Laptev, name: Daniil Gavrilov, name: Nikita Balagansky
Published: 2025-05-28 11:41:11

Sparse Autoencoders (SAEs) have demonstrated significant promise in
interpreting the hidden states of language models by decomposing them into
interpretable latent directions. However, training SAEs at scale remains
challenging, especially when large dictionary sizes are used. While decoders
can leverage sparse-aware kernels for efficiency, encoders still require
computationally intensive linear operations with large output dimensions. To
address this, we propose KronSAE, a novel architecture that factorizes the
latent representation via Kronecker product decomposition, drastically reducing
memory and computational overhead. Furthermore, we introduce mAND, a
differentiable activation function approximating the binary AND operation,
which improves interpretability and performance in our factorized framework.
FAMA: The First Large-Scale Open-Science Speech Foundation Model for
  English and Italian
Authors: name: Sara Papi, name: Marco Gaido, name: Luisa Bentivogli, name: Alessio Brutti, name: Mauro Cettolo, name: Roberto Gretter, name: Marco Matassoni, name: Mohamed Nabih, name: Matteo Negri
Published: 2025-05-28 18:19:34

The development of speech foundation models (SFMs) like Whisper and
SeamlessM4T has significantly advanced the field of speech processing. However,
their closed nature--with inaccessible training data and code--poses major
reproducibility and fair evaluation challenges. While other domains have made
substantial progress toward open science by developing fully transparent models
trained on open-source (OS) code and data, similar efforts in speech remain
limited. To fill this gap, we introduce FAMA, the first family of open science
SFMs for English and Italian, trained on 150k+ hours of OS speech data.
Moreover, we present a new dataset containing 16k hours of cleaned and
pseudo-labeled speech for both languages. Results show that FAMA achieves
competitive performance compared to existing SFMs while being up to 8 times
faster. All artifacts, including code, datasets, and models, are released under
OS-compliant licenses, promoting openness in speech technology research.
Afterburner: Reinforcement Learning Facilitates Self-Improving Code
  Efficiency Optimization
Authors: name: Mingzhe Du, name: Luu Tuan Tuan, name: Yue Liu, name: Yuhao Qing, name: Dong Huang, name: Xinyi He, name: Qian Liu, name: Zejun Ma, name: See-kiong Ng
Published: 2025-05-29 12:14:29

Large Language Models (LLMs) generate functionally correct solutions but
often fall short in code efficiency, a critical bottleneck for real-world
deployment. In this paper, we introduce a novel test-time iterative
optimization framework to address this, employing a closed-loop system where
LLMs iteratively refine code based on empirical performance feedback from an
execution sandbox. We explore three training strategies: Supervised Fine-Tuning
(SFT), Direct Preference Optimization (DPO), and Group Relative Policy
Optimization~(GRPO). Experiments on our Venus dataset and the APPS benchmark
show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO,
using reinforcement learning (RL) with execution feedback, continuously
optimizes code performance, significantly boosting both pass@1 (from 47% to
62%) and the likelihood of outperforming human submissions in efficiency (from
31% to 45%). Our work demonstrates effective test-time code efficiency
improvement and critically reveals the power of RL in teaching LLMs to truly
self-improve code efficiency.
Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or
  True Temporal Understanding?
Authors: name: Bo Feng, name: Zhengfeng Lai, name: Shiyu Li, name: Zizhen Wang, name: Simon Wang, name: Ping Huang, name: Meng Cao
Published: 2025-05-20 13:07:55

Existing video understanding benchmarks often conflate knowledge-based and
purely image-based questions, rather than clearly isolating a model's temporal
reasoning ability, which is the key aspect that distinguishes video
understanding from other modalities. We identify two major limitations that
obscure whether higher scores truly indicate stronger understanding of the
dynamic content in videos: (1) strong language priors, where models can answer
questions without watching the video; and (2) shuffling invariance, where
models maintain similar performance on certain questions even when video frames
are temporally shuffled. To alleviate these issues, we propose VBenchComp, an
automated pipeline that categorizes questions into different domains:
LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions
can be answered without viewing the video; Semantic questions remain answerable
even when the video frames are shuffled; and Temporal questions require
understanding the correct temporal order of frames. The rest of the questions
are labeled as Others. This can enable fine-grained evaluation of different
capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that
are hidden by traditional overall scores, and we offer insights and
recommendations for designing future benchmarks that more accurately assess
video LLMs.
Re-ttention: Ultra Sparse Visual Generation via Attention Statistical
  Reshape
Authors: name: Ruichen Chen, name: Keith G. Mills, name: Liyao Jiang, name: Chao Gao, name: Di Niu
Published: 2025-05-28 22:39:12

Diffusion Transformers (DiT) have become the de-facto model for generating
high-quality visual content like videos and images. A huge bottleneck is the
attention mechanism where complexity scales quadratically with resolution and
video length. One logical way to lessen this burden is sparse attention, where
only a subset of tokens or patches are included in the calculation. However,
existing techniques fail to preserve visual quality at extremely high sparsity
levels and might even incur non-negligible compute overheads. % To address this
concern, we propose Re-ttention, which implements very high sparse attention
for visual generation models by leveraging the temporal redundancy of Diffusion
Models to overcome the probabilistic normalization shift within the attention
mechanism. Specifically, Re-ttention reshapes attention scores based on the
prior softmax distribution history in order to preserve the visual quality of
the full quadratic attention at very high sparsity levels. % Experimental
results on T2V/T2I models such as CogVideoX and the PixArt DiTs demonstrate
that Re-ttention requires as few as 3.1\% of the tokens during inference,
outperforming contemporary methods like FastDiTAttn, Sparse VideoGen and
MInference. Further, we measure latency to show that our method can attain over
45\% end-to-end % and over 92\% self-attention latency reduction on an H100 GPU
at negligible overhead cost.
  Code available online here:
https://github.com/cccrrrccc/Re-ttention{https://github.com/cccrrrccc/Re-ttention}
Unsupervised Word-level Quality Estimation for Machine Translation
  Through the Lens of Annotators (Dis)agreement
Authors: name: Gabriele Sarti, name: Vilém Zouhar, name: Malvina Nissim, name: Arianna Bisazza
Published: 2025-05-29 07:20:36

Word-level quality estimation (WQE) aims to automatically identify
fine-grained error spans in machine-translated outputs and has found many uses,
including assisting translators during post-editing. Modern WQE techniques are
often expensive, involving prompting of large language models or ad-hoc
training on large amounts of human-labeled data. In this work, we investigate
efficient alternatives exploiting recent advances in language model
interpretability and uncertainty quantification to identify translation errors
from the inner workings of translation models. In our evaluation spanning 14
metrics across 12 translation directions, we quantify the impact of human label
variation on metric performance by using multiple sets of human labels. Our
results highlight the untapped potential of unsupervised metrics, the
shortcomings of supervised methods when faced with label uncertainty, and the
brittleness of single-annotator evaluation practices.
VidText: Towards Comprehensive Evaluation for Video Text Understanding
Authors: name: Zhoufaran Yang, name: Yan Shu, name: Zhifei Yang, name: Yan Zhang, name: Yu Li, name: Keyang Lu, name: Gangyan Zeng, name: Shaohui Liu, name: Yu Zhou, name: Nicu Sebe
Published: 2025-05-28 19:39:35

Visual texts embedded in videos carry rich semantic information, which is
crucial for both holistic video understanding and fine-grained reasoning about
local human actions. However, existing video understanding benchmarks largely
overlook textual information, while OCR-specific benchmarks are constrained to
static images, limiting their ability to capture the interaction between text
and dynamic visual contexts. To address this gap, we propose VidText, a new
benchmark designed for comprehensive and in-depth evaluation of video text
understanding. VidText offers the following key features: 1) It covers a wide
range of real-world scenarios and supports multilingual content, encompassing
diverse settings where video text naturally appears. 2) It introduces a
hierarchical evaluation framework with video-level, clip-level, and
instance-level tasks, enabling assessment of both global summarization and
local retrieval capabilities. 3) The benchmark also introduces a set of paired
perception reasoning tasks, ranging from visual text perception to cross-modal
reasoning between textual and visual information. Extensive experiments on 18
state-of-the-art Large Multimodal Models (LMMs) reveal that current models
struggle across most tasks, with significant room for improvement. Further
analysis highlights the impact of both model-intrinsic factors, such as input
resolution and OCR capability, and external factors, including the use of
auxiliary information and Chain-of-Thought reasoning strategies. We hope
VidText will fill the current gap in video understanding benchmarks and serve
as a foundation for future research on multimodal reasoning with video text in
dynamic environments.
Differential Information: An Information-Theoretic Perspective on
  Preference Optimization
Authors: name: Yunjae Won, name: Hyunji Lee, name: Hyeonbin Hwang, name: Minjoon Seo
Published: 2025-05-29 17:59:50

Direct Preference Optimization (DPO) has become a standard technique for
aligning language models with human preferences in a supervised manner. Despite
its empirical success, the theoretical justification behind its log-ratio
reward parameterization remains incomplete. In this work, we address this gap
by utilizing the Differential Information Distribution (DID): a distribution
over token sequences that captures the information gained during policy
updates. First, we show that when preference labels encode the differential
information required to transform a reference policy into a target policy, the
log-ratio reward in DPO emerges as the uniquely optimal form for learning the
target policy via preference optimization. This result naturally yields a
closed-form expression for the optimal sampling distribution over rejected
responses. Second, we find that the condition for preferences to encode
differential information is fundamentally linked to an implicit assumption
regarding log-margin ordered policies-an inductive bias widely used in
preference optimization yet previously unrecognized. Finally, by analyzing the
entropy of the DID, we characterize how learning low-entropy differential
information reinforces the policy distribution, while high-entropy differential
information induces a smoothing effect, which explains the log-likelihood
displacement phenomenon. We validate our theoretical findings in synthetic
experiments and extend them to real-world instruction-following datasets. Our
results suggest that learning high-entropy differential information is crucial
for general instruction-following, while learning low-entropy differential
information benefits knowledge-intensive question answering. Overall, our work
presents a unifying perspective on the DPO objective, the structure of
preference data, and resulting policy behaviors through the lens of
differential information.
Differentiable Solver Search for Fast Diffusion Sampling
Authors: name: Shuai Wang, name: Zexian Li, name: Qipeng zhang, name: Tianhui Song, name: Xubin Li, name: Tiezheng Ge, name: Bo Zheng, name: Limin Wang
Published: 2025-05-27 12:33:43

Diffusion models have demonstrated remarkable generation quality but at the
cost of numerous function evaluations. Recently, advanced ODE-based solvers
have been developed to mitigate the substantial computational demands of
reverse-diffusion solving under limited sampling steps. However, these solvers,
heavily inspired by Adams-like multistep methods, rely solely on t-related
Lagrange interpolation. We show that t-related Lagrange interpolation is
suboptimal for diffusion model and reveal a compact search space comprised of
time steps and solver coefficients. Building on our analysis, we propose a
novel differentiable solver search algorithm to identify more optimal solver.
Equipped with the searched solver, rectified-flow models, e.g., SiT-XL/2 and
FlowDCN-XL/2, achieve FID scores of 2.40 and 2.35, respectively, on ImageNet256
with only 10 steps. Meanwhile, DDPM model, DiT-XL/2, reaches a FID score of
2.33 with only 10 steps. Notably, our searched solver outperforms traditional
solvers by a significant margin. Moreover, our searched solver demonstrates
generality across various model architectures, resolutions, and model sizes.
cadrille: Multi-modal CAD Reconstruction with Online Reinforcement
  Learning
Authors: name: Maksim Kolodiazhnyi, name: Denis Tarasov, name: Dmitrii Zhemchuzhnikov, name: Alexander Nikulin, name: Ilya Zisman, name: Anna Vorontsova, name: Anton Konushin, name: Vladislav Kurenkov, name: Danila Rukhovich
Published: 2025-05-28 22:32:31

Computer-Aided Design (CAD) plays a central role in engineering and
manufacturing, making it possible to create precise and editable 3D models.
Using a variety of sensor or user-provided data as inputs for CAD
reconstruction can democratize access to design applications. However, existing
methods typically focus on a single input modality, such as point clouds,
images, or text, which limits their generalizability and robustness. Leveraging
recent advances in vision-language models (VLM), we propose a multi-modal CAD
reconstruction model that simultaneously processes all three input modalities.
Inspired by large language model (LLM) training paradigms, we adopt a two-stage
pipeline: supervised fine-tuning (SFT) on large-scale procedurally generated
data, followed by reinforcement learning (RL) fine-tuning using online
feedback, obtained programatically. Furthermore, we are the first to explore RL
fine-tuning of LLMs for CAD tasks demonstrating that online RL algorithms such
as Group Relative Preference Optimization (GRPO) outperform offline
alternatives. In the DeepCAD benchmark, our SFT model outperforms existing
single-modal approaches in all three input modalities simultaneously. More
importantly, after RL fine-tuning, cadrille sets new state-of-the-art on three
challenging datasets, including a real-world one.
StressTest: Can YOUR Speech LM Handle the Stress?
Authors: name: Iddo Yosha, name: Gallil Maimon, name: Yossi Adi
Published: 2025-05-28 18:32:56

Sentence stress refers to emphasis, placed on specific words within a spoken
utterance to highlight or contrast an idea, or to introduce new information. It
is often used to imply an underlying intention that is not explicitly stated.
Recent advances in speech-aware language models (SLMs) have enabled direct
processing of audio, allowing models to bypass transcription and access the
full richness of the speech signal and perform audio reasoning tasks such as
spoken question answering. Despite the crucial role of sentence stress in
shaping meaning and speaker intent, it remains largely overlooked in evaluation
and development of such models. In this work, we address this gap by
introducing StressTest, a benchmark specifically designed to evaluate a model's
ability to distinguish between interpretations of spoken sentences based on the
stress pattern. We assess the performance of several leading SLMs and find
that, despite their overall capabilities, they perform poorly on such tasks. To
overcome this limitation, we propose a novel synthetic data generation
pipeline, and create Stress17k, a training set that simulates change of meaning
implied by stress variation. Then, we empirically show that optimizing models
with this synthetic dataset aligns well with real-world recordings and enables
effective finetuning of SLMs. Results suggest, that our finetuned model,
StresSLM, significantly outperforms existing models on both sentence stress
reasoning and detection tasks. Code, models, data, and audio samples -
pages.cs.huji.ac.il/adiyoss-lab/stresstest.
Are Reasoning Models More Prone to Hallucination?
Authors: name: Zijun Yao, name: Yantao Liu, name: Yanxu Chen, name: Jianhui Chen, name: Junfeng Fang, name: Lei Hou, name: Juanzi Li, name: Tat-Seng Chua
Published: 2025-05-29 16:53:41

Recently evolved large reasoning models (LRMs) show powerful performance in
solving complex tasks with long chain-of-thought (CoT) reasoning capability. As
these LRMs are mostly developed by post-training on formal reasoning tasks,
whether they generalize the reasoning capability to help reduce hallucination
in fact-seeking tasks remains unclear and debated. For instance, DeepSeek-R1
reports increased performance on SimpleQA, a fact-seeking benchmark, while
OpenAI-o3 observes even severer hallucination. This discrepancy naturally
raises the following research question: Are reasoning models more prone to
hallucination? This paper addresses the question from three perspectives. (1)
We first conduct a holistic evaluation for the hallucination in LRMs. Our
analysis reveals that LRMs undergo a full post-training pipeline with cold
start supervised fine-tuning (SFT) and verifiable reward RL generally alleviate
their hallucination. In contrast, both distillation alone and RL training
without cold start fine-tuning introduce more nuanced hallucinations. (2) To
explore why different post-training pipelines alters the impact on
hallucination in LRMs, we conduct behavior analysis. We characterize two
critical cognitive behaviors that directly affect the factuality of a LRM: Flaw
Repetition, where the surface-level reasoning attempts repeatedly follow the
same underlying flawed logic, and Think-Answer Mismatch, where the final answer
fails to faithfully match the previous CoT process. (3) Further, we investigate
the mechanism behind the hallucination of LRMs from the perspective of model
uncertainty. We find that increased hallucination of LRMs is usually associated
with the misalignment between model uncertainty and factual accuracy. Our work
provides an initial understanding of the hallucination in LRMs.
Multi-Domain Explainability of Preferences
Authors: name: Nitay Calderon, name: Liat Ein-Dor, name: Roi Reichart
Published: 2025-05-26 15:01:56

Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and
reward models, are central to aligning and evaluating large language models
(LLMs). Yet, the underlying concepts that drive these preferences remain poorly
understood. In this work, we propose a fully automated method for generating
local and global concept-based explanations of preferences across multiple
domains. Our method utilizes an LLM to identify concepts that distinguish
between chosen and rejected responses, and to represent them with concept-based
vectors. To model the relationships between concepts and preferences, we
propose a white-box Hierarchical Multi-Domain Regression model that captures
both domain-general and domain-specific effects. To evaluate our method, we
curate a dataset spanning eight challenging and diverse domains and explain
twelve mechanisms. Our method achieves strong preference prediction
performance, outperforming baselines while also being explainable.
Additionally, we assess explanations in two application-driven settings. First,
guiding LLM outputs with concepts from LaaJ explanations yields responses that
those judges consistently prefer. Second, prompting LaaJs with concepts
explaining humans improves their preference predictions. Together, our work
establishes a new paradigm for explainability in the era of LLMs.
SridBench: Benchmark of Scientific Research Illustration Drawing of
  Image Generation Model
Authors: name: Yifan Chang, name: Yukang Feng, name: Jianwen Sun, name: Jiaxin Ai, name: Chuanhao Li, name: S. Kevin Zhou, name: Kaipeng Zhang
Published: 2025-05-28 08:51:01

Recent years have seen rapid advances in AI-driven image generation. Early
diffusion models emphasized perceptual quality, while newer multimodal models
like GPT-4o-image integrate high-level reasoning, improving semantic
understanding and structural composition. Scientific illustration generation
exemplifies this evolution: unlike general image synthesis, it demands accurate
interpretation of technical content and transformation of abstract ideas into
clear, standardized visuals. This task is significantly more
knowledge-intensive and laborious, often requiring hours of manual work and
specialized tools. Automating it in a controllable, intelligent manner would
provide substantial practical value. Yet, no benchmark currently exists to
evaluate AI on this front. To fill this gap, we introduce SridBench, the first
benchmark for scientific figure generation. It comprises 1,120 instances
curated from leading scientific papers across 13 natural and computer science
disciplines, collected via human experts and MLLMs. Each sample is evaluated
along six dimensions, including semantic fidelity and structural accuracy.
Experimental results reveal that even top-tier models like GPT-4o-image lag
behind human performance, with common issues in text/visual clarity and
scientific correctness. These findings highlight the need for more advanced
reasoning-driven visual generation capabilities.
KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
Authors: name: Jang-Hyun Kim, name: Jinuk Kim, name: Sangwoo Kwon, name: Jae W. Lee, name: Sangdoo Yun, name: Hyun Oh Song
Published: 2025-05-29 13:05:47

Transformer-based large language models (LLMs) cache context as key-value
(KV) pairs during inference. As context length grows, KV cache sizes expand,
leading to substantial memory overhead and increased attention latency. This
paper introduces KVzip, a query-agnostic KV cache eviction method enabling
effective reuse of compressed KV caches across diverse queries. KVzip
quantifies the importance of a KV pair using the underlying LLM to reconstruct
original contexts from cached KV pairs, subsequently evicting pairs with lower
importance. Extensive empirical evaluations demonstrate that KVzip reduces KV
cache size by 3-4times and FlashAttention decoding latency by approximately
2times, with negligible performance loss in question-answering, retrieval,
reasoning, and code comprehension tasks. Evaluations include various models
such as LLaMA3.1-8B, Qwen2.5-14B, and Gemma3-12B, with context lengths reaching
up to 170K tokens. KVzip significantly outperforms existing query-aware KV
eviction methods, which suffer from performance degradation even at a 90% cache
budget ratio under multi-query scenarios.