Orionnnn/hub-stats huggingface dataset:
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
Authors: name: Shufan Li, name: Konstantinos Kallidromitis, name: Hritik Bansal, name: Akash Gokul, name: Yusuke Kato, name: Kazuki Kozuka, name: Jason Kuen, name: Zhe Lin, name: Kai-Wei Chang, name: Aditya Grover
Published: 2025-05-22 16:07:12

Modern Vision-Language Models (VLMs) can solve a wide range of tasks
requiring visual reasoning. In real-world scenarios, desirable properties for
VLMs include fast inference and controllable generation (e.g., constraining
outputs to adhere to a desired format). However, existing autoregressive (AR)
VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs)
offer a promising alternative, enabling parallel decoding for faster inference
and bidirectional context for controllable generation through text-infilling.
While effective in language-only settings, DMs' potential for multimodal tasks
is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build
LaViDa by equipping DMs with a vision encoder and jointly fine-tune the
combined parts for multimodal instruction following. To address challenges
encountered, LaViDa incorporates novel techniques such as complementary masking
for effective training, prefix KV cache for efficient inference, and timestep
shifting for high-quality sampling. Experiments show that LaViDa achieves
competitive or superior performance to AR VLMs on multi-modal benchmarks such
as MMMU, while offering unique advantages of DMs, including flexible
speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO
captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x
speedup. On bidirectional tasks, it achieves +59% improvement on Constrained
Poem Completion. These results demonstrate LaViDa as a strong alternative to AR
VLMs. Code and models will be released in the camera-ready version.
Reinforcement Learning Finetunes Small Subnetworks in Large Language
  Models
Authors: name: Sagnik Mukherjee, name: Lifan Yuan, name: Dilek Hakkani-Tur, name: Hao Peng
Published: 2025-05-16 21:42:28

Reinforcement learning (RL) yields substantial improvements in large language
models (LLMs) downstream task performance and alignment with human values.
Surprisingly, such large gains result from updating only a small subnetwork
comprising just 5 percent to 30 percent of the parameters, with the rest
effectively unchanged. We refer to this phenomenon as parameter update sparsity
induced by RL. It is observed across all 7 widely used RL algorithms (e.g.,
PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.
This sparsity is intrinsic and occurs without any explicit sparsity promoting
regularizations or architectural constraints. Finetuning the subnetwork alone
recovers the test accuracy, and, remarkably, produces a model nearly identical
to the one obtained via full finetuning. The subnetworks from different random
seeds, training data, and even RL algorithms show substantially greater overlap
than expected by chance. Our analysis suggests that this sparsity is not due to
updating only a subset of layers, instead, nearly all parameter matrices
receive similarly sparse updates. Moreover, the updates to almost all parameter
matrices are nearly full-rank, suggesting RL updates a small subset of
parameters that nevertheless span almost the full subspaces that the parameter
matrices can represent. We conjecture that the this update sparsity can be
primarily attributed to training on data that is near the policy distribution,
techniques that encourage the policy to remain close to the pretrained model,
such as the KL regularization and gradient clipping, have limited impact.
In-Context Learning Boosts Speech Recognition via Human-like Adaptation
  to Speakers and Language Varieties
Authors: name: Nathan Roll, name: Calbert Graham, name: Yuka Tatsumi, name: Kim Tien Nguyen, name: Meghan Sumner, name: Dan Jurafsky
Published: 2025-05-20 20:20:37

Human listeners readily adjust to unfamiliar speakers and language varieties
through exposure, but do these adaptation benefits extend to state-of-the-art
spoken language models? We introduce a scalable framework that allows for
in-context learning (ICL) in Phi-4 Multimodal using interleaved task prompts
and audio-text pairs, and find that as few as 12 example utterances (~50
seconds) at inference time reduce word error rates by a relative 19.7% (1.2
pp.) on average across diverse English corpora. These improvements are most
pronounced in low-resource varieties, when the context and target speaker
match, and when more examples are provided--though scaling our procedure yields
diminishing marginal returns to context length. Overall, we find that our novel
ICL adaptation scheme (1) reveals a similar performance profile to human
listeners, and (2) demonstrates consistent improvements to automatic speech
recognition (ASR) robustness across diverse speakers and language backgrounds.
While adaptation succeeds broadly, significant gaps remain for certain
varieties, revealing where current models still fall short of human
flexibility. We release our prompts and code on GitHub.
BLEUBERI: BLEU is a surprisingly effective reward for instruction
  following
Authors: name: Yapei Chang, name: Yekyung Kim, name: Michael Krumdick, name: Amir Zadeh, name: Chuan Li, name: Chris Tanner, name: Mohit Iyyer
Published: 2025-05-16 10:11:43

Reward models are central to aligning LLMs with human preferences, but they
are costly to train, requiring large-scale human-labeled preference data and
powerful pretrained LLM backbones. Meanwhile, the increasing availability of
high-quality synthetic instruction-following datasets raises the question: can
simpler, reference-based metrics serve as viable alternatives to reward models
during RL-based alignment? In this paper, we show first that BLEU, a basic
string-matching metric, surprisingly matches strong reward models in agreement
with human preferences on general instruction-following datasets. Based on this
insight, we develop BLEUBERI, a method that first identifies challenging
instructions and then applies Group Relative Policy Optimization (GRPO) using
BLEU directly as the reward function. We demonstrate that BLEUBERI-trained
models are competitive with models trained via reward model-guided RL across
four challenging instruction-following benchmarks and three different base
language models. A human evaluation further supports that the quality of
BLEUBERI model outputs is on par with those from reward model-aligned models.
Moreover, BLEUBERI models generate outputs that are more factually grounded
than competing methods. Overall, we show that given access to high-quality
reference outputs (easily obtained via existing instruction-following datasets
or synthetic data generation), string matching-based metrics are cheap yet
effective proxies for reward models during alignment. We release our code and
data at https://github.com/lilakk/BLEUBERI.
The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Authors: name: Shivam Agarwal, name: Zimin Zhang, name: Lifan Yuan, name: Jiawei Han, name: Hao Peng
Published: 2025-05-21 05:39:11

Entropy minimization (EM) trains the model to concentrate even more
probability mass on its most confident outputs. We show that this simple
objective alone, without any labeled data, can substantially improve large
language models' (LLMs) performance on challenging math, physics, and coding
tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy
similarly to instruction finetuning, but on unlabeled outputs drawn from the
model; (2) EM-RL: reinforcement learning with negative entropy as the only
reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce
entropy without any training data or parameter updates. On Qwen-7B, EM-RL,
without any labeled data, achieves comparable or better performance than strong
RL baselines such as GRPO and RLOO that are trained on 60K labeled examples.
Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of
proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the
challenging SciCode benchmark, while being 3x more efficient than
self-consistency and sequential refinement. Our findings reveal that many
pretrained LLMs possess previously underappreciated reasoning capabilities that
can be effectively elicited through entropy minimization alone, without any
labeled data or even any parameter updates.
DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion
  Modeling
Authors: name: Yuang Ai, name: Qihang Fan, name: Xuefeng Hu, name: Zhenheng Yang, name: Ran He, name: Huaibo Huang
Published: 2025-05-16 12:54:04

Diffusion Transformer (DiT), a promising diffusion model for visual
generation, demonstrates impressive performance but incurs significant
computational overhead. Intriguingly, analysis of pre-trained DiT models
reveals that global self-attention is often redundant, predominantly capturing
local patterns-highlighting the potential for more efficient alternatives. In
this paper, we revisit convolution as an alternative building block for
constructing efficient and expressive diffusion models. However, naively
replacing self-attention with convolution typically results in degraded
performance. Our investigations attribute this performance gap to the higher
channel redundancy in ConvNets compared to Transformers. To resolve this, we
introduce a compact channel attention mechanism that promotes the activation of
more diverse channels, thereby enhancing feature diversity. This leads to
Diffusion ConvNet (DiCo), a family of diffusion models built entirely from
standard ConvNet modules, offering strong generative performance with
significant efficiency gains. On class-conditional ImageNet benchmarks, DiCo
outperforms previous diffusion models in both image quality and generation
speed. Notably, DiCo-XL achieves an FID of 2.05 at 256x256 resolution and 2.53
at 512x512, with a 2.7x and 3.1x speedup over DiT-XL/2, respectively.
Furthermore, our largest model, DiCo-H, scaled to 1B parameters, reaches an FID
of 1.90 on ImageNet 256x256-without any additional supervision during training.
Code: https://github.com/shallowdream204/DiCo.
VerifyBench: Benchmarking Reference-based Reward Systems for Large
  Language Models
Authors: name: Yuchen Yan, name: Jin Jiang, name: Zhenbang Ren, name: Yijun Li, name: Xudong Cai, name: Yang Liu, name: Xin Xu, name: Mengdi Zhang, name: Jian Shao, name: Yongliang Shen, name: Jun Xiao, name: Yueting Zhuang
Published: 2025-05-21 17:54:43

Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved
remarkable performance in the domain of reasoning. A key component of their
training is the incorporation of verifiable rewards within reinforcement
learning (RL). However, existing reward benchmarks do not evaluate
reference-based reward systems, leaving researchers with limited understanding
of the accuracy of verifiers used in RL. In this paper, we introduce two
benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the
performance of reference-based reward systems. These benchmarks are constructed
through meticulous data collection and curation, followed by careful human
annotation to ensure high quality. Current models still show considerable room
for improvement on both VerifyBench and VerifyBench-Hard, especially
smaller-scale models. Furthermore, we conduct a thorough and comprehensive
analysis of evaluation results, offering insights for understanding and
developing reference-based reward systems. Our proposed benchmarks serve as
effective tools for guiding the development of verifier accuracy and the
reasoning capabilities of models trained via RL in reasoning tasks.
BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms
Authors: name: Yunlong Hou, name: Fengzhuo Zhang, name: Cunxiao Du, name: Xuan Zhang, name: Jiachun Pan, name: Tianyu Pang, name: Chao Du, name: Vincent Y. F. Tan, name: Zhuoran Yang
Published: 2025-05-21 05:56:31

Speculative decoding has emerged as a popular method to accelerate the
inference of Large Language Models (LLMs) while retaining their superior text
generation performance. Previous methods either adopt a fixed speculative
decoding configuration regardless of the prefix tokens, or train draft models
in an offline or online manner to align them with the context. This paper
proposes a training-free online learning framework to adaptively choose the
configuration of the hyperparameters for speculative decoding as text is being
generated. We first formulate this hyperparameter selection problem as a
Multi-Armed Bandit problem and provide a general speculative decoding framework
BanditSpec. Furthermore, two bandit-based hyperparameter selection algorithms,
UCBSpec and EXP3Spec, are designed and analyzed in terms of a novel quantity,
the stopping time regret. We upper bound this regret under both stochastic and
adversarial reward settings. By deriving an information-theoretic impossibility
result, it is shown that the regret performance of UCBSpec is optimal up to
universal constants. Finally, extensive empirical experiments with LLaMA3 and
Qwen2 demonstrate that our algorithms are effective compared to existing
methods, and the throughput is close to the oracle best hyperparameter in
simulated real-life LLM serving scenarios with diverse input prompts.
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors
  Approach
Authors: name: Umberto Cappellazzo, name: Minsu Kim, name: Stavros Petridis, name: Daniele Falavigna, name: Alessio Brutti
Published: 2025-05-20 13:20:55

Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy
environments by integrating visual cues. While recent advances integrate Large
Language Models (LLMs) into AVSR, their high computational cost hinders
deployment in resource-constrained settings. To address this, we propose
Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of
Projectors (SMoP) module to scale model capacity without increasing inference
costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors,
Llama-SMoP enables the use of smaller LLMs while maintaining strong
performance. We explore three SMoP configurations and show that Llama-SMoP DEDR
(Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and
experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation
studies confirm its effectiveness in expert activation, scalability, and noise
robustness.
MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation
  of LLM Hallucinations
Authors: name: Ernests Lavrinovics, name: Russa Biswas, name: Katja Hose, name: Johannes Bjerva
Published: 2025-05-20 09:03:35

Large Language Models (LLMs) have inherent limitations of faithfulness and
factuality, commonly referred to as hallucinations. Several benchmarks have
been developed that provide a test bed for factuality evaluation within the
context of English-centric datasets, while relying on supplementary informative
context like web links or text passages but ignoring the available structured
factual resources. To this end, Knowledge Graphs (KGs) have been identified as
a useful aid for hallucination mitigation, as they provide a structured way to
represent the facts about entities and their relations with minimal linguistic
overhead. We bridge the lack of KG paths and multilinguality for factual
language modeling within the existing hallucination evaluation benchmarks and
propose a KG-based multilingual, multihop benchmark called MultiHal
framed for generative text evaluation. As part of our data collection pipeline,
we mined 140k KG-paths from open-domain KGs, from which we pruned noisy
KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation
shows an absolute scale increase by approximately 0.12 to 0.36 points for the
semantic similarity score in KG-RAG over vanilla QA across multiple languages
and multiple models, demonstrating the potential of KG integration. We
anticipate MultiHal will foster future research towards several graph-based
hallucination mitigation and fact-checking tasks.
WebNovelBench: Placing LLM Novelists on the Web Novel Distribution
Authors: name: Leon Lin, name: Jun Zheng, name: Haidong Wang
Published: 2025-05-20 18:32:28

Robustly evaluating the long-form storytelling capabilities of Large Language
Models (LLMs) remains a significant challenge, as existing benchmarks often
lack the necessary scale, diversity, or objective measures. To address this, we
introduce WebNovelBench, a novel benchmark specifically designed for evaluating
long-form novel generation. WebNovelBench leverages a large-scale dataset of
over 4,000 Chinese web novels, framing evaluation as a synopsis-to-story
generation task. We propose a multi-faceted framework encompassing eight
narrative quality dimensions, assessed automatically via an LLM-as-Judge
approach. Scores are aggregated using Principal Component Analysis and mapped
to a percentile rank against human-authored works. Our experiments demonstrate
that WebNovelBench effectively differentiates between human-written
masterpieces, popular web novels, and LLM-generated content. We provide a
comprehensive analysis of 24 state-of-the-art LLMs, ranking their storytelling
abilities and offering insights for future development. This benchmark provides
a scalable, replicable, and data-driven methodology for assessing and advancing
LLM-driven narrative generation.
This Time is Different: An Observability Perspective on Time Series
  Foundation Models
Authors: name: Ben Cohen, name: Emaad Khwaja, name: Youssef Doubli, name: Salahidine Lemaachi, name: Chris Lettieri, name: Charles Masson, name: Hugo Miccinilli, name: Elise Ramé, name: Qiqi Ren, name: Afshin Rostamizadeh, name: Jean Ogier du Terrail, name: Anna-Monica Toon, name: Kan Wang, name: Stephan Xie, name: David Asker, name: Ameet Talwalkar, name: Othmane Abou-Amal
Published: 2025-05-20 17:48:13

We introduce Toto, a time series forecasting foundation model with 151
million parameters. Toto uses a modern decoder-only architecture coupled with
architectural innovations designed to account for specific challenges found in
multivariate observability time series data. Toto's pre-training corpus is a
mixture of observability data, open datasets, and synthetic data, and is
4-10times larger than those of leading time series foundation models.
Additionally, we introduce BOOM, a large-scale benchmark consisting of 350
million observations across 2,807 real-world time series. For both Toto and
BOOM, we source observability data exclusively from Datadog's own telemetry and
internal observability metrics. Extensive evaluations demonstrate that Toto
achieves state-of-the-art performance on both BOOM and on established general
purpose time series forecasting benchmarks. Toto's model weights, inference
code, and evaluation scripts, as well as BOOM's data and evaluation code, are
all available as open source under the Apache 2.0 License available at
https://huggingface.co/Datadog/Toto-Open-Base-1.0 and
https://github.com/DataDog/toto.
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large
  Audio-Language Models
Authors: name: Zirui Song, name: Qian Jiang, name: Mingxuan Cui, name: Mingzhe Li, name: Lang Gao, name: Zeyu Zhang, name: Zixiang Xu, name: Yanbo Wang, name: Chenxi Wang, name: Guangxian Ouyang, name: Zhenhao Chen, name: Xiuying Chen
Published: 2025-05-21 11:47:47

The rise of Large Audio Language Models (LAMs) brings both potential and
risks, as their audio outputs may contain harmful or unethical content.
However, current research lacks a systematic, quantitative evaluation of LAM
safety especially against jailbreak attacks, which are challenging due to the
temporal and semantic nature of speech. To bridge this gap, we introduce
AJailBench, the first benchmark specifically designed to evaluate jailbreak
vulnerabilities in LAMs. We begin by constructing AJailBench-Base, a dataset of
1,495 adversarial audio prompts spanning 10 policy-violating categories,
converted from textual jailbreak attacks using realistic text to speech
synthesis. Using this dataset, we evaluate several state-of-the-art LAMs and
reveal that none exhibit consistent robustness across attacks. To further
strengthen jailbreak testing and simulate more realistic attack conditions, we
propose a method to generate dynamic adversarial variants. Our Audio
Perturbation Toolkit (APT) applies targeted distortions across time, frequency,
and amplitude domains. To preserve the original jailbreak intent, we enforce a
semantic consistency constraint and employ Bayesian optimization to efficiently
search for perturbations that are both subtle and highly effective. This
results in AJailBench-APT, an extended dataset of optimized adversarial audio
samples. Our findings demonstrate that even small, semantically preserved
perturbations can significantly reduce the safety performance of leading LAMs,
underscoring the need for more robust and semantically aware defense
mechanisms.
Evaluate Bias without Manual Test Sets: A Concept Representation
  Perspective for LLMs
Authors: name: Lang Gao, name: Kaiyang Wan, name: Wei Liu, name: Chenxi Wang, name: Zirui Song, name: Zixiang Xu, name: Yanbo Wang, name: Veselin Stoyanov, name: Xiuying Chen
Published: 2025-05-21 13:50:23

Bias in Large Language Models (LLMs) significantly undermines their
reliability and fairness. We focus on a common form of bias: when two reference
concepts in the model's concept space, such as sentiment polarities (e.g.,
"positive" and "negative"), are asymmetrically correlated with a third, target
concept, such as a reviewing aspect, the model exhibits unintended bias. For
instance, the understanding of "food" should not skew toward any particular
sentiment. Existing bias evaluation methods assess behavioral differences of
LLMs by constructing labeled data for different social groups and measuring
model responses across them, a process that requires substantial human effort
and captures only a limited set of social concepts. To overcome these
limitations, we propose BiasLens, a test-set-free bias analysis framework based
on the structure of the model's vector space. BiasLens combines Concept
Activation Vectors (CAVs) with Sparse Autoencoders (SAEs) to extract
interpretable concept representations, and quantifies bias by measuring the
variation in representational similarity between the target concept and each of
the reference concepts. Even without labeled data, BiasLens shows strong
agreement with traditional bias evaluation metrics (Spearman correlation r >
0.85). Moreover, BiasLens reveals forms of bias that are difficult to detect
using existing methods. For example, in simulated clinical scenarios, a
patient's insurance status can cause the LLM to produce biased diagnostic
assessments. Overall, BiasLens offers a scalable, interpretable, and efficient
paradigm for bias discovery, paving the way for improving fairness and
transparency in LLMs.