NeurIPS 2025

ALMGuard: Safety Shortcuts and Where to Find
Them as Guardrails for Audio–Language Models
Weifei Jin1, Yuxin Cao2, Junjie Su1, Minhui Xue3,4, Jie Hao1∗,
Ke Xu5, Jin Song Dong2, Derui Wang3
1Beijing University of Posts and Telecommunications
2National University of Singapore
3CSIRO’s Data61
4 Responsible AI Research (RAIR) Centre, The University of Adelaide
5Tsinghua University
Abstract
Recent advances in Audio-Language Models (ALMs) have significantly improved
multimodal understanding capabilities. However, the introduction of the audio
modality also brings new and unique vulnerability vectors. Previous studies have
proposed jailbreak attacks that specifically target ALMs, revealing that defenses di-
rectly transferred from traditional audio adversarial attacks or text-based Large Lan-
guage Model (LLM) jailbreaks are largely ineffective against these ALM-specific
threats. To address this issue, we propose ALMGuard, the first defense framework
tailored to ALMs. Based on the assumption that safety-aligned shortcuts naturally
exist in ALMs, we design a method to identify universal Shortcut Activation Pertur-
bations (SAPs) that serve as triggers that activate the safety shortcuts to safeguard
ALMs at inference time. To better sift out effective triggers while preserving the
model’s utility on benign tasks, we further propose Mel-Gradient Sparse Mask
(M-GSM), which restricts perturbations to Mel-frequency bins that are sensitive to
jailbreaks but insensitive to speech understanding. Both theoretical analyses and
empirical results demonstrate the robustness of our method against both seen and
unseen attacks. Overall, ALMGuard reduces the average success rate of advanced
ALM-specific jailbreak attacks to 4.6% across four models, while maintaining com-
parable utility on benign benchmarks, establishing it as the new state of the art. Our
code and data are available at https://github.com/WeifeiJin/ALMGuard.
1
Introduction
Audio-Language Models (ALMs) [18, 52] are revolutio...