AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models

Authors: Mintong Kang, Chejian Xu, Bo Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: jailbreak, adversarial attack, audio-language model

Abstract:
Recent advancements in large audio-language models (LALMs) have enabled speech-based user interactions, significantly enhancing user experience and accelerating the deployment of LALMs in real-world applications. However, ensuring the safety of LALMs is crucial to prevent risky outputs that may raise societal concerns or violate AI regulations. Despite the importance of this issue, research on jailbreaking LALMs remains limited due to their recent emergence and the additional technical challenges they present compared to attacks on DNN-based audio models. Specifically, the audio encoders in LALMs, which involve discretization operations, often lead to gradient shattering, hindering the effectiveness of attacks relying on gradient-based optimizations. The behavioral variability of LALMs further complicates the identification of effective (adversarial) optimization targets. Moreover, enforcing stealthiness constraints on adversarial audio waveforms introduces a reduced, non-convex feasible solution space, further intensifying the challenges of the optimization process. To overcome these challenges, we develop AdvWave, the first jailbreak framework against LALMs. We propose a dual-phase optimization method that addresses gradient shattering, enabling effective end-to-end gradient-based optimization. Additionally, we develop an adaptive adversarial target search algorithm that dynamically adjusts the adversarial optimization target based on the response patterns of LALMs for specific queries. To ensure that adversarial audio remains perceptually natural to human listeners, we design a classifier-guided optimization approach that generates adversarial noise resembling common urban sounds. Extensive evaluations on multiple advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving a 40\% higher average jailbreak attack success rate. Both audio stealthiness metrics and human evaluations confirm that adversarial audio generated by AdvWave is indistinguishable from natural sounds. We believe AdvWave will inspire future research aiming to enhance the safety alignment of LALMs, supporting their responsible deployment in real-world scenarios.

Decision: accept (poster)
Reviewer scores: 8, 8, 3, 3

Reviews and discussion:
<thread>
creation date:
1729429291334

thread_type:
official_review

<thread_object>
creator: reviewer_csgt

<content>
summary:
This paper presents an innovative adversarial attack method targeting LALMs, marking the first successful attack on LALMs with optimized audio stealth. The efforts are commendable.

strengths:
The work is groundbreaking as it introduces the first jailbreaking adversarial attack on LALMs. The authors have conducted extensive experimental comparisons, particularly by adapting jailbreaking methods from other domains to the audio sector to ensure the superiority of the proposed attack method. The contribution of this paper is indisputable. However, I still have some questions regarding the audio stealthiness mentioned by the authors.

weaknesses:
I believe the design motivation behind the authors' idea might be flawed. The audio provided clearly contains malicious content, so why consider the stealthiness of the adversarial disturbance? A normal listener would already notice something amiss with the content. Adding adversarial noise to silence segments inevitably leads to listeners hearing malicious content followed by eerie noises, which is utterly unconvincing from a realistic perspective. The authors should more reasonably consider the reasons for the stealthiness of adversarial disturbances and integrate them with the application scenarios of LALMs for a rational design.

questions:
In the supplementary materials provided, I am puzzled about adding adversarial noise: 1. The authors mention that the adversarial noise is naturalized using urban environmental sounds as a masking method. However, I can still hear the traditional adversarial disturbances beyond the environmental sounds, suggesting the presence of two types of perturbations, which the paper does not mention. 2. The attack audio samples provided have adversarial disturbances implanted at the end silence segments of the audio, occupying about half the duration of the audio itself. It's unlikely for such a high proportion of silence in most audio datasets, revealing a serious issue: can adversarial attacks unrestrictively zero-pad benign audio ensure attack success? This seems to relate to the authors' initial claim that audio attacks on LALMs would limit the optimization search space for adversarial disturbances. I imagine the authors extended the audio to ensure sufficient search space, yet this seems impractical in real situations. 3. I am curious why the adversarial disturbances were added to the silence segments. Semantically rich portions of the audio seem more susceptible to attacks, and placing disturbances in silent parts would make the noise more detectable by human ears.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730468766723

thread_type:
official_review

<thread_object>
creator: reviewer_7mqq

<content>
summary:
This paper introduces AdvWave, a framework for conducting white-box adversarial attacks against large audio-language models (LALMs) to elicit harmful information. The authors identify the unique challenges posed by LALMs, such as gradient shattering due to discretisation operations in audio encoders and maintaining stealthiness constraints. To address these issues, AdvWave implements a dual-phase optimisation strategy. The first phase optimises a discrete latent representation to circumvent the gradient shattering issue, while the second phase adjusts the audio waveform itself to align closely with this representation while preserving perceptual naturalness. AdvWave significantly outperforms transferring static jailbreak attacks optimised on text-only LLMs that are subsequently vocalised with text-to-speech (TTS).  The authors argue that their approach highlights the need for more robust defenses and safety measures in LALMs.

strengths:
- **Relevant and timely approach**: LALMs are becoming more prevalent with the recent release of audio capabilities in Gemini and GPT-4o. However, to the best of my knowledge, AdvWave is the first work that has successfully got white-box jailbreak attacks to work.
- **Innovative approach**: AdvWave uses a dual-phase optimisation strategy to address the issue of not being able to backpropagate through the full network when it contains discretisation. They also improve optimisation efficiency by adaptively finding a target string that matches the common structure the model uses for benign requests. These challenges are clearly explained, and the authors provide solutions.
- **Potential for future research**: AdvWave opens several avenues for future work, including further exploration of defensive mechanisms and applying the framework to LALMs that do not have a discretization bottleneck.

weaknesses:
- **AdvWave is not the first approach to jailbreaking LALMs:** The authors of the paper claim that AdvWave is a novel framework for jailbreaking LALMs, but I think this claim needs to be softened throughout the paper. Gemini and GPT4o both have audio-specific jailbreaks, where they vocalise text jailbreaks with TTS in their recent model cards. Therefore, claiming that AdvWave is a novel white-box attack methodology for LALMs is better.
- **Stealthiness constraints are not well-motivated or measured:** It isn’t clear to me why this stealthiness constraint is needed. Any jailbreak, no matter the stealthiness of the input, is bad. Also, stealthiness constraints are not new; they were used in white-box attacks of speech-to-text (STT) models, and the intro doesn’t explain why LALMs make it more difficult. Papers such as “Imperceptible, robust, and targeted adversarial examples for automatic speech recognition” should be cited. Did you ablate changing the whole audio file rather than just the suffix? What motivated the suffix and the environmental noises? You can have imperceptible changes to the audio without that, I believe. Also, what environmental classifier do you use? This needs to be cited for replication purposes.
    - I’m very confused by the stealth metric and why it is a useful comparison to the baselines. The baselines do not have adversarial suffixes added on; they are just TTS reading a harmful prompt. So why isn’t their S_stealth equal to 1? It should be maximally stealthy, just like the vanilla baseline. Also, the baselines do not have a car horn at the end of the utterance, which could be considered more stealthy than your method. You mention you need high stealthiness so it is less detectable by real-world guardrail systems, but I don’t think the results presented demonstrate this. Also, AdvWave is not superior to vanilla in terms of stealth. The three terms in S_stealth are confusing and not well-motivated.
- **Lack of relevant adaptive black-box baselines:** The paper only compares attacks that are optimised on text-only LLMs that are then transferred into audio with TTS. Using TTS to vocalise GCG attacks might not make sense - there could be lots of tokens that can’t be spoken properly so I would expect the attack to be very weak. You say there are no adaptive attacks due to gradient shattering, but there are plenty of good adaptive black box attacks. I expect PAIR and TAP to work well in the audio domain. AdvWave should be evaluated against much stronger baselines than currently used. How strong are the transfer GCG/BEAST attacks to the base text-only LLM? E.g. what is the GCG ASR on Llama3? That would inform if the baselines transfer to the audio domain effectively or if they are broken by the architectural / fine-tuning differences.
- **Lack of clarity on LALM architecture differences, what architecture AdvWave is aimed at, and motivation for why dual optimisation to solve gradient shattering is needed:** Not all LALMs have discretisation of audio before input to an LLM (like SpeechGPT). Many insert a continuous vector directly into the embedding space of the model (e.g. DiVA, Llama3.1, Salmonn, Llasm, AudioQwen). Therefore, these won’t have the gradient shattering problem, and the framework in Figure 1 isn’t relevant.  There needs to be better motivation and explanation of why AdvWave targets LALMs that have the discrete bottleneck. Ideally, the paper will explain all the different architectures and introduce a framework that works for all the variants. Also, many LALMs do not have a decoder that maps back to audio space. Lots just do audio-to-text. Only a few models are fully speech-to-speech (some are trained end-to-end, and others just put a TTS module on end). It is important to talk about these. Furthermore, why can’t you use a straight-through estimator or Gumbel softmax to differentiate through the discretisation instead of the dual optimisation approach? I need more motivation to believe this is necessary.
    - Also, is gradient shattering a well-known term? A quick search gets something different: https://arxiv.org/abs/1702.08591. Perhaps the problem could just be called “Non differentiable audio tokenisation” or similar? I don’t think the dual optimization method is novel, it would be good to find the original paper that implements something like this. PErhaps it would be in the VQVAE literature?
- **Lack of threat model:** I’d like to see your threat model go into depth more about why you focus on white-box attacks and why you need stealthiness constraints. E.g., you can just apply existing white-box attacks to text LLMs already and get bad outputs; why do we care about LALM defense when text isn’t solved? Isn’t an attack that elicits a harmful jailbreak that isn’t “stealthy” also a success from the red team’s perspective? Why does it need to be understandable? These can be addressed in your threat model. Also, you mention in related work that LALMs shouldn’t be deployed widely if they are not robust, but releasing them as closed source is fine since you can’t attack with AdvWave.
- **Presentation of equations, figures, and results needs to be polished:**
    - Figure 1: Phase 1 would be nicer on the left. A brief intuition on what each loss is trying to achieve in the caption would be helpful
    - Section 3.2, in general, is very hard to follow along. L_retent is talked about lot before being explained. Include an intuitive explanation earlier. You introduce the notation for the size mappings of each component, but this makes it more confusing, in my opinion. I would put this in the appendix.
    - Section 3.5 - There is lots of repetition of equations here (e.g. equ 7 is the same as 5 and 6 similar to 1), it would be great if it could be folded into the other sections for conciseness
    - I’m not sure what the perk of having ASR-W is in addition to ASR-L. Often, LLMs are still jailbroken if they say, “I’m sorry,” so I’d expect ASR-W to have many false negatives. It would be good to manually check the false positive rate of ASR-L.
    - Figures 2 & 3 need axes labels and should use a color-blind friendly palette (without gradients). Figure 4 has text that is too small.
- **Related work is majorly lacking citations and doesn’t contrast with AdvWave:**
    - Add related work to white-box attacks on VLMs - your work is very comparable to how people jailbreak VLMs, e.g., https://yunqing-me.github.io/AttackVLM/ , https://arxiv.org/pdf/2306.13213, https://arxiv.org/pdf/2402.02309. Also, vocalising the request is similar to putting typographic text into images (like FigStep, Images are Achilles Heel of Alignment, Jailbreak in pieces)
    - Add related work to white-box attacks on STT models - this is also very relevant, especially the imperceivable constraints. e.g. “Audio adversarial examples: Targeted attacks on speech-to-text”, “There is more than one kind of robustness: Fooling whisper with adversarial examples”.
    - There are many more papers than I provide here, and I’d recommend doing a proper literature review.
    - LALM section - I would cut the section around concerns of misuse. This should be discussed in the intro. You should cite frontier models like Gemini and GPT-4o advanced voice mode.
    - Jailbreak attacks on LLMs section - you should cite https://arxiv.org/abs/2404.02151
- **Adaptive target search seems overly complicated:** why did optimising just for “sure” as the first token not work? This works in VLM literature. When comparing to optimizing for “sure”, did you use a prompt like in https://arxiv.org/abs/2404.02151? If not, optimizing for “sure” alone may be much weaker. I’d expect if you did this, the ASR would increase. Essentially, using an “adaptively search optimisation target,” you find a good starting point, but prompting the model to start the response with “Sure, here is…” might mean you don’t need this component. Also, why can’t you find a target string from another jailbroken LLM even if it has a very different structure to the output of the LALM? Shouldn’t gradient-based approaches still be able to change the model to output this?

questions:
Have you thought about measuring how your attacks transfer between models? I’d love to see transferability in your work since the threat model I think is most concerning is people finding white-box attacks on open-source models that transfer to more powerful closed-source models. See examples here: https://arxiv.org/abs/2403.09766 , https://arxiv.org/abs/2407.15211

Small discussion point on using LALMs. Most of the field uses VLMs for vision language models, so do you think using ALMs would be a better acronym to popularise in the field?

I have weaved most of my questions into the weaknesses section. I think this paper has the potential for a much higher rating (especially given the timeliness of getting attacks working on LALMs, which is a neglected area of the adversarial attack literature), but not in its current form. I am happy to increase my score if the weaknesses I highlighted are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624631042

thread_type:
official_review

<thread_object>
creator: reviewer_vsxc

<content>
summary:
The authors introduce a novel jailbreak framework for optimising audio jailbreaks against audio-language models (ALMs). They overcome challenges in the design of ALMs: namely 1.) they find a dual-phase training process so they can optimise attacks even through discretisation operations in the tokeniser, and 2.) they develop an adaptive search method to find more flexible adversarial targets.

Finally, the authors introduce a realistic constraint on their work: that the audio jailbreaks are stealthy. They operationalise this as having human and ALM-based classifiers independently score the audio input for signs that it was adversarially tampered-with. The authors claim (it's hard without hearing audio samples myself) that their jailbreaks are hence indistinguishable from normal urban noise (e.g. a car horn).

strengths:
This paper tackles a important and as yet unaddressed issue in jailbreak literature, and does so with sensitivity to realism. I am particularly impressed with the authors' operationalisation of stealthiness as urban noise (pending audio samples that I can listen to when the paper is out). The authors' use of human judgment to verify and counterweight their classifier (itself a potentially valuable contribution to audio-jailbreak defense) strengthens my confidence in these results even if I can't aurally verify them myself. 

The results of their optimisation are strong. Their ASR results are comparable to or exceed the other audio-jailbreak papers I know of that were submitted to ICLR.

The methods developed to optimise jailbreaks against audio-models, given the challenges the authors list, are valuable and novel contributions themselves. In particular the method for adversarial optimisation target search seems to me to strengthen the jailbreak method over the baselines they test against. For example, GCG is reasonably well-known for optimising for outputs such as "Sure," even if immediately followed with "I can't help with that." The adaptivity and greater detail of the jailbreak targets listed in the appendix seem to me to increase the likelihood that jailbreaks listed as successful in this paper do in fact contain jailbroken information. I'm also given increasing confidence in the evaluations of this paper by the authors' use of both a word-based classifier that detects refusal strings, and an LLM graded response.

weaknesses:
While I'm overall very positive on this paper, I'm a little underwhelmed by the baselines. I would expect that the adversarial perturbations of GCG and BEAST to be quite brittle to being converted to spoken text and then fed into an ALM. These are worthwhile baselines to run, but more semantically-natural baselines like AutoDAN would have pushed the paper even further. The authors acknowledge the difficulty and novelty of introducing audio-based adaptive attacks, like transfers of PAIR or TAP: I would have been very excited to see the authors tackle adaptive jailbreaks in the audio domain, but understand why for reasons of cost and difficulty that this might not be feasible - though I am aware of an unpublished audio implementation of PAIR.

I think Fig 1 is quite challenging to parse. I would rather it be simplified quite a lot more before final release. In particular, I think there is too much text annotating Phase II, even if helpful for diving deeper into the method. I would prefer at least a much more abstracted version of the figure, without reference to variables from Equation 1, and with the annotation retooled to explain how the different branches refer to each other. At the moment I think it's too hard to understand without continuous reference to Equation 1, and the figure struggles to explain itself on its own.

questions:
1. Did you try other audio-transcribed jailbreak classes, including more naturalistic text like in Zheng et al's persuasion paper? [1]
2. What made you think GCG and BEAST were strong baselines when translated into audio? 
3. Did you attempt your jailbreaks on any versions of Gemini or 4o? To my understanding some of the more capable models are only trained to recognise speech data - which would presumably make your noise perturbations less effective?
4. Who were the humans judging your stealthiness? was there a more detailed rubric you can share?




[1] https://arxiv.org/abs/2401.06373
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697159497

thread_type:
official_review

<thread_object>
creator: reviewer_eqtj

<content>
summary:
This paper presents a gradient-based jailbreak attack against Large Audio Language Models (LALM). The proposed method optimizes an adversarial audio suffix that bypasses the safety alignment of the LALM and causes it to produce harmful outputs. To account for the discretization performed to convert continuous audio representations into discrete tokens, a "dual-phase" optimization method is proposed whereby, first, the discrete token sequence is optimized to produce the desired harmful output and then the audio suffix is optimized to yield the discrete audio token sequence. Additionally, an adaptive search procedure is proposed to determine the best target for the adversarial loss optimization, and a loss component is introduced to make the adversarial suffix resemble a given environmental sound. Results show that compared to baselines the proposed approach greatly improves attack success rates.

strengths:
1. The paper is well written and generally clear
1. The proposed approach is novel and fills an important gap in the current literature.
1. The proposed attack is successful on diverse models which indicates its generalizability
1. Using an audio classification loss to make the adversarial suffix resemble natural sounds is an interesting and novel approach

weaknesses:
1. The -Trans baselines seem to weak because these attacks tend to introduce symbols, like special characters, punctuations and emojis, that are not vocalizable so it is expected that generating speech from them will produce weak results. I recommend presenting results for the text-only attack along with the -Trans attack. This way the actual advantage of exploiting the auditory modality will become apparent.
   1. A better baseline could be to adversarially attack an ASR model that uses the same audio encoder as the LALM such that the target transcription is the text-only attack string.

1. More details about the human evaluation score ($S_{\text{Human}}$) are needed, including the number of raters, inter-rater agreement, and did all raters rate all the test audios.
1. The normalization used for the stealth scores seems to be weight the components unfairly. The NSR and cosine are normalized by their theoretic maximum, while the human score is unnormalized so if the actual NSR and cosine scores occupy a smaller range then their contribution to the score will be penalized. A better normalization scheme might be to normalize the mean to 0.5 and standard deviation to 0.125.
1. The presentation can be improved:
   1. Phase II is to the left of Phase I in Figure 1. I suggest reorganizing it to make it appear to the right.
   1. The phrase "gradient shattering" or "shattered gradients" is confusing here because in prior work it refers to the specific phenomenon that as neural networks become deeper their gradients resemble white noise [1]. The particular phenomenon of relevance in this study is generally referred to as "gradient obfuscation" or "obfuscated gradients".
   1. The phrase "retention loss" is confusing because it is not clear what is being retained. The target discrete token sequence can not be "retained" because the encoder currently does not output it and it is being optimized to do so. Perhaps, "alignment loss" or "sequence loss" might be better.
   1. It is not clear from equation 2 that only a suffix is being optimized. It appears that the entire audio is being optimized.


[1] Balduzzi, David, et al. "The shattered gradients problem: If resnets are the answer, then what is the question?." International conference on machine learning. PMLR, 2017.

questions:
1. Why is noise-to-signal ratio used instead of the more common signal-to-noise ratio? Is it computed in a similar manner as SNR? The normalization and subtraction yields a quantity that is proportional to SNR so perhaps its simpler to just use SNR.
1. How exactly is $S_{\text{Mel-Sim}}$ computed? The mel spectrogram is a matrix so how exactly is the cosine similarity computed? 
   1. Why is cosine similarity used instead of L2 distance that is commonly used to compare mel spectrograms? I am not sure if the cosine similarity has a reasonable interpretation for mel spectrograms.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732515876173

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summary

comment:
We thank all reviewers for their valuable comments and feedback! We are glad that the reviewers found our work novel (the first white-box jailbreak attempt against LALMs) and sound with solid evaluation results. Based on the reviews, we have conducted additional experiments and made the following updates:

1. Expanded discussions on the importance of stealthiness, added supplementary examples, and clarified stealthiness evaluation methods.

2. Introduced the PAIR attack as a strong black-box baseline to emphasize AdvWave’s superior performance.

3. Enhanced explanations on gradient shattering and its implications for AdvWave optimization.

4. Clarified the rationale for appending adversarial suffixes instead of modifying original queries.

5. Improved presentation throughout the manuscript, including figures and equations.

6. Included additional related work on adversarial attacks for speech models and VLM jailbreaks.

7. Expanded discussions on the threat model, highlighting the relevance of white-box jailbreak scenarios.

8. Evaluated cross-model transferability of adversarial audio on Realtime API on the Advbench-Audio dataset.

9. Acknowledged the limitations in adversarial audio transferability and outlined future research directions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a jailbreak attack against Large Audio Language Models (LALM). The method uses a dual-phase optimization method that addresses the non-differentiable issue, enabling effective end-to-end gradient-based optimization. The method also enforces stealthiness constraints on adversarial audio waveforms to avoid being filtered by noise-detection systems. All the reviewers acknowledge the novelty of this work. Some reviewers question the need to maintain stealthy. In my opinion, the authors did a good job answering most of the concerns raised by the reviewers. Therefore, I recommend accept for this paper.