Diffusion Attacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

Authors: Hao Wang, Hao Li, Junda Zhu, Xinyuan Wang, Chengwei Pan, Minlie Huang, Lei Sha
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM safety; LLM jailbreak; Diffusion Language Model; Gumbel Softmax;

Abstract:
Large Language Models can generate harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. As LLMs become more powerful, studying jailbreaking becomes a critical aspect of enhancing security and human value alignment. Currently, jailbreak is usually implemented by adding suffixes or using prompt templates, which suffers from low attack diversity. Inspired by diffusion models, this paper introduces the DiffusionAttacker, an end-to-end generative method for jailbreak rewriting. Our approach employs a seq2seq text diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. This method preserves the semantic content of the original prompt while producing harmful content. Additionally, we leverage the Gumbel-Softmax technique to make the sampling process from the output distribution of the diffusion model differentiable, thereby eliminating the need for an iterative token search. Through extensive experiments on the Advbench and Harmbench, we show that DiffusionAttacker outperforms previous methods in various evaluation indicators including attack success rate (ASR), fluency, and diversity.

Decision: reject
Reviewer scores: 3, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730271135778

thread_type:
official_review

<thread_object>
creator: reviewer_iywc

<content>
summary:
This paper proposes to use diffusion model for rewriting jailbreak prompts in order to increase the attack success rates. It starts with a simple harmful prompts and makes use of denoising process to generate a jailbreaking prompt that is semantically similar to the original prompt. To guide this denoising process, it makes use of Gumbel noise and a sampling process that makes the generated prompts stochastic. The performance of this method is shown on different open-source models with metrics like ASR, Self-BLEU, Perplexity score, in addition to inference times. The cross-model transferability of this attack is also demonstrated.

strengths:
1. Good representation of visualization for harmful and harmless prompts for different models.
2. The authors have performed extensive experiments on open-source methods to support their hypothesis.
3. The authors have performed good ablation study of their method.

weaknesses:
1. The method section of the paper is not very well written and it is difficult to understand the connection between different blocks in the method section. Symbols used in problem formulation are not consistent, in line 201, it should be f(X). Also using the same symbols across sections 3.1(problem formulation) and 3.2 would have made it easier to understand the method section.
2. The consistency of symbols is not maintained across equations (2) and (3), making it difficult to understand, like how do the authors arrive at z from g(x). 
3. In figure 3, the first plot is same as that in figure 2. Also the caption of figure 3 is not consistent with the figure. Also there a few grammatical errors in the paper.
4. No detailed description (number of training and test set samples) of the final dataset used for training the model in Section 4.1. It would be helpful if the authors provide details of the dataset and any other relevant details.
5. An algorithm or a step-by-step pseudocode would have been good to understand the overall method more clearly. Also, how the method changes during inference is not explained in the paper.
6.  Comparison of this attack strategy against baselines, guardrails and existing state-of-the-art defenses is missing.

questions:
1. Please specify examples of generated jailbreaking prompts using your method
2. In equation (11), how are the parameters t, T and M selected?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730496992322

thread_type:
official_review

<thread_object>
creator: reviewer_7eyz

<content>
summary:
This paper introduces a novel jailbreaking attack that uses a text diffusion model to generate jailbreaking prompts automatically. The core insight is that benign and harmful prompts occupy distinct regions within the embedding distribution. Thus, the proposed method paraphrases prompts towards the benign distribution to evade alignment constraints. Experimental results show high attack success rates, perplexity, and diversity in generated prompts.

strengths:
(1) Interesting idea to use text diffusion models for jailbreaking.

(2) Well-written and easy to follow.

(3) Effective evaluation results supporting the proposed method.

weaknesses:
(1) No guarantee that harmful responses directly address the intended harmful questions.

(2) Limited discussion on potential defenses against the attack.

(3) No evaluation of attack transferability to production models.

(4) Lack of concrete examples showing the quality of generated prompts.

questions:
This paper proposes a novel idea to leverage text diffusion model to automatically generate jailbreaking prompts. The idea is interesting but I have several concerns:

(1) There’s no guarantee that harmful responses will actually match the specific harmful intent of the question. While the prompts show high ASR, perplexity, and diversity, they may produce general harmful responses that don’t directly address the original question, which could limit the method’s practical value.

(2) The paper lacks discussion of the potential defense. There are several defense methods[1][2] against jailbreaking. It is suggested to discuss the potential defense in the paper. For instance, SmoothLLM[1] randomly perturbs the input to the model to prevent the potential strong adversarial tokens from being used for inference.

(3) The transferable evaluation is limited to open-source models, with no tests on large, production models like GPT-4 or Claude-3.5. Demonstrating the attack’s transferability to these models would improve the paper by showing that the approach is applicable in real-world settings.

(4) The paper lacks concrete examples of generated prompts to illustrate their quality, relevance, and effectiveness. Including specific examples would give readers a clearer picture of how well the model’s output aligns with the proposed objectives.

**Reference:**

[1] Robey, Alexander, et al. "Smoothllm: Defending large language models against jailbreaking attacks." arXiv preprint arXiv:2310.03684 (2023).

[2] Cao, Bochuan, et al. "Defending against alignment-breaking attacks via robustly aligned llm." arXiv preprint arXiv:2309.14348 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533668977

thread_type:
official_review

<thread_object>
creator: reviewer_f8tc

<content>
summary:
This paper proposes a jailbreak attack method, DiffusionAttacker, against large language models. It designs a general attack loss for jailbreak attacks derived from the analysis of the internal hidden states of LLMs. Additionally, they use a sequence-to-sequence diffusion model to generate the rephrase of harmful prompts to make the hidden states of LLMs close to the hidden states of benign prompts, which can make the LLM output answers to the rephrased harmful prompts and jailbreak successfully. They control the diffusion process by updating the intermediate states in the sampling process using the gradients back-propagated from the general attack loss. Various experiments are conducted to validate the effectiveness of their method in comparison to other baseline methods. Ablations studies and transfer attacks are also included in their evaluation.

strengths:
1.	The novel design of the general attack loss provides valuable insights in understanding the internal mechanism of jailbreak attacks. 
2.	The usage of conditional generation with seq-to-seq diffusion model is novel.
3.	Evaluation results shows both good attack performance in white-box and transferring attack settings.

weaknesses:
1.	Authors conduct dimension reduction before the classifier of harmful or not harmful, but didn’t discuss the reasons behind or conduct ablation studies to validate its effectiveness. Also, does the selection of different dimension reduction methods have impact on the attack performance?
2.	It is not clear about the motivation or advantage to use a diffusion model as the language generator to rephrase the harmful prompts. Why not use another LLM to rephrase? 
3.	The presentation needs improvement. It is better to refer to Figure 1 while you are explaining specific parts of your method, and some fonts in Figures are too small to read (e.g., Figure 2 and 3)
4.	No qualitative examples are provided in the evaluations. Authors may show some examples of the original and rephrased harmful prompts and the difference in LLM output to make their results more convincing.

questions:
Same to those in Weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659429985

thread_type:
official_review

<thread_object>
creator: reviewer_mrhd

<content>
summary:
In this paper, the authors proposed DiffusionAttacker as a while-box LLM jailbreaking method, which attacks a LLM by attacking a harmfulness probing classifier trained on its last layer hidden states instead, based on the observation that the jailbreaking prompts produced by some existing jailbreaking methods are often no longer sparable from non-harmful inputs by the classifier. In specific, DiffusionAttacker employs DiffusionSeq models to rewrite malicious requests through denoising to 1) preserve the semantic content, while 2) cheating the probing classifier by end-to-end training (with the help of Gumbel SoftMax for differentiability) with the victim LLM (frozen). DiffusionAttacker is tested on the AdvBench dataset against Llama 3 8b, Mistral 7b, Alpaca 7b (and Vicuna 1.5 7b) models and showed improvement in ASR (by prefix- and GPT-judge), PPL and self-bleu over existing attacks including GCG, AutoDan and ColdAttack. The transfer study showed that jailbreaking prompts generated by DiffusionAttacker are also more likely to jailbreak other models. The ablation study further investigated the impact of each component of DiffusionAttack.

strengths:
The ablation study design is both interesting and essential to clearing a number of doubts about this paper, although the results are not exactly convincing. The doubts include: 1) Why not directly attack the probing classifier? 2) Why use DiffusionSeq?, etc.

weaknesses:
1.  Inadequate Baseline Selection: While the likes of GCG and AutoDAN are of clear resemblance with DiffusionAttacker method-wise (white-box, proxy-target, guidance from gradient), they are not among the most powerful jailbreaking methods. A number of black-box attacks like [1] and [2] can also jailbreak the white-box models while being more fluent, efficient and effective.
2. Questionable Metric: ASR computed by prefix match is outdated and knowingly unable to reflect the true jailbreaking effectiveness. The GPT-judge used is also not a widely-used or "strong" one, in that it is only employing GPT-4o to decide the harmfulness without detailed criteria, judgement on a scale instead of binary, or consideration of content relatedness to the malicious request, etc. as identified by e.g. [2] and [3] as crucial properties for a GPT-judge to evaluate jailbreaking attacks more accurately. There are not adequate reasons to incorporate PPL and Self-Bleu scores as the major metrics. Why is fluency of the jailbreaking prompt an important characteristic to have? There does exist defensive mechanisms that uses PPL to filter suspectable inputs, but they haven't been widely used ever. For instance, in the interactive with LLMs, rare token sequences have good reasons to appear, e.g. ASCII arts, ciphertexts, etc. Bleu-based diversity is also not a guarantee that a jailbreaking method is hard to defend against.
3. Non-prominent Performance: The ASRs (GPT-based), while higher than the few baselines, are not actually high from today's point of view given that a number of black-box attacks have surpassed them. Many of them also make use of the idea about turning the prompt less harmful by the look to generate jailbreaking prompts. In a sense, DiffusionAttacker is only resorting to the internal hidden states of the victim LLMs which black-box attacks refrain from to do a similar thing while not receiving any benefit in terms of fluency, effectiveness or efficiency. The transfer attack ASRs are also very low to be of practical use.
4. Questionable Design Choices: Many design choices in the paper are not well-justified, e.g. 1) Why choosing the last-layer hidden states for the classifier, when a number of works has identified middle layers as more effective at probing? 2) Why using DiffusionSeq when there are alternatives like encoder-decoder transfomers and LSTMs? What is special about diffusion models that make it fit for this task? 3) 
5. Inadequate Experiments: Being an attack that stems from representation engineering [4], clearly [5] is a better baseline as it is finetuned specifically to direct harmful representations to benign ones. A number of jailbreaking methods have failed at attack it while exile otherwise, so if DiffusionAttack manages to break [5] due to its similar concepts, then the relatively inadequate performance might be justified a bit as it at least finds some unique situations where it is the only successful attack. Additionally, even in the transfer experiment, only white-box models are used. It is important to see how the prompts generated by DiffusionAttack attacks GPTs, Geminis and Claudes.
6. Poor Demonstration: The paper provides no examples of the jailbreaking outcomes. There is also no graph to illustrate the change in classifier decision after the attack to validate the hypothesize which motivates this paper.
7. Questionable Ablation: The advantage of DiffusionAttack over the other design choices is not significant except for fluency which is obviously going to be the case.

[1] Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues
[2] WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response.
[3] A StrongREJECT for Empty Jailbreaks
[4] Representation Engineering: A Top-Down Approach to AI Transparency
[5] Improving Alignment and Robustness with Circuit Breakers

questions:
My questions and suggestions are mostly covers in the weaknesses outlined above.
</content>
</thread_object>

</thread>


Metareview: