AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

Authors: Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, Yuandong Tian
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: adversarial attacks, prompt optimization, red-teaming LLMs

Abstract:
While recently Large Language Models (LLMs) have achieved remarkable successes, they are vulnerable to certain `jailbreaking attacks` that lead to generation of inappropriate or harmful content. Manual red-teaming requires finding adversarial prompts that cause such jailbreaking, e.g. by appending a suffix to a given instruction, which is inefficient and time-consuming. 
On the other hand, automatic adversarial prompt generation often leads to semantically meaningless attacks that can easily be detected by perplexity-based filters, may require gradient information from the TargetLLM, or do not scale well due to time-consuming discrete optimization processes over the token space. In this paper, we present a novel method that uses another LLM, called the `AdvPrompter`, to generate human-readable adversarial prompts in seconds, $\sim800\times$ faster than existing optimization-based approaches.
We train the AdvPrompter using a novel algorithm that `does not require gradients` of the TargetLLM. This process alternates between two steps: (1) generating high-quality target adversarial suffixes by optimizing the AdvPrompter predictions, and (2) fine-tuning of the AdvPrompter with the generated adversarial suffixes. The trained AdvPrompter generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show state-of-the-art results on the AdvBench dataset, that also transfer to closed-source black-box LLM APIs. Further, we demonstrate that by fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made more robust against jailbreaking attacks while maintaining performance, i.e. high MMLU scores.

Decision: reject
Reviewer scores: 5, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1729543151624

thread_type:
official_review

<thread_object>
creator: reviewer_qn1h

<content>
summary:
The paper titled "AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs" presents a novel approach to generating adversarial prompts that jailbreak large language models (LLMs), enabling the generation of harmful or inappropriate content. Traditional methods for adversarial prompt generation, such as manual red-teaming or optimization-based methods, can be slow, inefficient, and prone to generating semantically meaningless attacks. In contrast, the authors propose AdvPrompter, an LLM trained using a novel algorithm to rapidly generate human-readable adversarial prompts without requiring gradient information from the target LLM.

The core innovation of the paper lies in its alternating training method, AdvPrompterTrain, which alternates between generating adversarial suffixes and fine-tuning the AdvPrompter model. The resulting adversarial prompts are highly effective, achieving state-of-the-art results on the AdvBench and HarmBench datasets, with improved attack success rates, faster generation times, and strong transferability to black-box LLMs. Additionally, the paper demonstrates that by fine-tuning LLMs on datasets generated by AdvPrompter, models can become more robust against jailbreaking attacks while maintaining high performance on benchmarks like MMLU and MT-bench.

strengths:
1. **Clarity**: The paper is well-structured and provides clear explanations of its methodology, backed by comprehensive experimental results and ablation studies. The use of figures and tables, such as Table 1, aids in understanding the comparative advantages of the method. The training algorithm and attack framework are concisely explained, which improves accessibility.

2. **Significance**: AdvPrompter presents significant contributions to the area of LLM robustness and safety, offering a highly scalable solution to automatic red-teaming. Its gradient-free approach makes it applicable in both white-box and black-box settings, which broadens its impact for securing deployed LLM systems. The paper’s findings on fine-tuning models with adversarial data to improve safety also open up new avenues for automated adversarial training in LLMs.

weaknesses:
1. **Lack of Comparison with BEAST**: One notable omission is the lack of detailed comparison with BEAST (introduced in "Fast Adversarial Attacks on Language Models in One GPU Minute")​. BEAST also focuses on gradient-free attacks and is highly efficient, achieving impressive success rates within one GPU minute, making it an essential baseline. The authors of AdvPrompter reference BEAST, but they fail to provide head-to-head benchmarks, especially in terms of speed and success rates. This limits the ability to assess whether AdvPrompter's claim of "fast" generation holds up against a method already proven to be both rapid and effective.

2. "Unclear Computational Efficiency": While AdvPrompter claims faster generation of adversarial prompts compared to gradient-based methods, the paper does not include detailed benchmarks or profiling to demonstrate computational efficiency on a per-prompt basis. For instance, BEAST reports precise GPU utilization metrics and compares the attack time per prompt across different models. AdvPrompter lacks such concrete data, making its claims of speed improvements less convincing. Without these comparisons, it is unclear whether the method is truly fast or simply optimized for a limited set of tasks.

Incomplete Discussion of Attack Readability: While AdvPrompter claims to generate human-readable adversarial prompts, there is limited qualitative analysis of the readability or coherence of these prompts. I would like to see more analysis on this.

questions:
I encourage the authors to address the points raised in the weaknesses section and to conduct additional experiments where further investigation is required.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712940642

thread_type:
official_review

<thread_object>
creator: reviewer_qcah

<content>
summary:
This paper proposes a novel method to enhance jailbreaking attacks on safety-aligned large language models (LLMs). The proposed method involves constructing a framework that fine-tunes an LLM from a base model by encouraging it to generate human-readable adversarial suffixes for harmful requests. Extensive experimental results demonstrate that the AdvPrompter can produce low-perplexity adversarial suffixes and achieve performance comparable to two baseline methods, i.e., GCG and AutoDAN.

strengths:
* The proposed method enables the fast generation of specific adversarial suffixes for individual harmful requests.
* The experimental results show that the proposed method achieves great performance.

weaknesses:
Overall, I think the method is sound, but there are a few concerns.

* The advantages of AdvPrompter mentioned in lines 108-136 should be specified under certain comparative conditions. For example, the "adaptivity to input" should be highlighted in the context of generating at a low time cost, as both GCG-individual and AutoDAN-individual are also adaptive to input. The "fast generation" compared to GCG and AutoDAN should be specified in the context of generating individual adversarial suffixes, since GCG-universal and AutoDAN-universal are ready to be used once obtained.

* Since both AutoDAN-universal and AdvPrompter generate human-readable adversarial suffixes quickly, it would be beneficial to discuss their performance in more detail, especially in Table 3.

* I think the "Gradient-free TargetLLM" is not a significant advantage, and it is unnecessary to emphasize the "gray-box TargetLLM" since it is actually a "white-box" model.

* More comparisons to existing methods, such as TAP and PAP, should be included. These methods also generate human-readable adversarial prompts.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742457340

thread_type:
official_review

<thread_object>
creator: reviewer_8khr

<content>
summary:
The paper introduces a new method, and potentially a new perspective, for jailbreak prompting, the main contribution is that the prompter is itself a trained model, thus it can generate the jailbreak prompts extremely efficiently. The authors also mentioned several other properties such as human-readable or gradient-free, but these properties have been well discussed before.

strengths:
1. The idea of using a pretrained model to directly generate jailbreak as next token prediction is interesting. 

2. The propose method can generate jailbreak much faster than existing methods, especially since most existing methods are optimizing for every sample.

weaknesses:
1. the idea of training a generation model for jailbreak has many natural limitations, such as the evaluation will require a different split of data, need to validating that the model can generalize across different target LLMs, and also different benchmark datasets. The authors didn't fully address these. 

     -. 1.1 For example, since the method does not require any gradient of the target LLMs, the authors need to report more results on the commerlized LLMs where the strengths might be more obvious (referring to Table 2). 

    -. 1.2. The evaluation is only limited to AdvBench, the empirical scope is too small. Some other choices are like HarmBench or JAMBench. 

    -. 1.3. The evaluation needs to demonstrate the power of the advPrompter while the model is trained on AdvBench, and tested on other benchmarks such as HarmBench or JamBench. This is very important to show the advantages of proposed methods over the per-sample optimization method. 

    -. 1.4. Similarly, the authors might want to offer more detailed and comprehensive discussions on the differences between the targetLLM during training vs. during testing, although a gentle discussion has been offered in 4.2. 

2. The empirical scope is also fairly limited in terms of the methods being compared. Newer methods in jailbreaks, even just the ones published and presented in recent conferences (excluding arXiv ones), need to be discussed and compared. There are more methods that can deliver human-readable jailbreaks. (Although another method that can simultaneously fulfill all the properties in Table 1 might not exist). 

3. While the authors present a unique method, and might be the only one at this moment can achieve all the properties in Table 1, the performance is unfortunately achieved by trade-offs. For example, in Table 2, the proposed method is not necessarily always the best performing method in ASR. The perplexity is always the lowest, but comparison to newer method might be needed, e.g., [1]. This is important because in AI security research, ASR and perplexity are probably more important factors than generation time. Authors might need to offer more convincing discussions why the method is favored although being lower in ASR in certain cases. 


[1]. Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of LLMs

questions:
1. Does the method require gradient during training? (i.e., does the method have to be trained with white-box LLM?) if that's the case, then that is another point needs to be made clear. If not, results showing how the trained advprompter from highly aligned models such as GPT then applied to less aligned models will be interesting. 

2. Training time and requirement of computing might also need to be discussed, although less important. 

3. The authors might need to compare their results in Sec. 4.3 with other jailbreak defensive methods for LLMs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730872939436

thread_type:
official_review

<thread_object>
creator: reviewer_efra

<content>
summary:
This paper studies how efficiently generate transferrable and interpretable surfix for jailbreak. Unlike previous white-box attack methods that adopt search-based optimization, the authors propose a learning-based method, i.e. finetuning a LLM to generate adversarial prompts using annotated harmful QA data. A major benefit of this approach is inference-time efficiency. To train the LLM, the authors propose an alternated optimization by first searching for the best surfix that prompts the target LLM to answer harmful queries, and subsequently use it to finetune the Prompter LLM. Experiments are mainly conducted by comparing with some white-box attacks, and both direct search and transfer settings are considered. The result show that a major inference efficiency boost, with mixed results in terms of ASR. The proposed method also exhibits stronger transferability to close-source proprietary models than baselines.

strengths:
1. Exploring the leanring-based paradigm for generating adversarial prompts is novel and relevant, to the best of the reviewer’s knowledge.
2. Compared with search-based paradigm, methods along the learning-based paradigm, like this one, naturally enjoy the benefit of inference-time efficiency.
3. Experiment results suggest that the proposed method surpasses previous method in terms of transferrability to black-box models, which is arguably a more practical scenario than white-box attack.

weaknesses:
1. [major] It seems that, intuitively, the solution of equation 1 depends on the targetLLM. i.e. the optimal surfix that triggers a LLM to output the target response (e.g. “Sure, here are detailed xxx”) might be different. I’d imagine that this would hurt the transferrability in theory. It does appear that the transferability of AdvPrompter is at least better than early white-box attackers, but it might be due to poor transferability of white-box attackers in the first place.
2. [major] Following 1, I have some doubt about the practicality of jailbreak methods that requires transferrability in general. I would suggest comparing with SOTA blackbox methods on Figure 2. While I acknowledge that it is debatable whether such comparison is academically fair, the general practice usually guides us towards using whichever that is most effective. But I am happy to hear the author’s rebuttal and take them into consideration.
3. [minor] Learning-based paradigm, compared with search-based ones, incurs high training cost.

questions:
1. Necessity of alternated update: Is it necessary to put the AdvPrompter in the loop of suffix generation (Figure 1 bottom right)? My understanding is that, the purpose of including it is to generate topk candidate tokens for the suffix. I am curious whether the author has tried to use a separate LLM to do this? The upside is that the dataset used to finetune the AdvPrompter can generated offline (without alternatively updating AdvPrompter); The downside is that the generated most likely tokens will not be adaptive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732149486901

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Checking in after Rebuttal

comment:
Dear Reviewers,

We hope our rebuttal has addressed your concerns and enhanced the quality of our paper. Your feedback has been crucial in improving our work. If the changes meet the paper's objectives and your concerns, we hope this could be reflected in an improved score.

Please let us know if you have further questions or need additional information to aid your review. Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel method to enhance jailbreaking attacks on safety-aligned large language models (LLMs). The proposed approach involves developing a framework to fine-tune an LLM from a base model, encouraging it to generate human-readable adversarial suffixes for harmful requests. Extensive experimental results demonstrate that the method, named AdvPrompter, produces low-perplexity adversarial suffixes and achieves performance comparable to two baseline methods: GCG and AutoDAN.  

The paper’s primary innovation lies in its alternating training method, AdvPrompterTrain, which alternates between generating adversarial suffixes and fine-tuning the AdvPrompter model. This process results in highly effective adversarial prompts, achieving state-of-the-art performance on the AdvBench and HarmBench datasets. AdvPrompter demonstrates improved attack success rates, faster generation times, and strong transferability to black-box LLMs. Furthermore, the paper shows that fine-tuning LLMs on datasets generated by AdvPrompter enhances their robustness against jailbreaking attacks while maintaining high performance on benchmarks like MMLU and MT-Bench.  

AdvPrompter makes notable contributions to the field of LLM robustness and safety, offering a scalable and gradient-free solution to automatic red-teaming in both white-box and black-box settings, which broadens its applicability for securing deployed LLM systems.  

However, there are significant shortcomings. One major omission is the lack of a detailed comparison with BEAST, a method introduced in *"Fast Adversarial Attacks on Language Models in One GPU Minute."* BEAST also employs a gradient-free approach, achieving high success rates with exceptional speed. Although the authors reference BEAST, they fail to provide head-to-head benchmarks, particularly regarding speed and success rates. This omission makes it difficult to verify AdvPrompter's claims of "fast" generation relative to an already proven rapid method.  

Additionally, the empirical scope of the comparisons is limited. The authors do not sufficiently engage with newer jailbreak methods, including those presented at recent conferences. Many of these methods deliver human-readable jailbreaks and warrant a more thorough discussion and comparison. Reviewers also suggested comparing AdvPrompter's results with existing defensive techniques for LLMs, a point that remains underexplored.  

While the authors addressed some concerns raised during the review process, several comments remain unresolved. The paper would benefit from an additional round of revision, including more comprehensive evaluations, detailed comparisons, and expanded discussions of related methods.