Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning

Authors: Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, Moksh Jain
Primary area: foundation or frontier models, including LLMs
Keywords: red-teaming, LLM, diversity

Abstract:
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate *diverse* and *effective* attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730113684937

thread_type:
official_review

<thread_object>
creator: reviewer_ydlu

<content>
summary:
This paper introduces GFlowNet fine-tuning plus a follow-up smoothing phase for attack generation in LLM red teaming. Their approach overcomes the typical problems of lacking diversity, effectiveness and model collapse that arise in RL-based automated red teaming. The authors not only show the effectiveness of their method for red teaming (toxicity focus) but also that their method can be used to generate highly effective fine-tuning data for safety tuning.

strengths:
The method is well motivated, presented in an easy to understand way, and backed by a rigorous experimental setup. Especially the performance in the transfer setting is impressive, as most other methods completely fail at this task. This paper advances the state of the art in a significant way and addresses a crucial problem in AI security.

weaknesses:
Experiments on the performance against LLM guardrails would have been of interest, as real-life deployments will always include input and output guardrail models. Given the strong performance of the method in transfer settings, this could also prove to be another potential strongpoint of this method.

questions:
Could you please elaborate on how you expect your method to fare against current SOTA guardrails and the challenges you see in overcoming those with attacks using your method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387357332

thread_type:
official_review

<thread_object>
creator: reviewer_puvq

<content>
summary:
The paper studies methods of training a language model to generate adversarial prompts that, when provided to more standard language models (such as GPT, Gemma, or Llama), produce responses deemed to be violating by a safety classifier. The main contribution of the paper is to apply the GFlowNet reinforcement learning method for this fine tuning of the attacker model. The paper produces attacker models that generate prompts of better diversity and higher attack success rate than other methods that take the same approach to red teaming.

strengths:
On a standardized dataset and with standardized evaluations, and within the class of reinforcement learning methods that train an attacker model to produce a single adversarial prompt, this paper proposes a method that does achieve better diversity and stronger attack success rate. It is important for the community to be aware that this reinforcement learning method can produce attacker models with stronger adversarial “power” and diversity. The discussion of the adjustments needed to the GFlowNet method is also important.

weaknesses:
I would say that this paper suffers from weaknesses that the whole field of automated jailbreaking is currently subject to. I do not consider this to be a reason for rejection but I also think it is important to record this critique for future iterations of automated red teaming work.
To be more precise, it is not clear to me that the majority of methods in this field – this work included – discover ways to elicit meaningfully harmful responses from models. It appears to me that the responses provided in the Appendix are all generally informational. To put this another way, this paper – along with most works who take AdvBench off the shelf as the goal set and slap some classifier on the responses – sidestep a robust definition of what is and is not considered harmful. This is not necessarily a problem of the work itself and rather a shortcoming of the yardsticks (harmful prompt datasets and classifiers) that have been widely adopted. 

However, what is important for the authors of this work, is that the end result of this leads to methods that generate adversarial prompts that likely exploit the model’s willingness to be generically helpful. In particular, the prompts listed in tables B.5 and B.6 are themselves very generic. For example, it is not clear why or how “records in the financial database” are being manipulated. Is this necessarily some harmful or unauthorized manipulation? The model response itself assumes that you would be doing this with a properly authorized user. This is likely because the prompt leaves enough vagueness in it to be interpreted as asking for something that is perfectly legal and acceptable (help with interacting with a database). 

Thus, I believe methods in this space in the future should also consider specificity of the harmful request as another axis of desirable properties, in addition to diversity and attack success rate judged by a classifier. So instead of ending up with prompts that dance around a vaguely specified line, methods should a) be explicit about the line and b) make sure that their attacks clearly cross it. It would be interesting if GFlowNet adversarial methods can help elicit specific and truly harmful responses from language models.

questions:
On linea 305-309, it is not clear to me what dataset is being used. Can you be more explicit if you are adopting the datasets for all methods listed on these lines? Or do you use a different dataset for your fine tuning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664092167

thread_type:
official_review

<thread_object>
creator: reviewer_3l4u

<content>
summary:
The paper proposes a two-stage approach combining GFlowNet fine-tuning with MLE smoothing to automatically generate diverse and effective adversarial prompts for testing language model safety, demonstrating good performance over a selection of baseline red teaming methods, balancing attack success rate and prompt diversity while enabling reasonable transfer to new target models.

strengths:
1. The proposed method could sample diverse prompts and appear more efficient than the baseline methods. 
2. The main claims are well-supported by the experiments.
3. The writing is clear and the paper is easy to follow.

weaknesses:
1. There's a lack of evaluation against stronger attacks and defenses. The paper did not consider many of the non-optimization-based gray /black box attacks, which might perform better at lower computation budget. The paper also did not consider robustified versions of the models such as models that underwent circuit breaking or (latent) adversarial training.
2. It's unclear how to adapt this for targeted attack against specific harmful requests and how well that would work.

questions:
I'm curious to know whether the method scales with stronger base model for the attacker model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690287734

thread_type:
official_review

<thread_object>
creator: reviewer_uqkl

<content>
summary:
The paper proposes an automatic two step red-teaming method for LLMs that aims at generating prompts that are very diverse as well as effective. Diversity is important to transfer attacks across different models and for generalization purposes. The first step of the red-teaming process consists of collecting several prompts that score a high reward using GFlowNets algorithms. In these methods, the learned policy sample prompts with probability proportional to the reward the prompts received. The reward is defined as the likelihood that a prompt is classified as toxic by a toxicity classifier (plus a KL term). The second step takes into account some issues related to tuning some hyperparameters in the reward function. In particular, the prompts collected at the previous step are filtered and used to fine-tune the initial policy to maximize model log-likelihood. The red-teaming is tested against different baselines and across different models. The results show that prompts generated with this method are more diverse and effective compared to the ones produced by other baselines and transfer across different models.

strengths:
- Attack diversity is important in red-teaming and finding automatic methods to produce diverse and effective prompts is valuable.
- The second step seems to significantly improve over the baseline consisting only of the first step.
- The method is compared against many different baselines.

weaknesses:
- The tested models are quite small. Even if the attacks generalize to different models, it’d be good to have an evaluation of bigger and closed-source models (since the technique doesn’t require access to model weights).
- Although inexpensive, it seems not very efficient to retrain the policy on a subset of generated prompts. For example, filtering the prompt can be done on the fly during the GFlowNet stage.

questions:
- Why do you use two different toxicity classifiers for different models?
- The difference in performance between GFlowNets and GFlowNets+MLE changes quite a bit across different models. For example, for Dolly and GPT-2 the performance is almost the same, for Gemma there is a huge gap, for the Llamas the difference is a bit smaller. Do you have a sense of why this is happening?
- The attacks produced with this method are very diverse. Is this diversity mainly semantic or also stylistic?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730840285949

thread_type:
official_review

<thread_object>
creator: reviewer_nedw

<content>
summary:
This paper applies GFlowNet to the problem of automated red-teaming, achieving a favorable balance between attack success rate and attack diversity. A model is trained to sample attacks with probability proportionally to their reward, followed by a cheaper second stage of training a smoother version of the model on rollouts of the trained model. The method achieves high levels of diversity combined with strong attack success rates, and, when including the smoothing step, consistently strong results across different hyper-parameters and different target models.

strengths:
- Clearly strong results and exciting possibilities for improving automated red-teaming and finding diverse attacks with strong motivations as one of the core challenges.
- The paper is well written and easy to follow. Experiments and ablations are documented well and replication of the results seems straightforward.
- I appreciate comparing to multiple RL-based baselines, as well as red-teaming multiple models. This gives confidence that the results will hold up in a wider range of settings.

weaknesses:
- Given the focus on learning diverse attacks, there could have been more in-depth ablations and experiments to investigate which parts of the method most strongly influence diversity (also see Questions section).
- It would be nice to also plot error bars, at least on a version in the appendix. It was not clear to me if the various plots are also based on multiple random seeds (as table 2 is).

questions:
- Where does the diversity come from (besides the ablation on $\beta$ values)? $\gamma$? sampling temperature $\tau$? replay buffer / "off-policy-ness"? The entropy/diversity of $p_{ref}$? The mix between replay buffer and online sampling in each iteration? Given that this is a main focus of the paper I would have been excited about more ablations / investigations in this direction.
- The results for REINFORCE, ICL, SFT seem mostly consistent across the different red-teamed models. PPO+Novelty results are less consistent and so is GFlowNet - is this due to hyper-parameter sensitivity? Or variance between runs? GFlowNet+MLE looks more consistently strong, so the core results of the paper are not impacted by this.
- How strong was the toxicity classifier, how often did the method "hack" shortcomings of the classifier rather than finding actual toxic outputs? This was hinted at in the discussion of setting $\beta$ too low (high weight of R1), but wondering if there are some more concrete results on this?
- What are sensible values for $r1, r2$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732147819954

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We express our sincere gratitude to all the reviewers for their constructive comments and feedback and generally positive reception of our work. 

We particularly appreciate their recognition of the **strong experimental results** (nedW, 3L4U, pUvQ, UQkL), **strong motivations** (nedW, YDLU), and **high quality of writing** (nedW, 3L4U, YDLU).


**Additional experimental results** 
We would like to highlight some additional experiments we performed based on suggestions by the reviewers.

- **Transfer to GPT-4o**: Based on a suggestion by Reviewer UQkL, we transfer the attack prompts, which originally targeted the Llama-2-7b-chat model, to GPT-4o and evaluate how many of them can elicit harmful responses from GPT-4o. In average of 5 different runs, **65% of 1,024 prompts can successfully attack GPT-4o**. 

- **Attackers with larger models**: Suggested by Reviewer 3L4U, we train the attacker model using Llama-3.2-1B-Instruct to red-team the Llama-3.1-8B-instrcut model. As shown in the table below, the larger model achieves a larger number of toxic prompts than smaller model GPT2-Small. 


| Base Model   | Success rate (%) |  Cosine Distance |
|--------------|:--------------------------:|:----------------:|
| GPT2-Small   |       $81.05\pm 0.96$      | $0.829\pm 0.001$ |
| Llama-3.2-1B |       $\textbf{92.71}\pm 1.12$      |  $0.833\pm0.002$ |


We have responded to all the individual comments from the reviewers below. Please let us know if you have any further questions or suggestions!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper applies GFlowNet to the problem of automated red-teaming, reaching a favorable balance between attack success rate and attack diversity. The authors are encouraged to include scalable evaluation and thorough results analysis in the final version.