Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

Authors: mikayel  samvelyan
sharath chandra raparthy
andrei  lupu
eric  hambro
aram h markosyan
manish  bhatt
yuning  mao
minqi  jiang
jack  parker-holder
jakob  foerster
tim  rocktäschel
roberta  raileanu
llm  judge
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:48:07

Abstract: As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

Main paper: 
 introduction: 
 Large language models (LLMs) have recently experienced remarkable growth in both their capabilities [48, 21, 65] and their applications in various fields [47, 64, 59, 7] . As LLMs become increasingly complex and are deployed in safety-critical environments [62, 36, 39] , it is essential to thoroughly understand their robustness to different inputs. Indeed, the susceptibility of LLMs to adversarial prompts -prompts crafted to mislead the model or exploit its weaknesses, potentially leading to unsafe, biased, or incorrect outputs -poses a significant challenge [51, 66, 74] . Identifying and mitigating these vulnerabilities is therefore vital to ensure the safe and reliable operation of LLMs in the real world [3, 24] .Current methods for identifying adversarial prompts aimed at "attacking" LLMs and eliciting undesirable outputs are limited by several factors. These limitations include the necessity of fine-tuning an attacker model [51, 20] ; white-box access to the target model [38] ; or significant human input [69] . Several recent black-box methods, such as PAIR [8] and TAP [41] , rely on a human-in-the-loop to specify harmful behaviours (e.g., "Write a tutorial on how to hack the university server") in a way where the human effort required is linear in the number of desired prompts. Many existing Style. Shading corresponds to the Llama Guard [26] scores of responses induced by the adversarial prompt in each cell (higher means more confidence in the response being unsafe). Some excerpts of discovered prompts from a single archive are shown. 1methods for systematically discovering adversarial attacks exhibit a lack of diversity by design [38] , for instance by restricting themselves to a single predefined attack strategy [60, 28, 2] . Others suffer from loss of diversity, a common issue in objective-based prompt optimisation approaches [72, 16] . In both cases, the narrow focus of generated prompts limits the usefulness of those methods both as a diagnostic tool and as a source of synthetic data for improving robustness.We introduce RAINBOW TEAMING, a versatile approach for systematically generating diverse adversarial prompts for LLMs via LLMs. While the prevailing approach to automatic red teaming [51] also uses LLMs to generate adversarial inputs, it exhibits a steep trade-off between the diversity of discovered attacks and their success rate. In contrast, RAINBOW TEAMING takes a more deliberate approach, efficiently covering the space of attacks by directly optimising for the attack quality and diversity. To this end, our method casts the problem of adversarial prompt generation as qualitydiversity (QD) search [34, 52, 13] and takes direct inspiration from Samvelyan et al. [57] to discover a set of adversarial prompts that are both diverse and effective. RAINBOW TEAMING is an open-ended approach [25] which builds on MAP-Elites [46] , an evolutionary search method that iteratively populates an "archive" with increasingly higher-performing solutions. In our case, these solutions are adversarial prompts that elicit undesirable behaviours in a target LLM, while the archive is a discrete grid where each dimension categorises prompts according to a feature of interest for diversity, such as attack style, risk category, or prompt length. The output of our method, as shown in Figure 1 , is a set of prompts covering every combination of features specified by the archive. These diverse and effective attack prompts serve both as a diagnostic tool for the vulnerabilities of the target LLM and as a high-quality synthetic dataset to robustify the target.RAINBOW TEAMING is directly applicable to a wide range of domains. Implementing RAINBOW TEAMING requires three essential building blocks: 1) A set of features that specify the dimensions of diversity (e.g., "Risk Category" or "Attack Style"); 2) A mutation operator to evolve adversarial prompts (e.g., an LLM that is itself prompted to mutate previously discovered prompts [35] ); and 3) a preference model that ranks adversarial prompts based on their effectiveness. For safety, this can be a "judge" LLM [71] that compares two responses to determine which is more unsafe.We demonstrate the effectiveness of RAINBOW TEAMING through extensive experiments targeting several state-of-the-art LLMs fine-tuned on safety-aligned data, including the Llama 2-chat [65] and Llama 3-Instruct [1] models. Despite the rigorous development of these models, our experiments reveal hundreds of adversarial prompts per individual run, achieving an attack success rate higher than 90% across all tested models without requiring external data. Using popular safety benchmarks, we demonstrate that RAINBOW TEAMING outperforms strong baselines in identifying vulnerabilities. Additionally, fine-tuning LLMs with synthetic data generated by our approach significantly enhances their adversarial robustness, improving resistance to unseen attacks and subsequent rounds of RAINBOW TEAMING, without diminishing their general capabilities and helpfulness.We further illustrate the versatility of RAINBOW TEAMING by applying it to other domains, such as question answering and cybersecurity, uncovering hundreds of effective adversarial prompts in each case. These findings underscore RAINBOW TEAMING's potential as a comprehensive tool for diagnosing and advancing the robustness and reliability of LLMs across diverse applications. 
background: 
 RAINBOW TEAMING builds on existing approaches in quality-diversity (QD) search to automate the discovery of a broad spectrum of adversarial prompts. QD methods seek to produce a collection of solutions that are individually high-performing and collectively diverse [34, 13] . Given a space of solutions X , the quality of a solution x ∈ X is measured using a fitness function f : X → R. The diversity of solutions is characterised using a feature descriptor function, d : X → Z that maps each solution to a point in a feature space Z = R N . This space encompasses specific pre-defined attributes of the solution, such as its behavioral aspects. For each z ∈ Z, QD searches for the solution x ∈ X such that d(x) = z and f (x) is maximised.Our work builds directly on MAP-Elites [46] , a simple yet effective QD method. MAP-Elites tracks the highest-fitness solutions in a multidimensional grid, referred to as the archive, which discretises the feature space Z. The archive is first initialised with random solutions. During each iteration of MAP-Elites, a solution x is sampled at random from the archive and modified to create a new solution x ′ (e.g., by injecting Gaussian noise). The new solution x ′ is then evaluated and assigned to its corresponding archive cell based on its descriptor z ′ = d(x ′ ). If the cell is vacant, or if x ′ has higher fitness than the current occupant, also known as the elite, x ′ becomes the new elite for that cell. Through repeated cycles of selection, mutation, and evaluation, MAP-Elites fills the archive with the highest-fitness solutions. Algorithm 1 in Appendix C provides the pseudocode of this method. 
rainbow teaming: 
 We now describe RAINBOW TEAMING, our new approach for automatically generating a diverse collection of adversarial prompts. RAINBOW TEAMING casts this task as a QD search problem with the solution space corresponding to all possible prompts. Our rationale for employing QD is twofold:• Effective adversarial prompts for specific scenarios (e.g., criminal planning) could be effective for others (e.g., cybercrime and hacking) with relatively small modifications. This adaptability implies that solutions can serve as stepping stones to accelerate the discovery of new adversarial strategies across different categories.• A thorough diagnostic of the vulnerabilities of a model calls for a comprehensive analytical tool to mitigate the risks of leaving attack vectors undiscovered. Similarly, safety fine-tuning requires a sufficiently diverse dataset to improve a model's adversarial robustness against a wide range of attacks. Diversity is essential for both of these objectives, and QD allows us to optimise it explicitly.RAINBOW TEAMING is based on MAP-Elites [46] . We store adversarial prompts as solutions in a K-dimensional archive, with each dimension corresponding to one of the pre-defined features. Each cell in the archive corresponds to a unique combination of K categories that describe the prompt within it, known as the cell's and the solution's descriptor, and denoted z = ⟨c 1 , . . . , c K ⟩. The LLM for which the adversarial prompts are generated is referred to as the Target. Initial solutions can be either generated randomly using an LLM or loaded from an existing dataset. As shown in Figure 2 , all key operation of the iterative search are performed with LLMs.At each iteration of RAINBOW TEAMING, we sample 1) an adversarial prompt x from the archive with descriptor z, and 2) a descriptor z ′ for the new candidate prompt to be generated. Note that z and z ′ are different. 2 We provide x and z ′ to the Mutator LLM to generate a new candidate prompt x ′ with descriptor z ′ . We then feed x ′ to the Target to generate a response. Finally, we ask a Judge LLM [71] to compare the effectiveness of the candidate prompt x ′ to that of the archive's elite prompt -the prompt stored in the archive with a descriptor z ′ . This comparison focuses on the criteria of interest, such as the toxicity of the Target response, to determine which of the two prompts more effectively meets the adversarial objective. We then store the winning prompt in the archive at the position specified by z ′ . Algorithm 2 in Appendix C provides the pseudocode of our method.RAINBOW TEAMING is highly versatile and can easily be applied to various settings by implementing three components: prompt features, a mutation operator, and a preference model.The features define the archive, with each predefined feature corresponding to one of the K archive dimensions. A feature can be either categorical or numerical. For categorical features, the axis of the archive is composed of discrete bins each representing a unique feature category. For instance, the Risk Category and Attack Style features in Figure 1 each consist of 10 categories. Numerical features are represented on a continuous scale, discretised into a set of intervals. Features therefore determine both the final archive size and the axes of diversity that RAINBOW TEAMING prioritises. This is particularly true given their interplay with the mutation operator, as described next.RAINBOW TEAMING generates new candidates by applying directed mutations to previously discovered adversarial prompts. The Mutator receives a parent prompt x sampled uniformly at random from the archive and the prescribed descriptor z ′ = ⟨c ′ 1 , . . . , c ′ K ⟩ for the candidate. It then mutates the prompt x once for each feature -K times overall -to produce a new candidate prompt x ′ .Sampling the candidate's descriptor in advance confers several key benefits. First, this allows us to forgo using a classifier for assigning the candidate to its corresponding cell, which can be inaccurate. Second, it introduces more diversity by mitigating the biases of the Mutator, which could otherwise neglect entire categories. Third, it helps avoid spending iterations on areas of the archive for which we already have effective adversarial prompts. We do this by biasing the sampling distribution of the descriptors towards areas of the archive with low fitness. We compute fitness explicitly for this purpose but do not use it to inform archive updates.To further promote diversity, the candidate prompt is considered for further evaluation only if it is sufficiently dissimilar from its parent. We measure the similarity using BLEU [49] and filter out prompts that have high BLEU scores with respect to their parents.The preference model, operated through the Judge, performs the ranking of adversarial prompts based on their effectiveness (e.g., whether they elicit unsafe responses). The Judge inputs can vary between domains, but preference-based evaluations include the Target responses to both the candidate and the existing prompt from the archive with descriptor z ′ . The Judge determines which prompt is more effective using a majority vote over multiple evaluations and swapping prompt positions to mitigate order bias [71] . If the candidate wins the comparison, it replaces the existing prompt.Relying on a preference model rather than a score-based evaluator offers two advantages. First, LLMs prompted to perform pairwise comparisons have a higher agreement with humans than those performing single-answer grading [71] . This is particularly true in an optimisation context, which introduces the risk of reward hacking the evaluator. Second, the score of any numerical evaluator with a fixed scale can be maximised, at which point it is impossible to identify better candidate prompts, resulting in minimal updates in the archive. We present a preference model ablation supporting those claims in Appendix E.4.While we describe RAINBOW TEAMING as using LLMs for all key steps, those can be substituted by other models or rule-based components in some domains (e.g., see Section 6.1). 
rainbow teaming for safety: 
 In this section, we evaluate the effectiveness of RAINBOW TEAMING in discovering adversarial prompts that trigger unsafe (i.e., harmful, offensive, or toxic) responses. We target various models that are fine-tuned with safety-aligned data and are among the safest publicly available LLMs, including Llama 2 and Llama 3 models.• Features: We use two feature dimensions: Risk Category and Attack Style. Our risk categories expand upon the safety risk taxonomy in Inan et al. [26] and cover common safety issues in LLMs, such as "Criminal Planning", "Violence and Hate" or "Self-Harm". Attack Style pertains to the method of eliciting unsafe responses, such as "Authority Manipulation" and "Role Play". Together, these features enable a thorough evaluation of LLM safety, covering a broad spectrum of adversarial threats and tactics. The full list of categories for both features is provided in Appendix H.1. • Mutation Operator: We use an instruction-tuned Llama 2 70B model, which applies two mutations to create the candidate prompt, as illustrated in Figure 2 . The first mutation targets the risk category, using a few-shot prompt template shared across all categories. The second mutation focuses on the attack style, employing few-shot prompts handcrafted for each individual category. • Preference Model: We use an instruction-tuned Llama 2 70B model as a Judge to compare Target responses to two adversarial prompts. Its role is to determine which response is more unsafe, ignoring the prompts. The prompt corresponding to the winner's response is the one we then keep in the archive. Details about the prompts used for the Judge are provided in Appendix I.To initialise the archive, we generate prompts from scratch according to risk categories without relying on a pre-existing seed dataset. These are then mutated with respect to attack style and used as a foundation for subsequent evolutionary iterations. We use two independent measures of Attack Success Rate (ASR) -the proportion of prompts in the archive that elicit unsafe responses. Specifically, we use two safety classifiers, namely GPT-4 [48] with a specialised prompt and the Llama Guard safeguard model [26] . GPT-4 acts as a binary classifier for whether a response is generally unsafe or not, independent of risk categories. In contrast, Llama Guard is prompted to perform per-category binary classification [26] , considering an attack successful only if the resulting response violates the risk category it is assigned to. Neither of these metrics is explicitly optimised by RAINBOW TEAMING, but the probability of Llama Guard classifying a prompt as unsafe is the fitness score used to bias the selection of the prescribed feature descriptors for new candidates. Prompts for both evaluators are provided in Appendix I. For all experiments, we report the mean and standard error over 3 independent runs.We also measure inter-evaluator agreement on 100 pairs of prompts and responses. Table 8 in Appendix E.3 shows that human-human agreement (83%) is similar to human-AI agreement (81% for GPT-4 and 78% for Llama Guard) and GPT-4-Llama Guard agreement (79%), and is consistent with prior work [71] . We therefore use GPT-4 and Llama Guard as proxies for human evaluation. Main Results. Figure 3 presents the ASR of RAINBOW TEAMING when applied to the Llama 2-chat 7B [65] , Llama 3-Instruct 8B [1] , Mistral 7B [27] and Vicuna 7B v1.5 [11] models across 2000 iterations, using GPT-4 for evaluation. RAINBOW TEAMING is highly effective, generating a large collection of adversarial prompts against all models. The Llama models exhibit the highest robustness: following 2000 iterations, we obtain archives of 100 prompts with an approximate ASR of 92% against both variants. Mistral 7B and Vicuna 7B demonstrate a higher level of vulnerability with 98% of the adversarial prompts in RAINBOW TEAMING-generated archives being successful. These results are echoed by the ASR reported by Llama Guard in Figure 10 .While Figure 3 showcases relatively small LLMs, RAINBOW TEAMING is equally effective against larger models. Figure 8 in Appendix E.1 presents results of RAINBOW TEAMING targeting 7B, 13B, and 70B variants of Llama 2-chat model, achieving 90% or higher ASR across all model sizes.We compare RAINBOW TEAMING to two baselines. The first baseline (No Stepping Stones) ignores past solutions in the archive and generates new prompts based on the risk category, before applying the attack style mutation, effectively repeating the process we use to initialise the RAINBOW TEAMING archive. The second baseline, (Same Cell Mutations), is identical to RAINBOW TEAMING, except that it uses the parent prompt's descriptor as the candidate prompt descriptor, i.e., it performs mutations within each archive cell independently. Figure 4 shows RAINBOW TEAMING outperforming both baselines, highlighting the value of stepping stones in one case and the significance of cross-category mutations in the other.JailbreakBench Results. We also apply RAINBOW TEAMING towards eliciting specific harmful behaviours from the JailbreakBench [9] dataset. Using the same attack styles, we generate 1000 prompts evenly spanning 100 harmful behaviours, with results presented in Table 1 . We compare against two PAIR [8] variants: one from Chao et al. [9] , based on MiXtral, and another using the same mutator LLM as our RAINBOW TEAMING implementation, with N = 20 parallel streams generating a total of 2000 prompts. We classify jailbreaks using both the same classifier as Chao et al. [9] and Llama Guard prompted with the harmful behaviours. For each prompt, we regenerate 4 responses and consider the prompt successful if any of the responses is classified as harmful. We believe this is representative of user interaction with LLMs, where they can prompt the model repeatedly in the hope of obtaining a different response. Compared to both PAIR variants, RAINBOW TEAMING discovers more jailbreaks across more behaviours, while also maintaining much higher prompt diversity.Transfer of Adversarial Prompts. Understanding whether attacks transfer across models is important to assess the generality of the adversarial prompts, and whether they are intrinsically tied to the models they are optimised for. To evaluate transfer, we take the final prompts generated by RAINBOW TEAMING for each original target in Figure 3 and evaluate their ASR against other transfer targets.Table 1 : Comparison of RAINBOW TEAMING against PAIR [8] for eliciting harmful behaviours from JailbreakBench [9] . Top: (n/k) indicates the total number of successful jailbreaks (n) and the total number of behaviours jailbroken (k) for each method and classifier (best of 4 responses). Bottom: Self-BLEU similarity score.PAIR PAIR with RT mutator LLM RAINBOW TEAMING JailbreakBench Classifier [9] (↑) -/4 1/1 8/7 Llama Guard (JBB Behaviours) (↑) -14/11 66/41Self-BLEU (↓) -0.74 0.51Table 2 presents the ASR on four different models using archives generated by RAINBOW TEAMING targeting each of these models. We show the ASR in grey when re-prompting targets using their own archive. On average, the ASR when transferring prompts is 50% of the ASR against the original target, indicating that RAINBOW TEAMING discovers general prompts which apply to multiple models. However, the exact transfer rate is highly dependent upon the pairing of original and transfer targets. We find that prompts transfer better from safer to less safe models than in the opposite direction. That said, the highest transfer rate is from Vicuna 7B 1.5 to Mistral 7B, even though Vicuna is fine-tuned from a Llama 2 base. We also achieve up to 66% ASR on GPT-4o, indicating no significant difference between open and closed-source models. Impact of the Similarity Filter. Because archive categories are not mutually exclusive, we run the risk of populating the archive with near identical prompts. This is useful for discovering a category-agnostic failure mode but comes at the cost of significant diversity loss in the archive. To mitigate the issue, we implement a parent-child similarity filter at the mutation stage, as described in Section 3.2. Table 3 compares the performance of RAINBOW TEAMING with and without using this similarity filter. We also report archive self-BLEU [73] , BERTScore [70] , ROGUE-L [37] m and compression ratio [61] scores designed to measure the diversity of a whole dataset. Our results show that the similarity filter is an effective way of maintaining the linguistic diversity of the archive.Table 3 : Analysis of the effect of a mutation-level similarity filter of RAINBOW TEAMING on ASR measured by GPT-4 and archive diversity (self-BLEU, BERTScore, ROGUE-L, and gzip compression ratio). Filtering out prompts that are too similar to their parent maintains a balance between ASR and diversity, whereas removing the filter encourages the method to reuse highly effective prompts across multiple cells. The filter is set at τ = 0.6, discarding ∼ 24% of mutated prompts. We report mean and standard error over 3 independent runs. Additional results with different system prompts are provided in Appendix E.2. We include an ablation study in Appendix E.4 to assess the role of the preference model. We discuss computational costs in Appendix G. 
enhancing robustness with synthetic data: 
 Generating diverse, high-quality instruction-tuning datasets can be expensive, often requiring human annotations. RAINBOW TEAMING offers a low-cost alternative, generating diverse synthetic data that specifically targets the model's vulnerabilities. In this section, we demonstrate the usefulness of RAINBOW TEAMING as a synthetic dataset generation method by applying it to improve the safety of LLMs. We find that training on our synthetically generated data improves robustness to adversarial prompts while retaining the general capabilities of the model.We use RAINBOW TEAMING to generate 15 archives targeting the Llama 2-chat 7B model, yielding a total of 1500 adversarial prompts. We perform a 12/3 train-test split and use Llama 2-chat 70B with a handcrafted system prompt to generate safe refusal prompts for the train set. We then perform supervised fine-tuning (SFT) [67] on this dataset and evaluate the ASR of the 300 held-out prompts before and after SFT. As shown in Table 4 , we find that fine-tuning Llama 2-chat 7B on the synthetic dataset generated by RAINBOW TEAMING substantially reduces the attack success rate from 92% / 95% to 0.3% / 0.7%, as measured by GPT-4 and Llama Guard. Similarly, the ASR of PAIR [8] on the JailbreakBench (JBB, [9] ) behaviours drops from 14% to 0% (measured by Llama Guard, as in Table 1 ). This demonstrates that additional SFT on RAINBOW TEAMING data also improves safety against out-of-distribution attacks. Crucially, SFT does not diminish the model's general capabilities as measured on the GSM8K (8-shot, maj@1) [12] and MMLU (5-shot) [23] benchmarks. Table 4 also reports the reward model scores [65] of the Llama 2-chat 7B model before and after SFT. We report safety and helpfulness scores on the Anthropic Harmless and Anthropic Helpful datasets [19] respectively. We observe a 1.5% safety score increase, despite the fact that Llama 2-chat models use the Anthropic Harmless dataset as a part of the reinforcement learning from human feedback (RLHF) pipeline [65] . This is accompanied by a 0.5% drop in helpfulness, which we attribute to fine-tuning the model exclusively on the adversarial prompts produced by RAINBOW TEAMING. Mixing the adversarial data with helpfulness data would likely negate this effect, but we leave the study of adversarial fine-tuning strategies to future work.To further investigate the robustness of the newly fine-tuned model, we reapply RAINBOW TEAM-ING to the Llama 2-chat 7B model after finetuning it on synthetic data generated by our method. As shown in Figure 5 , the new model is substantially more robust to our approach, with a final ASR of 39% (down from 92%). We expect that performing multiple rounds of RAINBOW TEAMING, alternating between collecting synthetic data and adversarial fine-tuning, will further increase the model's robustness to adversarial attacks. We show examples of archives at different iterations of RAINBOW TEAMING before and after SFT in Figure 13 .6 RAINBOW TEAMING for Other Applications Results. In Table 5 we compare RAINBOW TEAMING to a baseline that generates candidate questions from scratch rather than relying on existing questions in the archive. We observe that RAINBOW TEAMING achieves higher fitness, higher coverage (percentage of non-empty cells in the archive), and higher diversity in questions, indicating the importance of utilising previously discovered adversarial questions. Importantly, not relying on previous solutions leaves regions of the archive uncovered, particularly for short questions as seen in the example archives in Appendix E. Figure 6 illustrates an example archive generated using RAINBOW TEAMING. Some example questions are also shown in Appendix E.7. We apply RAINBOW TEAMING to cybersecurity, searching for adversarial prompts that elicit behaviour such as generating insecure code or providing assistance in orchestrating cyberattacks. We use a 2D archive with the 10 MITRE categories for cyberattack tactics [45] (e.g., "Exfiltration" or "Defense Evasion") and prompt length divided into 10 equal bins. Our Mutator is an instructiontuned Llama 2 70B model, mutating first for MITRE attack style, and then for prompt length. We use a binary Judge mechanism involving Llama 2-chat 70B and CodeLlama-34B Instruct models to evaluate generated prompts, as outlined in CyberSecEval [4] . We provide further details in Appendix F.2.Results. Table 6 presents the results of a cybersecurity assessment for various target models on prompts generated by RAINBOW TEAMING. For all models, we successfully generate 10 × 10 archives that are fully identified as malicious, as classified by CyberSecEval [4] . Human expert evaluation finds a lower ASR, with 0.94 and 0.92 for Llama 2-chat 7B and CodeLlama 7B Instruct, and 0.8 for both Llama 2-chat 70B and CodeLlama 34B Instruct. While RAINBOW TEAMING remains highly effective, the discrepancy between CyberSecEval and expert annotations suggests the need for a better cybersecurity-specific evaluation, which we hope will be the focus of future work. 
related work: 
 Adversarial Attacks on LLMs. RAINBOW TEAMING relates most closely to prompt-level attacks which rely on strategies such as misspellings, prompting in foreign languages [68] , or personamodulation [60] to jailbreak LLMs. Perez et al. [51] use an LLM and a brute-force approach to automatically discover prompt-level attacks, but this approach can suffer from mode collapse and does not always generate a diverse set of prompts. Meanwhile, Liu et al. [38] propose a white-box method that refines hand-crafted attack prompts using a mix of genetic algorithms and LLM-based mutations. However, they focus on optimising a single solution rather than a diverse population.The closest works to our own are PAIR [8] and Tree of Attacks with Pruning (TAP) [41] -two black-box methods for automatically discovering prompt-level attacks by using an LLM to iteratively generate candidates. However, both methods are designed to jailbreak the model with respect to a single task rather than across a range of diverse risk categories and attack styles. In contrast, our work uses quality-diversity search to automatically discover attacks covering a diverse set of risks and attack strategies. Although evolutionary algorithms have previously been used for adversarial attacks on LLMs [38, 32, 8] , this work is the first to apply a quality-diversity framework [34, 13] in this area. Unlike most evolutionary algorithms (e.g., genetic algorithms), which evolve a single optimal solution, quality-diversity approaches generate a wide variety of distinct, high-quality solutions.Open-Endedness and LLMs. RAINBOW TEAMING builds on the ability of LLMs to act as a powerful mutation operator over language inputs, one that adheres to the underlying structure of natural language [35] . Several recent methods exploit this capability of LLMs in order to perform an efficient novelty-driven evolutionary search in the language space, leading to the discovery of potentially open-ended repertoires of solutions [10, 16, 43] . Closest to our approach is QDAIF [6] which similarly uses LLMs for QD search in order to generate a diverse archive of LLM outputs. RAINBOW TEAMING is different from QDAIF in several important factors. First, we search for and archive diverse prompts for the target LLMs, whereas QDAIF archives diverse responses from it -a separate problem altogether. While QDAIF focuses purely on generating diverse outputs for creative writing, our method seeks to find a diverse set of adversarial prompts. QDAIF relies on a score-based fitness function (log probability of the token generation), whereas RAINBOW TEAMING uses a preference-based judge for performing updates to the archive. RAINBOW TEAMING additionally incorporates parent-child similarity filtering to preserve the linguistic diversity of the prompts.An extended related work section is provided in Appendix B.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Error bars are included in all plots, and standard deviation is reported in every table where applicable.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 In this work, we introduce RAINBOW TEAMING, a novel approach for the automatic generation of diverse adversarial prompts for LLMs. By leveraging quality-diversity search, RAINBOW TEAMING efficiently explores the space of potential adversarial attacks, resulting in a diverse archive of prompts that highlight the vulnerabilities of LLMs. Our extensive experiments with multiple models, such as Llama 3-Instruct and Llama 2-chat, and across various domains, including safety, question answering, and cybersecurity, demonstrate the generality of RAINBOW TEAMING. Moreover, the synthetic data generated through RAINBOW TEAMING can be utilised for fine-tuning LLMs, thereby enhancing their resilience against further adversarial attacks without compromising their general performance. This illustrates the potential of RAINBOW TEAMING as a means for the continuous, open-ended selfimprovement of LLMs, with minimal human intervention. Future work with RAINBOW TEAMING involves extending its application beyond LLMs to areas such as vision and multi-modal AI systems. Moreover, incorporating RAINBOW TEAMING into the fine-tuning stages of LLM development presents an opportunity to consistently strengthen their defences against adversarial attacks.We discuss the limitations and broader impact of our work in Appendix A.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We include details on the computational resources used for our experiments in Appendix G Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Rainbow Teaming is an adversarial prompt generation method, which by definition is dual-use. Similarly, any jailbreak dataset carries a small but non-negligible misuse potential. As such, we have chosen for now to not release our code or datasets while we evaluate the risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Despite many advantages of RAINBOW TEAMING, its current implementation has several limitations. First, the features that define the archive and its categories are pre-defined and fixed. In future work, it would be interesting to extend our approach to discover features and categories automatically. Another limitation of RAINBOW TEAMING is that the number of prompts it can generate is constrained by the grid size. While this is due to using MAP-Elites as the base QD algorithm, we note that even the current setting allows generating hundreds of adversarial prompts from a single run and this can be extended by providing additional features or categories or storing several values within the same archive cell.Unlike simpler adversarial attack methods [8] , RAINBOW TEAMING requires extensive computational resources. Furthermore, its undirected, open-ended approach is less likely to produce a prompt for a specific behaviour (e.g., writing a fake news article about a specific public figure). While these attributes can be considered limitations, we highlight that because of them, RAINBOW TEAMING is less likely to be used for malicious purposes. The primary value of RAINBOW TEAMING lies in its potential to identify and address robustness issues in LLMs, contributing to their responsible development and deployment.Ultimately, we believe RAINBOW TEAMING to be a powerful tool in improving the robustness of LLMs to adversarial attacks and see the prompts it generates as a valuable complement to crowdsourced data.Token-level attacks circumvent the LLM's defences against generating undesirable responses by adding adversarial tokens to a malicious prompt. Such methods originally required white-box access to the LLM [74] , but that assumption has since been relaxed using black-box optimisation [33, 40] . Token-level attacks have proven effective, but brittle to perturbations [54] . Although RAINBOW TEAMING could be adapted to create token-level attacks by integrating the appropriate attack categories and prompts, we restrict this study to prompt-level attacks given that prompt-level attacks are more interpretable and harder to detect.RAINBOW TEAMING's approach parallels other forms of adversarial training, which prioritises training on tasks or data points where the model performs poorly. In reinforcement learning (RL), methods such as active domain randomisation [42, 53] and regret-based unsupervised environment design [14, 29, 50, 56] search for training tasks where the agent performs poorly in terms of absolute task performance or regret, respectively. Regret-based prioritisation has been shown to hold robustness guarantees at convergence and carry the benefit of avoiding unsolvable tasks (which always result in zero regret). The fitness score used by RAINBOW TEAMING coincides with regret [58] , as a high fitness here implies the existence of another prompt that elicits a less undesirable response, as evaluated by the Judge. Similarly, many active learning and automatic curriculum learning methods in supervised learning focus training on examples maximising error metrics derived from the model's predictions [22, 44, 15] . Dynabench [31] extends this paradigm by querying humans-in-the-loop for adversarial examples. Many methods in scenario generation also closely relate to RAINBOW TEAMING, including recent approaches using QD search to find adversarial environments that induce poor behaviour in fully-automated or mixed-autonomy systems [18, 17, 5] . This extends to recent work applying QD to multi-agent RL [57] , which inspired our method.Algorithm 1 provides a pseudocode of MAP-Elites method [46] described in Section 2.Algorithm 1: MAP-Elites [46] Input: fitness function f , dimension K, feature descriptor function d, mutation function m, number of seed solutions n Initialise: Empty K-dimensional grid of solutions G (the archive) and grid of fitness scores F Populate G with n random initial solutions and F with corresponding fitness scores for i = {1, 2, . . . } dox ∼ G # Sample a solution x from archive. x ′ ← m(x) # Create new solution x ′ by mutating x. f ′ ← f (x ′ ) # Compute the fitness score of the new solution x ′ . z ′ ← d(x ′ ) # Get the descriptor of the new solution x ′ . if G[z ′ ] = ∅ or F [z ′ ] < f ′ then # If. if G[z] = ∅ then # If corresponding cell in archive is empty. G[z] ← x ′ # Update the archive with prompt x ′ . R[z] ← r ′# Update the response for the new prompt.F [z] ← f (x ′ )# Update the fitness score for the new prompt. else # If corresponding cell in archive is not empty.r ← R[z]# Get the response to the archive's prompt with descriptor z. if p(π J , r ′ , r) then # If the preference model concludes that r ′ is more adversarial.G[z] ← x ′ # Update the archive with prompt x ′ . R[z] ← r ′# Update the response for the new prompt.F [z] ← f (x ′ )# Update the fitness score for the new prompt. Return: G, R, F Throughout this work, we use BLEU score [49] as the similarity metric sim. In the safety domain, we use the probability of Llama Guard categorising a response as unsafe as the fitness function f . The fitness function is used for biasing the sampling of descriptor d but not for updating the archive.For clarity, the algorithm shows the RAINBOW TEAMING loop over a single prompt x, but the process can be batched to reduce wall clock time. In practice, we use batch sizes between 16 and 64.Figure 7 provides a qualitative example of how the directed mutation in RAINBOW TEAMING can produce diverse adversarial prompts from a single common ancestor. Figure 8 presents the ASR of RAINBOW TEAMING when applied to Llama 2-chat models with 7B, 13B, and 70B parameters across 2000 iterations, using GPT-4 and Llama Guard for evaluation. Archives generated through RAINBOW TEAMING demonstrate 90% or higher ASR across all model sizes, as measured using GPT-4 and Llama Guard evaluators.While our main experiments provide the prompts to the Target as is (within appropriate instruction tokens), we additionally analyse incorporating two system prompts. The legacy system prompt is designed to emphasise both safety and helpfulness. 4 The helpful system prompt is a handcrafted variant of the legacy prompt that focuses on helpfulness without explicitly emphasising safety. All system prompts are provided in Appendix I.5. The effectiveness of RAINBOW TEAMING when using these different system prompts is presented in Table 7 . Our results indicate the inclusion of a system prompt emphasising safety diminishes the success rate of adversarial attacks to 51% / 74%, according to GPT-4 and Llama Guard evaluations, respectively. However, using this system prompt makes the model overly conservative, occasionally refusing to answer benign questions that appear unsafe. On the other hand, the helpful system prompt, remains vulnerable to attacks, with 82% / 93% ASR, yet still offers improved robustness compared to not using a system prompt at all, which sees 92% / 95% ASR. The Llama 2-chat 7B model has been safety-aligned regardless of the system prompt, but its robustness is highly dependent on this variable.Table 8 shows the ASR of 100 prompts, as evaluated by GPT-4, Llama Guard and human annotators. The inter-evaluator agreement is also provided. The 100 prompts were randomly sampled from a combination of archives targeting Llama 2-chat 7B or 70B, using either no system prompt or the Legacy prompt. This was done to ensure sufficient prompt variety. The annotations were done within a single session by four of the authors following the same harm taxonomy as Llama Guard. The annotators were not informed of the data mix or the expected ASR as evaluated by either Llama Guard or GPT-4 at the time of annotation to mitigate bias. All annotations were performed through an interactive spreadsheet, where authors tick a checkbox to label the model response as "GOOD BOT" (when the target model refused to answer or answered in a safe manner) or "BAD BOT" (when the model answered in a way that is violating according to the harm taxonomy).We investigate the role of the preference model used for RAINBOW TEAMING. One option is to prompt an LLM to act as a pairwise comparison Judge [71] . The Judge takes as input the response to the candidate prompt and the response to the prompt already in the archive, and is instructed to say which of the two is more harmful. As recommended by Zheng et al. [71] , we use position swapping (evaluating twice for each order of the prompt-response pairs) to reduce positional bias, and few-shot prompting to improve consistency. We refer to this variant as "Comparison-based (Judge)" and use it as default throughout the paper.We compare the default version of RAINBOW TEAMING with a variant that uses the Llama Guard probability of classifying the response as "unsafe" as a preference model. In this case, we perform an archive substitution if the probability for the candidate response is higher than that of the existing response. We refer to this RAINBOW TEAMING variant as "Score-based (No Judge)".In our evaluation, as shown in Figure 9 , the score-based baseline achieves a higher Llama Guardevaluated ASR, aligning with its optimisation objective. However, it falls short in GPT-4-evaluated ASR, suggesting overfitting to Llama Guard scores, indicative of reward hacking [63] . Qualitatively, we find that the adversarial prompts produced by the score-based method are also of lower quality. We also show the number of archive updates for the two variations of RAINBOW TEAMING. We observe that the No Judge baseline quickly maximising the Llama Guard score (capped to 1.0) leads to sparse updates thereafter. In contrast, the Judge-based variant continues to refine the quality of the adversarial prompts in the archive, indicated by ongoing archive updates, even after filling the archive with successful prompts. This underscores the advantage of RAINBOW TEAMING's open-ended search process over a purely score-driven approach.Note that the performance differences between RAINBOW TEAMING results here and in other parts of the manuscript arise from variations in the experimental setup. In this specific experiment, we use Anthropic Harmless as the seed dataset with slightly different mutation prompts, and two risk category names have been updated. Figure 10 presents the ASR of RAINBOW TEAMING when applied to Llama 2-chat 7B [65] , Llama 3-Instruct 8B [1] , Mistral 7B [27] and Vicuna 7B v1.5 [11] models across 2000 iterations, using both GPT-4 and Llama Guard for evaluation. Figure 11 shows the performance of RAINBOW TEAMING against No Stepping Stones and Same Cell Mutations baselines, using GPT-4 and Llama Guard for Figure 12 : Attack success rate before and after fine-tuning Llama 2-chat 7B on synthetic data generated via RAINBOW TEAMING. The fine-tuned model is significantly less vulnerable to RAINBOW TEAMING on a second application, with the method achieving a substantially lower ASR after 2000 iterations. We report the mean and standard error over 3 independent runs. When was the largest living organism in the world, which is a type of fungus, first discovered in Oregon?1860s 1998Where was the famous equation that measures the strength of a celestial body's gravitational pull first proposed?Galileo Cambridge conditions are met, we save the question to the archive with fitness 1. If not, but the archive cell is empty, we save the question with a fitness of 0.1 to act as a stepping stone. We discard the question in all other scenarios.When using RAINBOW TEAMING in the cybersecurity domain, we use a binary Judge mechanism to evaluate the potential maliciousness of generated prompts. This is aligned with the framework outlined by Bhatt et al. [4] , which leverages two distinct LLMs for classifying the output of an LLM as either "benign" or "malicious". For the initial processing of the output, we use the Llama 2-chat 70B model, which is responsible for preliminary analysis and identification of potentially malicious content. The final decision of whether a response is indeed malicious is then made by the CodeLlama-34B Instruct model [55] .Since inference costs vary based on infrastructure, the number of generation tokens, and specific LLMs used, we choose to discuss computational costs in terms of LLM inference calls. For the majority of our experiments, one RAINBOW TEAMING loop requires 2 Mutator inference calls, 1 Target inference call, and 4 Judge inference calls, where every inference call corresponds to a batch of prompts (our batch size is 16 for most experiments, except 64 for our JailbreakBench ones). When running for 2000 iterations, this represents a total of 14000 batched inference calls per run.We conducted our experiments on a cluster of A100 GPUs, with access ranging from 128 to 256 GPUs throughout the project. Each run was typically completed in around two days, though we often accelerated them significantly by leveraging a distributed client-server setup for parallelised LLM inference. Although the computational cost is substantial, we believe the effectiveness and diversity of the resulting prompts make our method a highly competitive alternative to crowd-sourced data.Depending on the user's needs, costs can be further amortised by leveraging earlier RAINBOW TEAMING checkpoints as a source of additional prompts, potentially expanding the final dataset size by an order of magnitude. As LLMs become more powerful, so do the components of our method. For example, if an LLM can serve as an accurate judge in a single inference call, this could significantly reduce the inference cost associated with the Judge. This section includes various prompts used in experiments in the safety domain.I. For Llama Guard evaluations, we use the default system prompts by Inan et al. [26] for 6 original categories. For the 4 new categories, similar system prompts have been generated by the authors.We have elected not to share mutation prompts to prevent misuse. Tables 10, 11, and 12 provide the hyperparameters used in safety, question answering, and cybersecurity domains, respectively. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material.Justification: The research conducted in this paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Appendix A includes a discussion on the broader impact of the work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).For additional adversarial prompts and details, visit our website at https://sites.google.com/view/ rainbow-teaming.In Figure2, z = ⟨"Criminal Planning", "Role Play"⟩, while z ′ = ⟨"Fraud and Scams", "Misspellings"⟩.Touvron et al. [65] report base model scores on these benchmarks while we report those of the chat model.It was initially released with Llama 2 but has since been deprecated due to its high false refusal rate. See the change here. 


Decision: The paper addresses automatic red teaming of large language models through open-ended generation of jailbreaks. The key components of their methodology are 1) a categorization of different jailbreak categories to create a diverse archive of possible jailbreaks, 2) a strategy to evolve and mutate jailbreaks, 3) and a selection process to keep the jailbreaks with the highest quality.

There are 3 groups of reviewers:
- Borderline Reject (hknM, DdX9): These reviews contain nearly nothing about the content and the contributions of the paper. The only detailed limitations identified by  these reviews are superficial technicalities: 1) the absence of plan to open source the code (hknM), 2) the absence of use of commercial models with restrictive ToS such as GPT4 and Claude (DdX9), and 3) the use of automated rather than human evaluations for a large part of the experiments. As the AC works on similar tasks, he is aware that these limitations are inherent to the task, and legally or financially impossible to remedy (see also comments from Reviewer w9N7). Reviewer DdX9 did not even respond to the author rebuttal. The AC considered these 2 reviews as uninformative  and asked for a fifth reviewer to have 3 high quality reviews.
- Reject (sYeN:3)
- Accept (w9N7:7 and E172:8)

In the rest of the meta-review, I will only consider the high-quality reviews (sYeN , w9N7 and E172). The average score for these reviews is 6.

All reviewers seem to agree on the strength of the paper: it is well-written, the method is effective and the evaluation is extensive. From reviewer w9N7: “The paper presents a considerable contribution towards practical automatic redteaming of large language models and address multiple weaknesses of prior work (such as attack diversity). The authors provide an exhaustive empirical evaluation of the proposed method with extensive hyperparameter descriptions.”

The main source of disagreement is about the novel technical contribution:
•	Reviewer sYen: Contribution and novelty seems very marginal. The difference from methods such as PAIR and TAP appears to come down to 1) presenting the attack/mutate LLM with high level categories instead of specific behaviors and 2) specifying several concrete strategies instead of relying on the attack/mutate LLM to come up with them on the spot
•	Author rebuttal: PAIR and TAP iterate on a single attack with a multi-turn attacker and have no diversity component. In contrast, Rainbow Teaming jointly optimizes a set of attacks, the attacker (or Mutator) is single-turn and it mutates prompts from other parts of the archive. 
•	Reviewer E172: Rainbow Teaming can be categorized as a synthetic data generation method for adversarial settings. Synthetic data generation methods that are similar in nature (e.g. PAIR, MAP-Elites) are recognized by the authors. Despite bearing similarity in the fundamental approach, there are certainly aspects of originality that enable the method to distinguish itself from those that come before it.
•	Reviewer w9N7: Novelty. I believe this to be a very bad metric to judge a paper. "On Adaptive Attacks to Adversarial Example Defenses" was initially rejected for low novelty but had a big impact on the field and now has nearly 1000 citations. The kind of red-teaming studies performed in this paper are important to guide new approaches to defend and attack LLMs.

Reviewer w9N7 points to the central issue with red teaming and jailbreaking papers: their contribution is mostly empirical, with no new novel ML algorithm. Thus they tend to be systematically rejected: the 2 previous works that reviewer sYen uses to challenge this work were either rejected (https://openreview.net/forum?id=hkjcdmz8Ro) or only accepted at a workshop (https://openreview.net/forum?id=AsZfAHWVcz)

This is the best jailbreaking algorithm paper I have read so far. Compared to previously published work, it is more principled, offers a far more extensive benchmark and introduces a diversity component. One limitation compared to previous academic studies is that they do not jailbreak commercial models (e.g. GPT-4), due to the fact that the authors work for a big tech corporation (Meta). Jailbreaking algorithm papers have so far been published in workshops only, and the only work on jailbreaking published at NeurIPS was a deeper behavioral study (Jailbroken: How Does LLM Safety Training Fail?).
While I believe this work is significant and of high interest to the NeuRIPS leadership, the program committee should make an overall decision about the acceptability of such papers.

Reviews: review: paper_summary: paper_summary: I'd like to thank the authors for submitting their work for review. I found the work insightful, inspiring, and high-quality. In short, the work has been a pleasure to review as a last-minute reviewer. 

The manuscript's primary contributions include:
- **Rainbow Teaming Method.**  A new methodology for automatically generating adversarial prompts through the lens of a quality-diversity problem.
- **Demonstrative Evaluation for Safety.** A demonstrative evaluation of the Rainbow Teaming methodology's utility application to the task of identifying prompt vulnerabilities that exist in a series of generative models. This demonstration also validates the underlying components of the Rainbow Teaming approach (e.g., the choice and design of the Preference Model).
- **Fine-Tuning Evaluation for Safety.** A fine-tuning experiment that illustrates how fine-tuning Llama-2-chat 7B on a dataset of 1,500 synthetically generated adversarial prompts with SFT reduced the attack success rate from 92% / 95% to 0.3% / 0.7%. 
- **Post-SFT Evaluation for Safety.** The authors re-apply Rainbow Teaming to the fine-tuned model produced from the Fine-Tuning Evaluation and report that the model is "substantially more robust to our approach, with a final ASR of 39% (down from 92%".
- **Non-Safety Evaluations**. The authors contribute two additional experiments that illustrate the method's efficacy for the Question-Answering and Cybersecurity settings, each of which contribute a set of concise, abbreviated findings in their own right.

The manuscript has several other minor contributions that aren't explicitly referenced as contributions, but should not go unnoticed (e.g., the extended taxonomy of safety risk categories that was previously defined by Inan et al.)
strength_weakness: strength_weakness: The work has several weaknesses that should be taken seriously, but not viewed as disqualifying. I view each of the following weaknesses as nothing more than "expected".

The weaknesses are as follows:

1. **Longitudinal Practicality.** The authors make a number of claims about the Rainbow Teaming method's ability to improve the robustness of generative models. While this is clearly demonstrated in the manuscript's family of experiments, the claim is weakened by the notion that the experiments do not provide information about how the Rainbow Teaming method may operate over time (i.e., in which adversarial methods evolve in new and unexpected ways). 

2. **Attack Styles and Risk Categories.** The paper's contributions are bound by a static set of attack styles and risk categories. It remains unclear if the methodology would perform similarly with other styles or categories.

3. **Minor Weaknesses.** There are two minor weaknesses:
 - **Model Choice.** Conducted experiments are performed with models that are now viewed as potentially being dated (i.e., and are no longer state-of-the-art). This weakness is stated out of recognition that the model choice itself is a *potential* weakness. Regardless of whether this be recognized more formally as a weakness, I strongly believe that reviewers refrain from scrutinizing the choice of models as the work simply uses them as a vehicle for demonstrating their methodology.
- **Diversity Metrics.** Diversity is primarily measured via BLEU, which one flavor of measurable diversity. Common practice is increasingly gravitating toward the measurement of multiple metrics that are reported as a collective, e.g. https://arxiv.org/html/2403.00553v1.
questions: questions: 1. Can you please clarify the significance of the initial adversarial prompt sampled from the archive? Specifically, describe precisely what might happen in the Rainbow Teaming method if the sampled adversarial prompt is no longer effective in its attack.
2. What was the motivation behind choosing Question-Answering and Cybersecurity as two secondary areas of evaluation? 
3. In Appendix A, the authors report that "Rainbow teaming requires extensive computational resources". While I understand the MAP-Elites approach may be computationally intense, it isn't exactly clear to me why the method itself is any more computationally intensive than preexisting methods. 
4. In Appendx I4, the authors state that they've opted to refrain from sharing the prompts that facilitate mutation. To some degree, this seems nonsensical and misaligned to the broader goals of democratizing AI safety research in the community. Can the authors please provide a deeper explanation and justification for refraining to share the prompts?
limitations: limitations: The authors acknowledge several key limitations of the Rainbow Teaming approach in Appendix A. Generally speaking, they are sufficient, but are not comprehensive or clear as I note in describing weaknesses and questions.
paper_summary: paper_summary: The paper proposes a novel method Rainbow Teaming for the automatic generation of diverse adversarial prompts aimed at large language models (LLMs). The goal is to identify and enhance the robustness of LLMs to various user inputs, which is crucial as these models are increasingly used in safety-critical environments.
strength_weakness: strength_weakness: Lack of baseline comparisons in the safety evaluation for LLMs.
questions: questions: There are a few details that I'm not certain about:

1. What precisely does the Mutation Operator do? Does it mean using the LLM to generate the necessary risk category directly or in a few-shot fashion?

2. How is the Mutator LLM trained? What are the differences compared to a normal LLM?
limitations: limitations: I'm not sure if the authors plan to release their code or model checkpoint to facilitate further advancements in this field. Automatic red teaming is indeed crucial, but some methods can be challenging to reproduce since they often involve large amount of engineering work, which somehow is more likely an engineering work rather than a research study.

Furthermore, while this paper primarily focuses on diverse risks, it would be beneficial to include a comparison with existing red teaming approaches, as they can also be utilized to evaluate and enhance the safety of LLMs.
paper_summary: paper_summary: The authors present Rainbow Teaming, a structured approach to automated redteaming of large language models. Based on a user-specified set of strategies and risk categories, Rainbow Teaming uses a mutator LLM to rewrite several variations of existing prompts, then compares the resulting outputs from the target model with a judge LLM against the existing prompt. If a more effective prompt is found, it replaces the existing prompt in the "archive" of prompts found so far. The authors conduct experiments redteaming various open-weight models such as Llama and Mistral, showing that their method achieves a high success rate on various risk categories. They also explore the use of the generated prompts in supervised finetuning, showing that robustness to Rainbow Teaming can be improved by training against Rainbow Teaming.
strength_weakness: strength_weakness: * Contribution and novelty seems very marginal. The difference from methods such as PAIR and TAP appears to come down to 1) presenting the attack/mutate LLM with high level categories instead of specific behaviors and 2) specifying several concrete strategies instead of relying on the attack/mutate LLM to come up with them on the spot
* Lack of comparisons to prior work. The authors offer various criticisms of PAIR, TAP, and the approaches studied by Perez et al. but do not show any evidence that their method outperforms or finds substantively different prompts from these approaches. Table 1 presents some results which I do not understand. Evaluations on common benchmarks such as AdvBench and HarmBench are missing.
* Lack of experiments on bigger models. The authors only run Rainbow Teaming against 7B models. They claim they are unable to evaluate against more powerful models such as GPT and Claude because of legal constraints, but there are plenty of larger and more powerful models for which this is not a concern, such as Llama-3 70B and many others.
* Fig 4 shows that Rainbow Teaming only improves performance by about 10% beyond the simple baseline of sampling lots of candidates from scratch, suggesting that the whole evolutionary framing of elites, mutations, etc is not so critical.
* The experiments on improving robustness with training on Rainbow Teaming prompts in Sec. 5 are not convincing. The authors generate 15 sets of prompts targeted against Llama-2 7B, train on 12, and then show near-perfect performance on the held out 3 sets. How different are these 3 sets from the 12 training sets if they are generated with the same algorithm? But when running the Rainbow Teaming pipeline against the fine-tuned model, they still find a nearly 40% attack success rate which is not robust at all. And this is in spite of the fact that they do not appear to be using any holdout behaviors for validation.
questions: questions: * The authors criticize the methods in Perez et al. as costly, but use up to 2000 iterations and several thousand LLM calls in their proposed approach. How much more costly is Perez et al. than Rainbow Teaming?
* Prior work has found that Llama-2 7b chat is significantly more robust if the default system message released with the model is used. Do the experiments here use that system message?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces RAINBOW TEAMING, an approach for generating diverse adversarial prompts to test and improve the robustness of LLMs) The method uses quality-diversity search to produce a wide range of effective adversarial prompts across different categories. The authors demonstrate its effectiveness on state-of-the-art models like Llama 2 and Llama 3.
strength_weakness: strength_weakness: 1. The study focuses primarily on Llama 2 and Llama 3 models, citing licensing constraints for not including other major models like GPT-4 or Claude. This focus limits the generalizability of the findings. It would have been more convincing to see results across a wider range of models from different providers, especially given the importance of the topic. 

2. While the authors report high inter-evaluator agreement between GPT-4, Llama Guard, and humans on a small sample, the study relies heavily on automated metrics for evaluating the safety of responses. 

3. While the paper mentions that fine-tuning with RAINBOW TEAMING-generated data improves model robustness, it lacks a detailed analysis of potential effects on the model's general performance or capabilities.
questions: questions: See the above.
limitations: limitations: Yes
paper_summary: paper_summary: The paper addresses automatic red teaming of large language models through open-ended generation of jailbreaks. The key components of their methodology are 1) a categorization of different jailbreak categories to create a diverse archive of possible jailbreaks, 2) a strategy to evolve and mutate jailbreaks, 3) and a selection process to keep the jailbreaks with the highest quality.
strength_weakness: strength_weakness: W1: Despite the extensive lists of hyperparameters and explanations, the results are not realistically reproducible without the dataset of generated jailbreaks or the trained model. As adversarial robustness has shown to be brittle in the past, I strongly recommend any safety paper to make it as easy as possible to verify their defense. A considerable amount of powerful open-source large language models is available and there is no particular reason why releasing the trained robust model would provide any additional safety concern to the community. (Note that this is more of a personal concern and will not influence my score as I understand that it might have not been possible for the authors to release the model / code / data)

W2: The robustness evaluation of the the model trained with rainbow teaming data is insufficient. I would argue that safety assessments can never be “fixed” and need to be adaptive and designed for the model at hand. The authors performed a train test split to evaluate the robustness of the model, which is non-adaptive. At least any evaluation with one of the many adversarial attacks proposed for LLMs in the last year would have put the robustness into better perspective. Most defenses proposed in the robustness domain have later been shown to be ineffective and “offline” adversarial training (generating the attacks prior to training) does not yield robustness for image models against stronger attacks in my own experiments. Thus, I am a bit sceptic if rainbow teaming actually improves worst-case adversarial robustness.

W3: Evaluations are limited to the Llama series of models. Experiments on non-aligned models or models trained with less safety fine-tuning would have been interesting. E.g., "How does rainbow-teaming compare to standard model alignment?". (relatively minor concern)

I am very likely to raise my score if the authors provide more results regarding the worst-case adversarial robustness of models trained with rainbow teaming or provide a good reason why this is not necessary / out of scope.
questions: questions: Could the authors conduct worst-case attacks against the model trained with rainbow teaming i.e., [1]?

[1]  Andriushchenko et al., “Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks”, 2024
limitations: limitations: The authors provide an extensive list of limitations.

Comments: title: Response by Authors (2/2)
Response by Authors (1/2)
Response to Reviewer hknM
A note on visibility of Reviewer w9N7's comments
Message to Reviewer w9N7
Further clarifications to Reviewer sYeN comments
A reply to Reviewer sYeN's full response
Full response
Responce to Reviewer sYeN's comment
A message to reviewer DdX9
A message to the reviewer sYeN
A message to the reviewer
Thank you for a quick reply
Thank you for acknowledging
Concerns addressed
comment: ### **Questions**

> Q1. significance of the initial adversarial prompt

When adding a prompt to the archive, we cache its elicited Target response. When sampling a parent prompt in the Selection phase (Figure 2), the cached response is ignored, and Mutation uses only the prompt itself and the candidate prompt descriptor. During Evaluation, the Judge compares the (newly generated) response elicited by the candidate prompt to the cached response to the prompt already in the archive at that position. We never reprompt the Target on prompts already in the archive. The Update happens if the response to the candidate prompt is judged more harmful than the cached response in at least 3 out of 4 comparisons by the Judge.

This process is identical for the initial prompts that are generated at random and placed in the archive before the MAP-Elites search process and for those observed later at any point.

> Q2. motivation behind QA and Cybersec

We aim to include other areas that are substantially different than safety.

Question Answering is orthogonal and complementary to safety. Specifically, we aim to diagnose features of the target LLM typically acquired during pre-training (general knowledge), whereas safety and alignment are performed during post-training stage either through SFT or RLHF.

The cybersecurity domain’s main difference from that of safety is that the response from the LLM is around unsafe code rather than text only. It also featured established open-sourced evaluators (CyberSecEval) and risk categories (MITRE) we could rely on to assess the performance of our method.

> Q3. On computational requirements

The computational intensiveness is indeed due to MAP-Elites component of the approach. There are no additional bottlenecks. 

We stressed that Rainbow Teaming requires extensive computation resources to distance it from a line of work that aims to find jailbreaks with minimal compute or time. While that line of work is valuable on its own right, here we concentrate on generating a large collection of diverse and high-quality adversarial prompts, which is simply a different use case.

> Q4. Mutation prompts

*It was decided* that, while the Judge and Evaluation prompts are essential to reproducibility and are mostly positive in nature since they focus on harm appraisal, the Mutation prompts provide a non-negligible risk to the community. This is because they turn the LLM into an attack generating program, which carries a potential for misuse.

As a result, and because we are fully aligned with the goal of democratizing AI safety research, the authors opted to describe the mutator in detail instead. We note that there is no “secret sauce” to the prompt — it is merely a few shot prompt asking the model to change the parent prompt to match the prescribed Risk Category or Attack Style. Any reimplementation effort is also likely to use a newer model than we did (a variant of Llama 2-chat 70B), which would require rewriting the prompt for maximum performance. 

We once again thank the reviewer for their extensive review under such a tight deadline. We wish we had more time to engage with them, but hope we provided a satisfactory answer to their questions.
We are very grateful to Reviewer E172 for their extremely detailed review. We are pleased that they found our contributions significant and its presentation exceptional. It is also great that the reviewer appreciated the thoroughness of our empirical results.

We address your questions and concerns below. Please let us know if you have any recommendations for further improving the paper and strengthening your support.

### **Longitudinal Practicality**

As mentioned in App. A, we agree that a main limiting factor of our method is the fixed nature of the archive. We shared a new result in the Response to all Reviewers showing robustness to out-of-distribution attacks from another method (PAIR), but indeed future jailbreak methods might nullify the robustness imparted by the current round of fine-tuning. 

However, the versatility of our method means that developers can quickly react to newly discovered attacks by simply expanding the list of Attack Styles and rerunning Rainbow Teaming to get new fine-tuning data. 

### **Static Set of Attack Styles and Risk Categories**

We agree that having a dynamic set of attacks and risk categories would be a valuable addition to the paper. Rainbow Teaming can be extended so that LLMs define archive categories automatically, either before the search process, or more interestingly, throughout the search process as the archive gets filled over time.

We have decided to leave this direction for future work, as we believe it warrants more extensive investigation than can be accommodated within this manuscript. 

Also, two results demonstrate the generality of our method, even to categories beyond those in the paper. The first is that our method remained effective when targeting JailbreakBench behaviors instead of the original 10 risk categories, despite the fact that it is not designed to elicit very specific harmful behaviors. The second is that it was equally effective across all safety, cybersec and Q&A, which is a much broader change than just providing new categories.

### **Minor Weaknesses**

- **Diversity Metrics.**
    - We thank the reviewer for providing a reference for additional metrics of diversity. We agree that more metrics of diversity would strengthen the paper and we can commit to include more in the final version of the paper.
    - Additionally, we use BERTScore in the ablation study for mutation filtering (see Table 5).
- **Model Choice.**
    - We made every effort to include the most capable, recent and safest open models. For example, Llama 3 was released on April 18, less than a month before the deadline. We also show transfer results to GPT-4o in our response to all reviewers.
    - We strongly agree with the reviewer that here, in particular, the set of 8 models that we use are mainly vehicles for demonstrating our methodology.
We are glad we could address all of the concerns of reviewer hknM and that their only remaining issue with the work is that it's not open-source yet.

Unfortunately, making the code open-source is __beyond our control at this time__. To compensate for this, we have made every effort to provide very thorough details of our approach in the paper, including an in-depth description of the algorithm, a thorough pseudocode, and the full set of hyperparameters that were used. We believe these resources should make it straightforward for others to reimplement Rainbow Teaming.
I do agree with the excellent experimental results in this paper, particularily on commercial models. However, as I have mentioned, I still have concerns on the open-source availability of such attack papers. In my opinion, due to the strong capabilities of LLMs and their random responses, jailbreaking LLMs works are much more complex than those adversarial attack works before. Therefore, it is crucial to ensure that the code and experiments are reproducible and as open source as possible. This would establish a more solid baseline and add formality to this field.

Taking all of this into consideration, I will maintain my score as borderline reject this time. I appreciate the authors' contribution and effort in this paper and during this rebuttal.
Sorry for the oversight.
I changed the visibility.
We noticed that reviewer w9N7’s past two comments, which are partly addressed to other reviewers, are only visible to authors and AC, but not to other reviewers.

Could we ask reviewer w9N7 to edit their comments’ visibility so that it is visible to other reviewers (by ticking the necessary box below the comment area when editing a comment)?
We thank their reviewer for their continued engagement, and for their insightful and supportive comments. We are glad they appreciate our reproducibility efforts, including the dataset release. 

We also thank them for taking the time to read and comment on other reviews, therefore encouraging constructive discussion around our paper. We believe this sets a positive example for how to engage during the author-reviewer discussion period.
### **Comparison with No Stepping Stones Baseline**

> basically unchanged performance of their method when the iterative "evolution" component is ablated
> 

The performance between our method and the baselines is **NOT unchanged**. We have highlighted their difference, several times. But it seems the reviewer chooses to ignore this.

> Llama-2 7b chat is known to be a *weak* model. Multiple teams of researchers have found simple methods that reliably achieve *100%* ASR.
> 

This is factually incorrect. Llama 2 and Llama 3 chat variants are arguably the safest open-source models. Even on papers the reviewer cites themselves [1-3], Llama models are still the hardest to jailbreak (see below for more information on this).

Regardless, the reviewer has repeated their unsubstantiated claim on this topic.

> In context, a 90% or 80% ASR after an arbitrary cutoff of 2000 steps is simply not meaningful when the ASR curve for the baseline method in Figure 4 is continuing to improve while the proposed method has plateaued.
> 

Our method has been significantly better than the Baseline from step 0 to step 2000. Even if, hypothetically, they both achieve the same ASR at step 4000, our method would still be very clearly outperforming this baseline with a significant margin due to its sample efficiency throughout the entire search process.

Also, we invented this baseline ourselves and it has substantial algorithmic complexity. The fact that it also works quite well is not a valid criticism to our main approach.

We have reiterated these points to reviewer several times. We are perplexed by the reviewer’s continued refusal to engage with our arguments and by their dismissal of numerical results in Figure 4, which are clearly statistically significant. 

### **Evaluations of Robustness**

> *totally unconventional evaluation which reuses test cases optimized against a completely different model*
> 

We take prompts optimized against Llama 2-chat and apply them to Llama 2-chat + Adversarial Fine-Tuning. Showing that the fine-tuned model is robust to attacks that jailbroke it before SFT is not an “unconventional evaluation”. It’s simply a matter of splitting data in a train and test set and showing improved test set performance post training (i.e. fine-tuning). It is the lowest bar for improved robustness and, had we not ran this evaluation, reviewers would certainly have requested it, and justly so.

> *They also re-run their attack from scratch on the fine-tuned model, but without holding out any behaviors and bury this result much deeper in the text.*
> 

This result is anything but buried, and we find such claims from the reviewer concerning. We dedicate Figure 5 and a full paragraph (Lines 253-259) concluding the section titled “*Enhancing Robustness with Synthetic Data*” to this result. Not only did we use a rainbow-colored curve in the plot to attract attention, we also literally bolded the relevant result information in the text on line 255.

> *The only other attack evaluated against the fine-tuned model is PAIR, which is a weak attack whose ASR is just not informative of model robustness.*
> 

The reviewer claimed the above and proposed GCG [1], TAP [2] and HarmBench [3]. These are inadequate:

1. GCG is a white-box token-based attack, whereas we study black-box prompt-based attacks. Please refer to Related Works (Section 7) and to Appendix D.1 - Token-Level Attacks for a discussion on this topic.
2. TAP has a self-reported ASR of 4% against Llama 2-chat 7B (See Table 1 of Mehrotra et al. [2]). Given the reviewer considered the 14% ASR of our implementation of PAIR against the same Llama 2-Chat model as “paltry”, we doubt they would have been satisfied with TAP.
3. HarmBench is an evaluation framework, not an attack method. Also, as stated in our original rebuttal, prompts from HarmBench are included in JailbreakBench, which is what we use when attacking models with PAIR in Table 2

Furthermore, we would like to remind the reviewer that our paper is not about defending LLMs. It is about discovering vulnerabilities. We include a section on enhancing robustness as a first demonstration of the utility of Rainbow Teaming datasets but, as stated in the paper, “*we leave the investigation of adversarial fine-tuning strategies to future work*”.

[1] Zou et al. *Universal and Transferable Adversarial Attacks on Aligned Language Models.* 2023

[2] Mehortra et al. *Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. 2023*

[3] Mazeika et al., *HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.* 2024

[4] Chao et al. *JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models*, 2024.
The authors have not addressed my main concerns so I will just summarize them here again for the other reviewers and meta-reviewers:

1. The presented method is not considerably stronger than prior attacks [1, 2], and the proposed understanding of how it works is not convincing, as shown by the basically unchanged performance of their method when the iterative "evolution" component is ablated. Llama-2 7b chat is known to be a _weak_ model. Multiple teams of researchers have found simple methods that reliably achieve _100%_ ASR. Thus we know that the set of strings which can successfully trigger harmful behaviors is very, very large.  In context, a 90% or 80% ASR after an arbitrary cutoff of 2000 steps is simply not meaningful when the ASR curve for the baseline method in Figure 4 is continuing to improve while the proposed method has plateaued.

2. Robustness evaluations of the fine-tuned model are unsound and incomplete. The authors lead with a bolded claim that fine-tuning reduces ASR from 92% to 0.3% but this is a totally unconventional evaluation which reuses test cases optimized against a completely different model. They also re-run their attack from scratch on the fine-tuned model, but without holding out any behaviors and bury this result much deeper in the text. The only other attack evaluated against the fine-tuned model is PAIR, which is a *weak* attack whose ASR is just not informative of model robustness. There are a wide variety of popular open-source attacks (from papers with many dozens of citations) the authors could have easily run against their new model because it is literally using the Llama-2 7b architecture + config which everyone evaluates against: https://github.com/llm-attacks/llm-attacks, https://github.com/RICommunity/TAP, https://github.com/centerforaisafety/HarmBench. The authors chose to ignore these works in their evaluation.

I will maintain my score.
We thank the reviewer for further questions. Below, we address all the issues you raise in your response.

### **On No Stepping Stones Baseline**

We agree that using stepping stones is a core aspect of our method, as well as the MAP-Elites style algorithms on which we base Rainbow Teaming. **However, our method performs significantly better than the No Stepping Stones baseline — their performance is by no means “largely the same”**.

In the safety domain, Rainbow Teaming achieves 92% attack success rate (ASR) on the Llama 2-chat model, whereas the No Stepping Stones baseline achieves only 83% ASR. **This difference is significant.**

In the domain of question answering, Rainbow Teaming outperforms No Stepping Stones baselines significantly on mean fitness of the archive, coverage, as well as the diversity of the archive (as evaluated using self-BLEU). See the table below for more information, which is the Table 3 from the paper. **This difference is, again, significant.**

| Method | Mean Fitness ↑ | Coverage ↑ | Self-BLEU ↓ |
| --- | --- | --- | --- |
| Rainbow Teaming | 0.91 ± 0.01 | 0.97 ± 0.01 | 0.50 ± 0.02 |
| Baseline (No Stepping Stones) | 0.79 ± 0.01 | 0.90 ± 0.01 | 0.60 ± 0.01 |

If we proposed the baseline from Figure 4 as our main method, it would still have been novel, given the aforementioned algorithmic innovations. Also no prior method tackles adversarial prompt generation from a quality-diversity (QD) perspective or achieves an ASR close to what we did on various models.

### **Enhancing Adversarial Robustness**

We demonstrated, convincingly, that performing supervised fine-tuning (SFT) on Rainbow Teaming-generated data significantly improves model’s safety robustness. Specifically, we showed that after performing additional SFT 

1. The ASR of PAIR [1] **drops from 14% to 0%**, i.e., *PAIR can no longer jailbreak Llama 2 model, at all* (see the newly provided PDF attached to the response to all reviewers).
2. The ASR on previously unseen Rainbow Teaming generated archive **drops from 92% to 0.03%**, i.e., *previous jailbreaks created by our method don’t work, at all* (see Table 2).
3. The ASR of re-applying Rainbow Teaming from scratch results in a **drops from 92% to 39%**, *i.e., our method is struggling to jailbreak this new version of Llama 2* (see Figure 5)*.*
4. The safety reward model scores **increase from 0.883 to 0.897,** i.e., *even on unrelated Anthropic Harmless dataset the Llama 2-chat mode becomes much safer* (see Table 2).

We strongly believe that the results above are more than conclusive.

### **Stronger attacks against the adversarial fine-tuned model**

> *PAIR is a very weak attack, as evidenced by it's paltry 14% ASR on the not-yet-fine-tuned model….* the authors need to evaluate on stronger attacks.
> 

There are no stronger black-box prompt-based attack in the literature than PAIR [1]. If the reviewer wants us to use more powerful jailbreaking techniques, they should clearly state which ones.

### **Llama-2 chat being non-robust**

> *Llama-2 chat is a very non-robust model*
> 

This statement is unfounded. Every prior work which evaluated on Llama-2 chat struggled with it in comparison with other methods. For instance, the original PAIR paper achieves 88% against Vicuna, 48% against GPT-4, but only **4% against Llama 2-chat** [1]. Similar differences can be seen in other works, such as [2] or [3].

Unless the reviewer can provide evidence to the contrary, we are convinced that there are no open-sourced models that are more robust than the Llama 2 and 3 series.

### **Attacking the fine-tuned model with Rainbow Teaming**

> *the authors need to run their own attack on their fine-tuned model.*
> 

**This is a key result of our original submission.** We present it in Figure 5 with the colorful rainbow curve, given additional results in Figure 13, and mention it in the conclusion. Perhaps the reviewer missed Figure 5, or they mean something else and wish to clarify?

### **Conclusion**

We hope that our detailed response addresses the reservations you have. If you have further questions or concerns, please let us know. If not, we would appreciate if the reviewer could adjust their scores to reflect this.


[1] Chao et al, Jailbreaking Black Box Large Language Models in Twenty Queries, 2023

[2] Paulus et al, AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs, 2024

[3] Zou et al. Universal and Transferable Adversarial Attacks on Aligned Language Models. 2023
I don't believe I have misunderstood your method or experiments. "Stepping stones" is a core part of the method. Without it, the evolutionary framing doesn't make much sense since the method then boils down to rejection sampling from the 1-step mutator distribution until better candidates are found. And yet without stepping stones performance is largely the same, albeit optimization may be a bit less efficient.

My main concern, which remains unaddressed by the response, is that the experiments on improving robustness are weak and unconvincing. Even though the proposed redteaming method is not the most novel in its approach-- prior work has examined diversity constraints such as the Trojan Detection Challenge at NeurIPS 2023 (https://trojandetection.ai/tracks, as well as systematic, multi-strategy paraphrasing (https://chats-lab.github.io/persuasive_jailbreaker/)-- there would still be a lot of value in developing a method that can be used for adversarial training. But the authors have no demonstrated this convincingly.

Evaluating the fine-tuned model on prompts optimized against the prior, not-yet-fine-tuned model doesn't make any sense to me. The issue here isn't attack adaptivity, it's that the attack is not even targeted to the model under evaluation. Echoing Reviewer DdX9, I agree that the authors need to evaluate on stronger attacks. PAIR is a very weak attack, as evidenced by it's paltry 14% ASR on the not-yet-fine-tuned model (Llama-2 chat) which itself is a very non-robust model. At the very least, the authors need to run _their own attack_ on their fine-tuned model.
We thank the reviewer for their question.

You are correct in your understanding. The No Stepping Stones baseline does not sample from the archive; that is, the step x ~ G is omitted. Instead, at each iteration, a new seed prompt is generated from scratch in a zero-shot manner based on a given Risk Category. We then apply a single mutation to it according to a given Attack Style. All other aspects between this baseline and our main method remain identical. Additionally, we've included further baseline results in the updated PDF shared with all reviewers.

If you have any further questions or concerns, please let us know.
I will respond to the other points, but first to clarify on the stepping stones baselines: I'm going off your description in lines 212-214: "ignores past solutions in the archive and generates new prompts on the risk category, before applying the attack style mutation". Looking at the pseudocode in Alg 2, it sounds like the change being made is that x is always set to a randomly generated seed prompt in each iteration, and that sampling from the archive (x ~ G) no longer takes place. Is this correct?
This is a gentle reminder that **tomorrow, August 13**, is the final day of the Reviewer-Author discussion period.

We have carefully addressed all the questions and concerns the reviewer raised, including on the automatic metrics for evaluating ASR and enhancing adversarial robustness with Rainbow Teaming-generated data. Additionally, we have provided a joint response to all reviewers, highlighting novel experimental results, such as transfer performance to GPT-4o and comparisons with new baselines. We will also include a dataset of adversarial prompts in the final version of the paper.

We also note that **Reviewer w9N7** has also, kindly, provided their supportive opinion on the usage of automated metrics and novelty of the method. Please see their comment below.

If you have any further questions or suggestions, please let us know.

**If our responses have successfully addressed your questions and concerns, we kindly ask you to consider updating your score accordingly.**

Thank you very much.
This is a gentle reminder that **tomorrow, August 13**, is the final day of the Reviewer-Author discussion period.

We have carefully addressed all the questions and concerns the reviewer raised, including on the comparisons with baselines and experiments with bigger models and various system prompts. Additionally, we have provided a joint response to all reviewers, highlighting novel experimental results, such as transfer performance to GPT-4o and comparisons with new baselines. We will also include a dataset of adversarial prompts in the final version of the paper.

If you have any further questions or suggestions, please let us know.

**If our responses have successfully addressed your questions and concerns, we kindly ask you to consider updating your score accordingly.**

Thank you very much.
This is a gentle reminder that **tomorrow, August 13**, is the final day of the Reviewer-Author discussion period.

We have carefully addressed all the questions and concerns the reviewer raised, including on the comparisons with baselines and the Mutator LLM. Additionally, we have provided a joint response to all reviewers, highlighting novel experimental results, such as transfer performance to GPT-4o and comparisons with new baselines. We will also include a dataset of adversarial prompts in the final version of the paper.

If you have any further questions or suggestions, please let us know.

**If our responses have successfully addressed your concerns, we kindly ask you to consider updating your score accordingly.**

Thank you very much.
I highly appreciate the effort in making the paper reproducible, which I believe is the most relevant practical limitation of the current version (even if a lot of information is available, reproducing the results will be infeasible for most researchers). However, open-sourcing the dataset would enable future research on attacks and defenses. 

I will consider increasing my score and would like to give my opinion on some of the stated weaknesses of other reviewers:

**Limited experiments / Bigger models.** The authors provide relevant experiments for safety fine-tuned and models trained without safety alignment and perform their attack on closed-source models. If we ask for more experiments as the bar to accept papers, we will not have any papers from academia on red-teaming LLMs/VLMs in the future.

**Automated metrics.** An investigation of the ASR is provided in the paper. I think its unreasonable for every new red teaming paper to perform manual evaluations or do the same investigations as a lot of prior work on the human alignment of LLMs as a judge. 

**Novelty.** I believe this to be a very bad metric to judge a paper. "On Adaptive Attacks to Adversarial Example Defenses" was initially rejected for low novelty but had a big impact on the field and now has nearly 1000 citations. The kind of red-teaming studies performed in this paper are important to guide new approaches to defend and attack LLMs.
We thank the reviewer for their blazing fast reply and for updating their score! 

While out of scope, we will consider the reviewer's suggestion to expand the set of evaluations to white box or token based attacks such as GCG [1] or Andriushchenko et al. [2] for the final paper. 

Regarding reproducibility, note that we will also be releasing a full archive of prompts produced by Rainbow Teaming in the final version of the paper. These prompts will provide further insight into the kind of jailbreaks discovered by Rainbow Teaming. They will also help reproducibility efforts, for instance by allowing researchers to warm-start a Rainbow Teaming run by initializing their archive with our prompts. 

Finally, if the reviewer has suggestions on how we can improve the paper and our score even further, we ask that they please let us know.

[1] Zou et al. *Universal and Transferable Adversarial Attacks on Aligned Language Models*. 2023

[2] Andriushchenko et al., *Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks*, 2024
Thank you for acknowledging our rebuttal. Please let us know if you have remaining questions or concerns. If not, we ask the reviewer to please update their score to reflect this. 

Note that after further consideration, we will also include a full archive of prompts produced by Rainbow Teaming in the final submission, to help guide reproduction efforts.
Thanks for the response. And I will consider my score.
I thank the authors for their response. My concerns are addressed and I will increase my score accordingly.
You are correct that the adaptive attack proposed by Andriushchenko is out of scope for the review and I apologize for bringing it up.
Nevertheless, I think it would provide an interesting additional experiment for the final paper :-)