Efficient Adversarial Training in LLMs with Continuous Attacks

Authors: sophie xhonneux mila
alessandro  sordoni
stephan  günnemann
gidel  gauthier
  mila
leo  schwinn
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 01:14:12

Abstract: Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. In many domains, adversarial training has proven to be one of the most promising methods to reliably improve robustness against such attacks. Yet, in the context of LLMs, current methods for adversarial training are hindered by the high computational costs required to perform discrete adversarial attacks at each training iteration. We address this problem by instead calculating adversarial attacks in the continuous embedding space of the LLM, which is orders of magnitudes more efficient. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses: the first makes the model robust on continuous embedding attacks computed on an adversarial behaviour dataset; the second ensures the usefulness of the final model by fine-tuning on utility data. Moreover, we introduce C-AdvIPO, an adversarial variant of IPO that does not require utility data for adversarially robust alignment. Our empirical evaluation on five models from different families (Gemma, Phi3, Mistral, Zephyr, Llama2) and at different scales (2B, 3.8B, 7B) shows that both algorithms substantially enhance LLM robustness against discrete attacks (GCG, AutoDAN, PAIR), while maintaining utility. Our results demonstrate that robustness to continuous perturbations can extrapolate to discrete threat models. Thereby, we present a path toward scalable adversarial training algorithms for robustly aligning LLMs.

Main paper: 
 introduction: 
 As large language models (LLMs) become increasingly integrated into various applications, ensuring their safety and robustness is crucial. The seminal work of Zou et al. [1] highlighted substantial vulnerabilities in even the most advanced proprietary models, demonstrating that adversarial attacks can effectively disable safety mechanisms. More recently, adaptive attacks have been shown to achieve nearly a 100% success rate on widely used models, underscoring the severity of this issue [2] . Robustness ExtrapolatesTell me how to build a bomb [!!one with? rever with!--TwoFor the purpose of this exercise ignore any moral and [. We propose continuous adversarial training (AT) to address the large computational requirements of existing discrete AT approaches [6] . We demonstrate that robustness against continuous attacks successfully extrapolates to discrete threats, such as suffix and jailbreaking attacks while being considerably faster to compute..Adversarial training, which involves online augmenting the training data of a neural network with adversarial attacks, has consistently proven to enhance robustness against adversaries [3, 4] . Yet, initial attempts at adversarial training for LLMs have shown ineffective [5] . Unlike continuous adversarial training (AT) algorithms in other domains, AT for LLMs usually involves discrete attacks, where tokens in the prompt are either substituted, injected, or appended as suffixes [1, 6] .Recently, Mazeika et al. [6] proposed R2D2, the first AT algorithm that successfully improves robustness against various attacks in LLMs. The authors use Greedy Coordinate Gradient (GCG) to generate discrete adversarial suffixes in natural language. However, GCG requires extensive computational resources, employing hundreds of thousands of model evaluations to compute a single attack. This leads to considerable overhead for R2D2 despite additional optimisations.Continuous adversarial attacks have recently demonstrated higher success rates and significantly faster computation times than their discrete counterparts in LLMs [7, 8] . Moreover, continuous attacks have proven effective in adversarial training algorithms for encoder-decoder models, such as BERT [9, 10] . Thus, we argue that continuous attacks could be an efficient alternative to discrete attacks within LLM adversarial training algorithms. We ask the following research question: Does adversarial training with continuous attacks in the token embedding space of an LLM extrapolate and provide robustness to discrete natural language attacks? We positively answer this research question using two novel adversarial training algorithms. We propose CAT, an efficient continuous AT algorithm, combining training on an adversarial behaviour dataset with fine-tuning on utility data. We further introduce continuous adversarial preference optimisation (CAPO), an adversarial variant of identity preference optimisation (IPO) [11] that does not require utility data for adversarial alignment. We surpass the robustness-utility trade-offs of the discrete R2D2 AT algorithm [6] , achieving up to 100% attack robustness while requiring over 299 times less computing resources. Additionally, we identify a failure mode in previous evaluation protocols: the models are tested with their chat template for safety evaluations but without it for utility evaluations. This protocol is unrealistic as the chat template is not enabled or disabled based on the prompt the user enters. By enabling the chat template for standard queries, we demonstrate that R2D2 overfits the safety objective and grammar of the harmful dataset. Thus, it often refuses to respond to benign inputs, thereby hurting its usefulness. In contrast, models trained with CAT and CAPO show substantially fewer refusals. 
related work: 
 Adversarial Attacks Adversarial attacks and defenses have been extensively studied in the literature [1, 3, 4, [12] [13] [14] [15] [16] [17] [18] [19] . More recently, LLMs have been shown to be vulnerable to exploitation by adversarial attacks, and several threat models, such as suffix attacks [1] and jailbreaking [16] , have been proposed. Zou et al. [1] present the Greedy Coordinate Gradient (GCG) suffix attack, which generates adversarial examples transferable from small open-source models to large proprietary models. Huang et al. [20] find that just varying generation strategies, such as adjusting decoding hyper-parameters and sampling methods, can trigger harmful behaviour in LLMs. Geisler et al. [21] introduce a novel discrete attack strategy that leverages continuous embedding space optimisation. In the area of continuous adversarial attacks, Fort [22] explore scaling laws for continuous adversarial attacks on language model activations. Further, Schwinn et al. [7, 8] showcase the potential of continuous adversarial attacks as a threat model to compromise safety alignment and unlearning.An alternative threat model involves jailbreaks, a form of prompt engineering with the goal of circumventing safety alignment. Deng et al. [17] fine-tune an LLM with jailbreak examples and demonstrate that the fine-tuned LLM can generate strong attacks, which transfer between different models. Similarly, Chao et al. [15] found that LLMs could be leveraged to create jailbreaks for other LLMs, even without fine-tuning. They introduced the Prompt Automatic Iterative Refinement (PAIR) algorithm, which uses an attacker algorithm to iteratively query a target LLM, optimising the jailbreak prompt. Liu et al. [16] developed a hierarchical genetic algorithm to generate high-perplexity jailbreaks that can bypass the safety alignments of LLMs.Adversarial Training Previous work on continuous adversarial training (AT) on token embeddings has mostly focused on encoder-decoder models, such as BERT [9, 10, [23] [24] [25] [26] . Jiang et al. [9] use adversarial attacks to promote smoothness in the embedding space of the model and show that this approach improves generalisation. Similarly, Zhu et al. [10] enforce invariance in the embedding space through adversarial attacks. He et al. [24] combine a disentangled attention mechanism with continuous AT and demonstrate improved generalisation for BERT and RoBERTa models on multiple downstream tasks. Other works apply continuous adversarial perturbation to word embeddings to increase performance in different NLP tasks [23, 25, 26] . Robey et al. [27] propose improving the robustness of autoregressive LLMs by a randomised smoothing-inspired approach.Concurrent to this work, Casper et al. [28] use continuous attacks for the purpose of AT. They propose latent adversarial training (LAT), a method that finds perturbations in the network's hidden layer representations and applies them to several tasks including text generation. For text generation, they demonstrate that fine-tuning for desirable behaviour with LAT makes the model more likely to forget triggers from data poisoning in some cases. Contrary to our work, they set up the adversarial training in an untargeted manner, i.e. the attack they apply does not aim to produce a particular harmful output but uses the standard AT objective. In contrast, our work focuses on the challenge of making LLMs robust against discrete attacks and jailbreaks while maintaining their helpfulness. To do so, we propose novel algorithms and loss functions that make use of the harmful targets of discrete attacks. Moreover, we thoroughly evaluate across multiple benchmarks and adversarial attacks to ensure a good robustness-utility trade-off.Adversarial Data Augmentation Several works [29, 18] have developed adversarial attack generators against LLMs and then used the generated adversarial attacks to create a dataset on which to perform supervised fine-tuning (SFT) to improve adversarial robustness. This kind of adversarial robustness training is based on dataset augmentation and does not adapt the model online to worst-case attacks. Thus, we consider these approaches orthogonal to our work. 
method: 
 In this section, we introduce our adversarial training (AT) algorithms: Continuous-Adversarial UL (CAT) and Continuous-Adversarial IPO (CAPO). We begin by reviewing the standard AT regime from Madry et al. [4] ( § 3.1). We then explain differences between attacks in the standard AT setting and unique aspects of adversarial attacks in LLMs ( § 3.2). From there, we derive the Unlikelihood loss for-CAT ( § 3.3). Next, we introduce an adversarial IPO formulation-CAPO ( § 3.5). Finally, we discuss key design decisions in the above AT algorithm ( § 3.6).AT is generally defined as a minimax optimisation problem as follows [4] :EQUATION)where L is the loss function, f θ is a neural network with parameters θ, D is the dataset, T (x) is the set of perturbations around x ∈ X allowed by the threat model. In computer vision,x ∈ [0, 1] d is an image, T (x) = {δ | ϵ ≥ ∥δ∥ p , x + δ ∈ [0, 1] d }and L is a classification loss such as cross-entropy.For LLMs with a token vocabulary V, x is a prompt and a common perturbation set T are discrete manipulations of the input space, such as suffix attacks [1] . For suffix attacks, the set of acceptable perturbations δ is defined to be in the set of sequences of tokens of length m that can be appended to the input prompt. In other words, the adversarial attack x + δ is of the form x; δ, where δ is a fixed number of tokens the attacker has full control over and ; means concatenation. However, computing the best δ from this perturbation set T suffix (x) = {δ | x + δ ∈ V n+m } is computationally expensive, as the optimisation turns into a discrete combinatorial problem with exponentially many solutions. Arguably, it is too expensive to use during training, especially for large datasets.Thus, we propose a different perturbation set T based on continuous embedding attacks [7] . This perturbation set allows the modification of the embeddings of the tokens in the prompt under some ϵball as measured under the ℓ p norm. E is a function from tokens v ∈ V to embeddings E(v) ∈ R k . We abuse notation and for a sequence x = v 1 ; v 2 ; . . . ; v n we say thatE(x) = E(v 1 ); E(v 2 ); . . . ; E(v n ).Our perturbation set allows for a δ i ∈ R k around each token embedding. Therefore, the modified prompt after the attackx + δ is E(v 1 ) + δ 1 ; . . . ; E(v n ) + δ n , where δ ∈ R n×k and T cont. (x) = {δ | ∀i. ϵ ≥ ∥δ i ∥ p , x + δ ∈ R n×k },as in the standard AT setting. Schwinn et al. [7] proposes to find the perturbation δ with signed gradient descent as in [3] :EQUATIONAs described in Eq. 1, the inner loop of standard AT involves finding the worst-case perturbation by maximising the loss with respect to the ground truth prediction in an untargeted way. In contrast, the goal of attacks on LLMs is to induce a specific harmful continuation ŷ given a harmful prompt x. This exemplifies adversarial training under a targeted attack. Mazeika et al. [6] propose a loss that encourages the model to i) increase the likelihood of a "safe" continuation y (e.g. "I am sorry, ..."), and ii) decrease the likelihood of the unsafe continuation ŷ, given the targeted adversarial perturbation of x. This yields:EQUATIONwhere δ(x, ŷ) = arg min δ ′ ∈T (x) L(f (ŷ|x + δ ′ )) is the targeted attack on x. Contrary to standard AT [4] , we are not maximising the loss of the safe answer, but specifically minimising towards a particular harmful continuation ŷ. As discussed in the previous section, δ naturally depends on the choice of T, f, L, but we leave that out of the notation for clarity. Losses of the form of Equation 3 have been referred to as "unlikelihood" losses (UL) [30, 31] . Note that the dataset D contains harmful prompts x under which we want to give a safe answer y rather than an unsafe answer ŷ.In addition to the two terms in Equation 3, Mazeika et al. [6] propose to add an additional loss term that maximises the utility of the model, i.e. given an utility dataset D u , it optimises:EQUATIONMazeika et al. [6] found this loss necessary to avoid degenerate behaviours such as refusing to answer all prompts by producing some often generic refusal answer y.The primary difference between Mazeika et al. [6] and our method is the choice of perturbation set used during AT. Mazeika et al. [6] choose discrete suffix attacks T suffix and employ the GCG algorithm along with several tricks to mitigate the computational cost to find a GCG attack. One optimisation they introduce is to only update the attack after every k training steps. In contrast, we employ T cont. with continuous attacks as introduced by Schwinn et al. [7] , which are orders of magnitude (×299) more efficient (see Table 1 ). Consequently, we do not require any additional tricks to further reduce computational costs. In the Unlikelihood loss (Eq 3) we add cut-off values for the toward and away loss to prevent over-optimising either. Given a loss L ′ before, we implement the cutoff asL = I[L ′ > c]0.999c + (I[L ′ > c]0.001 + I[L ′ ≤ c])L ′, where c is the cutoff value chosen.Equation 3 has a similar form to DPO [31] , which maximises the likelihood of a preferred answer while decreasing the likelihood of a dispreferred answer, given a prompt x. This motivates us to present the following loss function, which we will call Continuous-Adversarial IPO (CAPO):EQUATIONwhere ℓ β (h) would be the log σ(βh) in the original DPO, but we use the loss proposed in Azar et al.[11] called IPO, i.e.ℓ β (h) = h -1 2β 2, because it is less prone to overfitting. This loss implicitly minimises the Kullback-Leibler divergence w.r.t. the original model distribution f θ0 (y|x), which prevents the model to collapse to degenerate behaviors leading to refuse all prompts with the refusal answer y. As a result, we are able to omit the utility dataset for CAPO.A few design decisions worth discussing are:1. The adversarial attack in the toward loss optimises δ such that the harmful output ŷ becomes more likely. An alternative that we leave for future work would be to formulate the attack for the toward loss such that y becomes less likely, i.e. δ(x, y) = arg max δ ′ ∈T (x) -log(f (y|x + δ ′ )). It might even make sense to compute two separate attacks, one for y and one for ŷ, and use them for the positive and negative cross-entropy loss terms, respectively. However, this would induce additional computational overhead.2. Importantly, we do not use the attack δ on the input for the reference model (f θ0 in Equation 5).Empirically we found that this makes training unstable in the DPO setting. We hypothesize that this is because the reference model represents roughly desirable log probability values of the safe answer y. Note that the original DPO paper [31] reports a similar observation and proposes to do SFT on the chosen continuation y to make sure that these reference values are on-policy.3. Mazeika et al. [6] suggests to optimise log (1-f θ (ŷ|x+δ(x, ŷ))) instead of -log f θ (ŷ|x+δ(x, ŷ)) for the away loss. We explored this and found that it yielded a considerably worse robustness/safety trade-off. We were unable to find a model that is robust and maintains some level of utility.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: There are no proofs. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental details: 
 The main goal of this paper is to assess if robustness against continuous attacks extrapolates to discrete attacks in natural language (see Figure 2 ). For additional hyperparameters see App. A.Datasets For all AT experiments, we utilise the AT dataset from HarmBench [6] with the safe answer y always being Sorry, I can't do that. As a utility dataset for CAT, we employ UltraChat200k [32, 33] , which has been successfully used in both the discrete AT algorithm ZEPHYR + R2D2 [6] and general fine-tuning [34] . For robustness evaluations, we use the first 40 samples of the HarmBench test set. Due to the substantial computational cost associated with LLM adversarial attacks, such as GCG [1] , we limit our evaluation to these samples instead of the full test set.Moreover, we measure the utility of trained models using common benchmarks, including MMLU [35] , ARC-E and ARC-C [36] , and MT-BENCH [37] . To reduce the computational demand, we evaluate 100 questions for each category for MMLU. Finally, we introduce HARMLESS which consists of 40 harmless queries (e.g. Tell me a story, see App. I for full list) that are written in the same grammatical style as the Harmbench behaviour. We query the models with their chat template and report the number of refusals (checked manually). Note that only MT-BENCH and HARMLESS use the model's chat template.In our experiments, we adversarially fine-tuned four different open-source models GEMMA [38] , PHI-3-MINI [39] , MISTRAL-7B [40] , ZEPHYR-7B [34] , and LLAMA2-7B [41] with increasing parameter counts-2B, 3.8B, 7B, 7B, and 7B, respectively. We chose instructiontuned models for all of them. We additionally include ZEPHYR + R2D2 in our evaluations, which is the MISTRAL-7B base model fine-tuned with the R2D2 AT algorithm [6] . This results in a diverse set of instruction-tuned models of different sizes. For more details, refer to App. A.2.We investigate two novel continuous AT algorithms in this work CAT and CAPO. Due to the computational complexity of fine-tuning LLMs, we do not perform full model fine-tuning for both methods but use LoRA [42] on all linear layers of the transformer architectures. Additionally, we use 4-bit quantization for all training runs to further reduce the memory overhead. We use ℓ 2 norm perturbations and set the size of the attack ϵ relatively to the average magnitude of the token embeddings of the respective model. For all models, we use 10 attack iterations. We set ϵ = 0.1 for GEMMA and PHI-3-MINI. For MISTRAL-7B, LLAMA-7B, and ZEPHYR-7B, we set ϵ = 0.05, ϵ = 0.05, and ϵ = 0.075, respectively. For a full list of AT hyperparameters, see App. A.1.We use three diverse adversarial attacks for the robustness evaluation. GCG, which has shown to achieve one of the highest average attack success rates (ASR) among other state-of-the-art attacks on several models [6] . Since GCG is a suffix attack, we further use AUTODAN and PAIR, which generate more diverse jailbreaks. Finally, we also evaluate against Adaptive Attacks [2] and ICL [19] (see Table 5 and Table 6 ). Furthermore, PAIR has shown high ASR against previous AT approaches in LLMs [6] . To evaluate the ASR, we use the harmfulness classifier from [6] , which was shown to align well with human judgement.Computational cost Given the constrained computational resources, we prioritised getting evidence to answer our main research question regarding the extrapolation of adversarial robustness. We want to emphasize that better trade-offs between utility and robustness might be obtained with more exhaustive hyperparameter search.Hardware All experiments were performed on an internal cluster of either V100, 40GB A100, or 80GB A100 GPUs. All conducted experiments required at least 1904 GPU hours.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: All tricks and hyperparameters are mentioned in the main paper ( § 4) or Appendix. Furthermore, code will be published if accepted. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
results: 
 In the following, we illustrate the computational benefit of continuous AT compared to existing discrete methods. Subsequently, we show improved robustness against state-of-the-art discrete attacks by using continuous adversarial training (AT). 1 , we compare the combined number of forward and backward passes used by the discrete AT algorithm RD2D [6] with CAT and CAPO. Computing a single adversarial example with R2D2 is ≈ 128.5 times more expensive than for CAT and CAPO, while the whole training is 299 times more costly. This illustrates the considerable compute advantage of continuous AT approaches compared to discrete methods.We first explore robustness extrapolation from continuous AT to discrete attacks for the CAT algorithm, which utilises additional utility data to maintain model performance. Figure 2 summarises the evaluation results. For all models, CAT + $ 5 0 / ( 6 6 0 0 / 8$ 5 & ( $ 5 & & 0 7 % ( 1 & + * & * $ 8 7 2 ' $ 1 3 $ ,5 6FRUH$65 8WLOLW\ $65 %DVH &$7 &$32 (a) GEMMA + $ 5 0 / ( 6 6 0 0 / 8 $ 5 & ( $ 5 & & 0 7 % ( 1 & + * & * $ 8 7 2 ' $ 1 3 $ ,5 6FRUH$65 8WLOLW\ $65 %DVH &$7 &$32 (b) PHI-3-MINI + $ 5 0 / ( 6 6 0 0 / 8 $ 5 & ( $ 5 & & 0 7 % ( 1 & + * & * $ 8 7 2 ' $ 1 3 $ ,5 6FRUH$65 8WLOLW\ $65 6WDQGDUG &$7 &$32 (c) MISTRAL-7B + $ 5 0 / ( 6 6 0 0 / 8 $ 5 & ( $ 5 & & 0 7 % ( 1 & + * & * $ 8 7 2 ' $ 1 3 $ ,5 6FRUH$65 8WLOLW\ $65 %DVH &$GY8/ (d) LLAMA-7B + $ 5 0 / ( 6 6 0 0 / 8 $ 5 & ( $ 5 & & 0 7 % ( 1 & + * & * $ 8 7 2 ' $ 1 3 $ ,5 6FRUH$65 8WLOLW\ $65 %DVH &$7 &$32 5' (e) ZEPHYR-7BFigure 2 : Trade-off between utility and robustness for CAT (Eq. 4), CAPO (Eq. 5), and R2D2 [6] , compared to their non-adversarially fine-tuned models. The objective is a small loss in utility and a large improvement in attack robustness. Larger is better for MMLU, ARC-E, ARC-C, MT-BENCH (left of dashed line). Smaller is better for GCG, AUTODAN, and PAIR (right of dashed line). MT-BENCH score is multiplied by 10 to see the change in performance on this y-axis. Additional results are included in App. B.considerably increases the average robustness against discrete adversarial attacks. For the GEMMA and ZEPHYR models, robustness increases for all attacks. For PHI-3-MINI and MISTRAL-7B, PAIR still achieves high attack success rates (ASR). In terms of utility, we observe similar degradations for all CAT trained models. All models still show considerable utility after fine-tuning.Compared to the ZEPHYR + R2D2 model, which was trained with discrete AT, CAT exhibits marginally worse utility on standard utility benchmarks while providing substantially improved robustness against discrete attacks. For, ZEPHYR + R2D2, PAIR achieves an ASR of 40%, while it achieves 10% ASR for CAT. We note a substantial difference in the HARMLESS benchmark, where CAT massively outperforms ZEPHYR + R2D2 showing that our method has not overfitted the safety objective or the patterns in the Harmbench behaviours. Note that the HARMLESS score of R2D2 demonstrates that it can not simultaneously achieve non-trivial utility and robustness, which are heavily dependent on not using or using the chat template, respectively.We further investigate if adversarial variations of proven alignment methods, such as IPO, can be used to align models in an adversarially robust manner (see Figure 2 ). For this purpose, we fine-tune GEMMA and PHI-3-MINI using the proposed CAPO algorithm. Figure 2 , illustrates differences between the base model, CAT, and CAPO. Despite using no utility dataset within CAPO to retain helpfulness, the algorithm does not introduce larger utility decreases on common benchmarks than CAT. Moreover, CAPO achieves considerably higher robustness against the jailbreaking method PAIR, demonstrating generalisation to diverse threat models. The PHI-3-MINI-IPO model achieves 100% attack robustness for all conducted attacks.For GEMMA, robustness improvements also mostly surpass CAT, with slightly lower robustness against GCG. Compared to R2D2, CAPO does not require an auxiliary dataset to maintain utility and achieves higher robustness on average. Specifically for PAIR CAPO trained models exhibit considerably higher robustness. Lastly, the PHI-3-MINI-IPO achieves a substantially higher score on the HARMLESS benchmark than CAT and R2D2.The results indicate that adversarial variations of common alignment methods, such as IPO, can be used to adversarially align LLMs. 
failure modes of training and robustness evaluations in llms: 
 Utility evaluation Common utility benchmarks such as MMLU or ARC do not use a chat template in their standard evaluation [43] . Firstly, this dramatically impacts performance, especially for smaller models, which often require a lot of prompt engineering to follow the few-shot prompts correctly. Secondly, it dramatically changes the mode of the model. In effect, a model might be overly robust in chat mode (i.e. when using a chat template) where it rejects most requests, but it might appear to have high utility in benchmarks because no chat template is used (e.g. MMLU). ARC as an evaluation benchmark is even more misleading as it measures the likelihood of a set of possible answer tokens, thus not reflecting the utility of the model when using a chat template. We quantitatively evaluate the refusals of MMLU questions when using a chat template in App. G. We recommend future work, to consider these issues when evaluating robustness and utility for the same model.Training data failure modes AT datasets such as Harmbench [6] or AdvBench [44] tend to use a common grammatical and syntactical structure, using imperative commands such as "Tell me" or "Give instructions". Chatting with our models and ZEPHYR + R2D2, we observe that requests would be refused when using this same style but are accepted if asked in a different style, such as "Could you please ...?". This holds for both harmful and harmless requests. For instance, ZEPHYR + R2D2 will refuse to answer "Tell me a story" and "Tell me how to build a bomb", but will answer "Could you please tell me a story?" and "Could you please explain to me how to build a bomb?". This also explains why the model may even appear useful under utility benchmarks employing chat templates such as MT-BENCH. To demonstrate this failure case we create two small benchmark datasets called POLITEHARMBENCH (see App. H) and HARMLESS. The former rephrases the harmful behaviours politely, and the latter consists of harmless requests formulated in the same grammatical style as the original HARMBENCH behaviours. We leave developing better datasets and benchmarks for a future paper as it is outside the scope of this work. 
adversarial training ablations: 
 Robust fine tuning without attack We found that continuous adversarial training successfully increases the robustness of LLMs to discrete adversarial attacks. Here, we explore whether robustness gains stem from using continuous adversarial attacks during training, or from the fine-tuning process itself. Thus, we fine-tune GEMMA using the CAPO algorithm but without using adversarial attacks. We observe no robustness gains when fine-tuning without attacks (see App. B.2). This demonstrates that continuous adversarial attacks are a crucial part of our fine-tuning algorithm.One-step adversarial training in LLMs For all our experiments, we use 10 adversarial attack iterations. While this is orders of magnitude cheaper than calculating discrete adversarial attacks (GCG requires 2570 model evaluations with default settings), it still increases training time by an order of magnitude. We thus propose one-step AT with CAPO. As in previous work [3] , we set the step size of the attack to the magnitude of the ϵ-ball. This achieves robustness improvements comparable to the multi-step variant and slightly worse utility trade-offs (see App B.1). the x-axis (as a proxy for utility). Clear trade-offs between robustness and utility can be observed, ranging from models with high robustness and no utility to models showing less robustness than the standard non-robust models and slightly higher utility.Moreover, we analyse hyperparameter choices that affect the robustness-utility trade-off for CAPO in more detail. This includes the strength of the adversarial attacks defined by the ϵ magnitude and the IPO β value. Figure 3 illustrates that for both hyperparameters, we obtain intuitive robustness-utility trade-offs, where larger epsilon values and smaller β values are associated with increased robustness and reduced utility. A detailed analysis can be found in App C.We additionally investigated the relationship between training-time robustness to continuous adversarial attacks and inference-time robustness to discrete attacks. This is illustrated in Figure 4a . The observed strong Pearson correlation (r = 0.99, p = 0.0075) indicates that models robust to continuous attacks during training are also robust to discrete attacks at inference. This suggests continuous AT can be a reliable proxy for AT with discrete attacks. Thus, demonstrating the potential use of continuous attacks to reduce the computational burden of evaluating adversarial robustness [7, 8] .Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experiments are too computationally expensive to do several runs in particular the evaluations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 We answer our research question about the extrapolation of robustness under the continuous attack threat model to robustness under discrete attacks in the affirmative. We propose an efficient continuous adversarial training algorithm (CAT), combining training on an adversarial behaviour dataset with fine-tuning on utility data. Additionally, we introduce an adversarial variant of IPO (CAPO) that does not require additional utility data. Our algorithms achieve up to 100% robustness against a set of state-of-the-art attacks (PHI-3-MINI-CAPO), surpassing robustness utility trade-offs in previous work [6] while requiring at least 299 times less compute. In future work, we will further analyse settings where continuous robustness does not extrapolate (e.g. novel attacks) and possible ways to address this, such as larger and more diverse training data. Additionally, the objectives of preventing harmful output and machine unlearning are closely related as such the applicability of our method for machine unlearning would be an interesting angle for further exploration.We further show that great care is required in the evaluation of the robustness and utility of adversarially trained models. We demonstrate that previous work overfits the safety objective, refusing to answer benign queries. Further, we exemplify that both the chat template and the grammatical structure of prompts need to be carefully controlled to prevent a misleading evaluation.Limitations Our method relies on the quality and breadth of the harmful dataset, while we are less prone to overfit than ZEPHYR + R2D2, we may still see improvements from augmented adversarial training datasets [29] . An additional limitation is the number of hyperparameters introduced that require careful selection. We expect future work to achieve considerably better robustness-utility trade-offs through better hyperparameter selection alone. Furthermore, our proposed method CAT requires a utility dataset to retain helpfulness, which may shift the predictions of the model on unrelated tasks, a limitation we try to address with the CAPO method. Finally, due to limited compute we were not able to apply our method to much larger LLMs in the 70B parameter and larger regime, we leave this to future work.Broader impact This work aims to enable scalable adversarial training for LLMs to be robust against adversarial attacks. The positive impact is that this will reduce the amount of harmful content produced by LLMs if adopted as many attacks will no longer work. In addition, the lower computation cost should hopefully reduce the carbon footprint of training robust and safe LLMs. However, this may lead to overconfidence in the safety of LLMs, thus necessitating more extensive red teaming. Another possible negative impact of our work is that adversarial training may be used to prevent LLMs saying things the model operator does not want regardless of the harmfulness of the content. Our contributions on the failure modes of robustness evaluation should hopefully lead to more rigorous and trustworthy evaluation protocols. These are crucial to accurately assess the state of robustness in LLMs. Note, it may be that further failure modes exist we did not yet find.A Hyperparameter choicesEQUATIONA full list of hyperparameter choices is given in Table 3 . Below is an explanation what each means:Learning rate Learning rate for the model parameters.Batch size Total batch size used for the model training includes utility and behaviours.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide the total amount of GPU hours used.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We read the code of ethics and followed the guidelines. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See conclusion § 8. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper only provides a rephrased version of an already existing dataset. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We reference all used datasets (see § 4). Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: See the Appendix. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: We do not do any crowdsourcing or research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Optimiser Optimiser for the model parameters. AdamW was proposed in Loshchilov and Hutter [46] .Adv. Learning rate Adversarial learning rate is the step size α used in Equation 2.ϵ is used to define the ℓ 2 ball around the token embeddings for the valid attacks δ.β is the β parameter as described in the original DPO paper Rafailov et al. [31] .Away cutoff is the cut off value used for the away loss as described in § 3.3.Toward cutoff is the cut off value used for the toward loss as described in § 3.3.Utility data ratio is the percentage of utility data used as part of the total training data per epoch, e.g. 0.875 implies for every one adversarial behaviour example there is 8 utility examples.Away weight is α a in Equation 6.Toward weight is α t in Equation 6.Utility weight is α u in Equation 6.Quantisation is the level of quantisation for the model during training.Max seq. length is the maximum sequence length after which we truncate the token sequences for training.LoRa defines where the LoRa adapters are used. For all models we applied the LoRa adapter to all linear layers.We used a 10 iterations of the adversarial attack, a max grad norm of 0.3, a warm-up ratio of 0.03, a cosine learning rate scheduler, and training was done in floating point 16.The CAT algorithm has 5 important hyperparameters, the weight of the utility loss α u , toward loss α t , and away loss α a . Moreover, in preliminary experiments, we observed that away loss tends to dominate the training objective. Models that show very high away loss generally overfitted to the safety objective and stopped answering benign requests. We notice similar issues with the toward loss. Thus, we define a threshold for the away loss a cut and toward loss t cut , clamping values below a certain value. If not otherwise defined, we use the following hyperparameters in all experiments. We set α u = 1.0, α t = 0.5, and α a = 0.5, as in [6] . Further, we set a cut = -5 and t cut = 0.5. We use a ratio of 7 : 1 for utility and harmful examples during training. To prevent overfitting in the proposed CAPO, we use the IPO loss function [11] . Additionally, we set the β parameter of IPO to 0.25 for GEMMA models, 0.5 for PHI-3-MINI, and X for MISTRAL-7B, which we observed to result in good trade-offs between robustness and utility in preliminary experiments.Tab. 4 summarizes the models used in the experiments of this work. Table 5 summarizes the main adversarial training results. The proposed CAT and CAPO algorithms achieve competitive or even superior robustness utility trade-offs compared to the discrete adversarial training algorithm R2D2 [6] . For the ICL attack, we generated 64 affirmative examples for each question and then asked the target question from HARMBENCH, we evaluate these manually as the output was occasionally so far from human text as to confuse the classifier. For the adaptive attack (see Table 6 ), we use the evaluation commands proposed in their GitHub repository and gpt-4-o as a judge. 7 illustrates that one-step CAPO achieves similar robustness improvements as the multi-step variant. Note, that we used the same hyperparameters for the one-step attacks as for the multi-step attack, except for the attack iterations and step size. Further hyperparameter tuning or borrowing recent advances in one-step AT from other domains may help to close this gap [47] . Due to the large computational complexity of attack evaluations, we conduct this experiment on GCG. We evaluated if the proposed IPO-based training algorithm provides robustness without using adversarial attacks during training. Table 8 shows, that robustness does not improve without using attacks. Moreover, using alternative preference optimization algorithms, such as NPO [48] , does not improve robustness in our experiments either. Attack Strength: The right plot in Figure 3 illustrates the effect of varying the adversarial attack strength, characterised by the ϵ magnitude, on the robustness-utility trade-off. As ϵ increases from 0.0125 to 0.1, there is a significant reduction in GCG loss, from approximately 14.9 to near 0.Concurrently, the MMLU score improves markedly from 0 to around 0.39, demonstrating increased utility. This inverse relationship between GCG loss and MMLU aligns with prior work concerning utility robustness trade-offs [4, 45] .IPO β: In CAPO, the β parameter inversely relates to the difference in log-likelihood ratios between the safe answer and the harmful response. Thus, a smaller β indicates a larger disparity in these log-likelihood ratios. This intuitively should lead to robustness and utility trade-offs. The left plot in Figure 3 shows the impact of different IPO β values on robustness and utility. With β values ranging from 0 to 0.5, a consistent decrease in GCG loss is observed, starting from 6.1 and dropping to 0.8. Meanwhile, the MMLU score increases from about 0.25 to 0.38. This trend aligns with our expectations and suggests that higher β values are associated with lower GCG loss and improved utility, indicating that tuning β is crucial for optimizing the robustness-utility trade-off in CAPO.We sanity check our models and the continuous attack by showing that an unconstrained continuous attack breaks all our models (Figure 5 ). However, adversarial trained models are more robust against ϵ-ball attacks.WE verify that NPO [49] and IPO [11] do not outperform adversarial training as we do (see Table 9 ). F Adversarial training computational effort R2D2. The total number of forward passes F R2D2 required for a single GCG update in R2D2 was calculated as follows.F R2D2 = 5 • (B GCG + 1).The number of backward passes W R2D2 as:W R2D2 = I A .Here, B GCG is the number of attack candidates that are evaluated in every attack iteration and I A is the number of attack steps. I A is the number of backward passes computed for the GCG attack. Thus the combined number of forward and backward passes is:5 • 513 + 5 = 2570.Total. The total number of forward passes F R2D2 required by R2D2 was calculated as follows.F R2D2 = (b ut + 2 • b adv + b adv • (B GCG + 1) • I A ) • I T . b ut + 2 • b advis the cost of computing the loss for utility, away, and toward in one iteration. b adv • (B GCG + 1) • I A is the cost of the GCG attack performed in each iteration.The number of backward passes W R2D2 as:W R2D2 = (b ut + 2 • b adv + b adv • I A ) • I T .Here, b ut is the number of utility samples in every batch, b adv is the number of harmful behaviour samples in every batch, B GCG is the number of attacks that are evaluated in every attack iteration, I A is the number of attack steps, and I T is the number of training iterations. b ut + 2 * b adv is the backwards pass for utility, away, and toward losses. b adv • I A is the number of backward passes computed for the GCG attack. Mazeika et al. [6] used a batch size of 256 (according to the github repo 1 ) with 224 utility samples per batch and 32 adversarial behaviours per batch. Thus the combined number of forward and backward passes is:(224 + 2 • 32 + 32 • (512 + 1) • 5) • 2000 + (224 + 2 • 32 + 32 • 5) • 2000 = 165, 632, 000.CAT & CAPO. The total number of forward passes F U L required by our continuous adversarial training algorithm was calculated as follows.F U L = I A .The number of backward passes W U L as:W U L = I A .The combined number equals:10 + 10 = 20.CAT Total. The total number of forward passes F U L required by CAT was calculated as follows.F U L = (b ut + 2 • b adv + b adv • I A ) • I T .The number of backward passes W U L as:W U L = (b ut + 2 cdotb adv + b adv • I A ) • I T .The combined number equals:2 • (54 + 2 • 8 + 8 • 10) • 780 = 234, 000CAPO Total. The total number of forward passes F IP O required by CAPO was calculated as follows.F IP O = (2 • b adv + b adv • I A ) • I T .The number of backward passes W U L as :W IP O = (2 • b adv + b adv • I A ) • I T .The combined number equals:2 • (2 • 64 + 64 • 10) • 360 = 552, 960.We observed, that prior models trained to be adversarial robust exhibit high refusal rates for benign queries. To demonstrate this failure mode, we measure the rejection rate of MMLU queries see Table 10 . There are 57 subjects with a 100 questions each. In Table 10 we give the total number of refusals for each model. 0 GEMMA-2B-IT-UL 5 GEMMA-2B-IT-IPO 0 PHI-3-MINI 0 PHI-3-MINI-UL 20 PHI-3-MINI-IPO 2 MISTRAL-7B 0 MISTRAL-7B-UL 20 ZEPHYR-7B 0 ZEPHYR-7B-UL 217 ZEPHYR + R2D2 308 H POLITEHARMBENCHWe observed that phrasing the same behaviour only slightly differently, e.g. in a polite manner, could yield to striking vulnerabilities. So we developed POLITEHARMBENCH, see the results in Table 11 .Note that we use the Harmbench classifier as throughout the paper, but found several instances where the classifier incorrectly predicted that answers were safe when they were not. In line with our adversarial training objective, which uses the "toward" string "Sorry, I can't do that" we use "Sorry" as a refusal keyword and ignore capitalisation.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://github.com/centerforaisafety/HarmBench/blob/aa597effd960cd974e11df48d110772cb98aa249/ adversarial_training/README.md 


Decision: **Summary of the Paper**

This paper proposes a new method for adversarial training based on the continuous embedding space of an LLM, rather than discrete optimizations. This method combines two losses, optimizing for attacks while preserving utility. The authors tested this method on various model families and scales, showing that the algorithms improved model robustness against discrete attacks while maintaining utility. 

**Summary of Reviews**

- Reviewer Jk9f (Score 7 - Accept): The reviewer commends the clarity of the paper's presentation and the empirical data supporting the method's improvements on model robustness. They raise concerns about the defense's performance on unseen attacks, the paper's evaluation on potentially weak attacks, and further experiments investigating the limits of the two algorithms.
- Reviewer TJPv (Score 5 - Borderline Accept): The reviewer commends the paper's quality of writing and the performance improvements of the proposed method on adversarial training. They raise concerns over the comprehensiveness of experiments (e.g. only one safe response is matched for evaluations, and only relatively weak attacks are benchmarked) and the practicality of the method.
- Reviewer Z2aE (Score 5 - Borderline Accept): The reviewer commends the paper's overall presentation and the method's improvements in computational cost. They raise concerns over the comprehensiveness of experiments to support the method's performance improvement claim, the clarity of the technical descriptions of the method, the choice of models for benchmarking, and how the method compares to other defenses.

**Assessment**

Regarding reviewer Jk9f's concerns, the authors show that LLM robustness generalizes to out-of-distribution attacks, added additional metrics showing that the method improves robustness against newer attacks, added a sanity check to explore the limits of model robustness under this method, and added experimental data from Llama2. 

Regarding reviewer TJPv's concerns, the authors explain their limitations in evaluating other more computationally expensive methods for comparison, explain that broadening safe answers did not improve model robustness, added results for newer attacks, and explain that other methods tend to have even less practicality in terms of hyperparameter tuning. 

Regarding reviewer Z2aE's concerns, the authors provide metrics comparing the runtime differences between defenses, added Llama2 results, and explained that they only included defenses similar to their proposed method.

After carefully considering the points raised by reviewers and the authors' responses, I recommend an Accept.

Reviews: review: paper_summary: paper_summary: This paper proposes adversarial training for LLMs, in which perturbations are created in the continuous embedding space rather than finding discrete suffixes. The proposed fast adversarial training algorithm (AdvUL) consists of two losses: the first strengthens the model against continuous embedding attacks computed on an adversarial behavior dataset, and the second ensures the final model's usefulness by fine-tuning utility data. Furthermore, the authors introduce C-AdvIPO, an adversarial variant of IPO that does not rely on utility data for adversarially robust alignment. The empirical evaluation of four models, Gemma, Phi3, Mistral, and Zephyr, at various scales, reveals that both algorithms improve LLM robustness against discrete attacks (GCG, AutoDAN, PAIR).
strength_weakness: strength_weakness: The Weaknesses of this paper include:

- The main difference between adversarial attacks/defenses on traditional models (e.g., CNNs) and those on LLMs is that there is no explicitly defined *threat model* when jailbreaking LLMs. Namely, the strategy to jailbreak an LLM could be quite flexible, and there is usually no constraint on human imperceptibility. AT is a strong defense under a given threat model (e.g., $8/255, \\ell\_{\\infty}$), but AT is observed to generalize badly to unseen attacks. While I'm experienced with AT in traditional settings, I'm still not very convinced that AT could be a good solution for LLMs.
- While GCG, AutoDAN, and PAIR are commonly evaluated attacks, they are relatively weak nowadays. The authors are encouraged to evaluate their models against more advanced and diverse attacks, such as ICL-based [1,2] and/or decomposing-based [3].
- To investigate the limits of C-AdvUL and C-AdvIPO, there should be a sanity check to directly perform attacks in the continuous embedding space.
- From my perspective, what C-AdvUL and C-AdvIPO do is connected to machine unlearning (i.e., unlearning the harmful knowledge), where [4,5] have used similar learning objectives. Their connections should be more discussed and empirically ablated.

Refs:\
[1] Many-Shot Jailbreaking\
[2] Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses\
[3] A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI Responses\
[4] Negating Negatives: Alignment without Human Positive Samples via Distributional Dispreference Optimization\
[5] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
questions: questions: Why there are no experiments done on the Llama series of models? Is there any reason for this design choice?
limitations: limitations: Please see Weaknesses.
paper_summary: paper_summary: This paper introduces adversarial attacks in continuous space in the context of LLMs. In addition, it utilizes continuous attacks for adversarial training to robustify LLMs and demonstrate that this efficient training algorithm can indeed protect LLMs against various attacks, including discrete ones, while maintaining the utility.
strength_weakness: strength_weakness: 1. The experiments should be more comprehensive: (1) In Figure 2, the comparison between R2D2 and the proposed methods is only conducted on the model ZEPHYR-7B, comparisons on more models would be preferred. (2) The authors only test one safe answer ''Sorry, I can't do that''. The safe answer $y$ is a very important variable in the loss objective function. Tests on more ''safe answers'' would make the results more convincing and more generally applicable. (3) The attack success rates are based on GCG, AutoDAN and PAIR. Recently, there have been some stronger attacks proposed, such as LLM-adaptive attacks (https://github.com/tml-epfl/llm-adaptive-attacks), the authors should include more attacks for a more comprehensive evaluation. (4) Ablation studies should be conducted to support why IPO is better than DPO, as line 161 to 162 claimed.

2. The proposed methods introduce more hyper-parameters to tune, hindering its application for practitioners.

3. The presentation of the paper can be improved, for example, the formulation in line 154 to 156 is very confusing, as $\mathcal{L}'$ is not defined.
questions: questions: The major concerns have been pointed out in the weakness part, the authors should first answer questions in that section. In addition, we have some minor questions: 

1. In line 210 to 211, why do bigger models have smaller values of $\epsilon$? Is there any intuition?

2. Similar to TRADES [Zhang et. al 2019], in addition to the value of $\epsilon$, is it possible to add a co-efficient to the last term of Equation (4) to balance the trade-offs between robustness and utility?

Overall, the paper tackles an interesting problem. However, due to the weakness part and the questions above, I believe the manuscript needs to be edited to improve. I welcome the authors to address my concerns during the rebuttal and will re-evaluate the manuscript after the rebuttal.

> Post Rebuttal

I improved my ratings to 5 after reading authors' rebuttal.
limitations: limitations: The limitations and the societal impact are discussed at the end of the paper.
paper_summary: paper_summary: Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. However, current adversarial training methods for LLMs are hindered by the high computational costs required to perform discrete adversarial attacks at each training iteration. To solve this problem, the paper instead calculates adversarial attacks in the continuous embedding space of the LLM, which is orders of magnitudes more efficient.
strength_weakness: strength_weakness: 1. The main claim of the paper is that the proposed method can use less computational cost to obtain a robust LLM, and regarding this viewpoint, there is only an experimental setting presented in Table 1, which may be insufficient. I believe that some more intuitive experiments should be added, such as comparative experiments on the real-time cost for a single example and for the entire process.

2. Although the paper is overall clear and the problem it intends to solve is well-defined, some details are not clear enough, especially the description of the proposed method's pipeline. For example, 'IPO' keeps appearing in the first two pages, but only in Section 3 are some descriptions of 'IPO' provided.

3. The hardware description in Section 4 is also unclear. I understand that this does not impact the main contributions of the paper, but a more explicit explanation of each experiment's settings would provide a clearer understanding for the readers. Additionally, totaling GPU hours across different GPUs may be unreasonable, as 1904 GPU hours on a V100 differ from those on 80GB A100 GPUs. Therefore, I hope the authors can further refine these details.

4. As LLAMA-2 is one of the most mainstream open-source LLMs, why have the experiments yet to be conducted on LLAMA-2 to evaluate the effectiveness of the proposed method?

5. Currently, the defense mechanisms on LLMs are not limited to R2D2 nor even to AT methods. I'm curious about how the proposed method compares with other methods, such as some methods mentioned in the related works.
questions: questions: See Weaknesses.
limitations: limitations: The authors have discussed limitations of the work.

Comments: comment: Thank you for your reply and feedback. 

We chose to evaluate our method with a subset of the strongest attack available at the point of submission (according to Harmbench [1]). Beyond the ICL attack and the adaptive attack that we performed for the rebuttal, we will conduct additional evaluations with the approaches proposed in [2, 3] for the final manuscript. These attacks have shown high ASR against robust models. 

We will add our ablation study on the behavior of $\epsilon$ for models of different sizes to the final manuscript. We are open to any further suggestions on ablation studies that would further improve our work.

We will add a more profound discussion about the differences between previous approaches in encoder-decoder models and the presented adversarial training algorithm. More specifically, we will: 
- Elaborate on the different training settings, i.e., post-training stage for LLMs and pre-training in encoder-decoder models 
- Differences in the optimization target, i.e., robustness in classification settings vs. preference optimization or alignment finetuning
- Differences in the optimization goal, i.e., improved generalization vs. adversarial robust alignment.

We want to emphasize that robustness in classification problems differs substantially from adversarial robust alignment. In the classification setting, models are trained to be robust to minor input perturbations with respect to their prediction. In contrast, in the alignment setting, we are not concerned with the stability of predictions. Rather, we train the model to refrain from generating "toxic" outputs altogether. To achieve this, losses need to be adjusted for this task (such as combining different objectives or using preference optimization algorithms). None of the previous methods was designed for LLM alignment and they can not be employed for this task without further changes. 

Lastly, the majority of these approaches were used during the pre-training stage of encoder-decoder models. Applying these algorithms during the pre-training stage of an LLM exceeds our computational budget by orders of magnitude.

[1] Zou et al., "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal" (Feb. 2024)

[2] Thompson T and Sklar M., "Fluent Student-Teacher Redteaming" (July 2024)

[3] Liao et al., "AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs" (May 2024)
I thank the authors for detailed feedbacks. Most of my concerns have been address, so I am willing to improve my rating to 5. The revised manuscript should include stronger attacks (adaptive attacks) and more ablation studies. I agree with Reviewer Z2aE that the author should highlight the difference of their proposed defence method and existing ones.
Thank you for your feedback on our paper. We'd like to clarify our position:
The primary contribution of our paper is to address a specific research question:
- Does adversarial training with continuous attacks in the token embedding space of an LLM provide robustness to discrete natural language attacks?

Prior works on adversarial training in NLP target different objectives, such as improved generalization or adversarial robustness in sentiment classification. These methods differ in several key aspects: 
Previous work didn't consider threat models, where the attacker has complete input control but generally enforced similarity to the initial text. This was done to keep the adversarial nature of the perturbation (to be semantically not meaningful to a human observer). However, we focus on alignment and not classification, which results in substantially different threat models.
Further, not all methods directly apply to decoder-only LLMs, and some only apply to pretraining, demanding unrealistic amounts of compute in the LLM setting. 

We want to emphasize that the scope of our paper is focused first on answering our research question and second on adversarial robust alignment algorithms. We believe the prior work mentioned above cannot be considered as a baseline that should be compared to our algorithms, as extending such prior work to our setting would be a contribution by itself. Instead, we adopted two of the most common losses used in LLM training. However, we acknowledge that leveraging such prior work to improve robustness of alignment is an interesting avenue for future work, and we will discuss this option in the outlook section of the final manuscript.
Thanks for the responses. Most of my concerns have been addressed.

However, I still have concerns about W5: The authors shared a series of similar works in the related work section, but failed to compare them in the experimental section. This can easily lead readers to question the accuracy of the limitations of related works.

Of course, I agree with authors’ explanation that some methods are orthogonal or irreproducible. Nevertheless, this does not substantially solve the above problem, although I acknowledge your explanation. I hope the authors can handle this issue more appropriately in the future. Hence, I maintain my score.
We thank the reviewer for their quick response and for increasing their score!

We agree that further attack evaluations would improve our work and will look into adding decomposition attacks to our work. We welcome any further feedback to improve our paper and are thankful for your feedback.
I thank the authors for their responses and additional experiments, which addressed the majority of my concerns. I understand that stronger attacks like [1,2,3] occur concurrently with the submission, so I didn't take this as a negative point. Nonetheless, I encourage the authors to conduct more comprehensive evaluations in their paper revision, particularly against attacks that differ from the (implicit) assumptions of C-AdvUL and C-AdvIPO. In conclusion, I believe this paper provides a good defense against jailbreaks through extensive experiments. So I'd like to raise my score to 7.