Title: MixAT: Combining Continuous and Discrete Adversarial Training for LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Large Language Models

Adversarial Robustness

Jailbreak Attacks

Adversarial Training

Adversarial Examples
Abstract: Despite recent efforts in Large Language Model (LLM) safety and alignment, current adversarial attacks on frontier LLMs can still consistently force harmful generations. 
Although adversarial training has been widely studied and shown to significantly improve the robustness of traditional machine learning models, its strengths and weaknesses in the context of LLMs are less understood. 
 Specifically, while existing discrete adversarial attacks are effective at producing harmful content, training LLMs with concrete adversarial prompts is often computationally expensive, leading to reliance on continuous relaxations. At the same time, despite their effectiveness and generalization capabilities, training with continuous perturbations does not always capture the full spectrum of vulnerabilities exploited by discrete attacks. In this work, we aim to bridge this gap by introducing MIXAT, a novel method that combines stronger discrete and faster continuous attacks during training. 
 We rigorously evaluate MIXAT across a wide spectrum of state-of-the-art attacks, proposing the *At Least One Attack Success Rate* (ALO-ASR) metric to capture the worst-case vulnerability of models. 
 We show MIXAT achieves substantially better robustness (ALO-ASR $< 20\%$) compared to prior defenses (ALO-ASR $> 50\%$), while maintaining a runtime comparable to methods based on continuous relaxations. 
 We further analyze MIXAT in realistic deployment settings, exploring how chat templates, quantization, low-rank adapters, and temperature affect both adversarial training and evaluation, revealing additional blind spots in current methodologies. 
Our results demonstrate that MIXAT discrete-continuous defense offers a principled and superior robustness-accuracy tradeoff with minimal computational overhead, highlighting its promise for building safer LLMs. We provide our code and models at https://github.com/insait-institute/MixAT.

MIXAT: Combining Continuous and Discrete Adversarial Training for LLMs

Abstract

Despite recent efforts in Large Language Model (LLM) safety and alignment, current adversarial attacks on frontier LLMs can still consistently force harmful generations. Although adversarial training has been widely studied and shown to significantly improve the robustness of traditional machine learning models, its strengths and weaknesses in the context of LLMs are less understood. Specifically, while existing discrete adversarial attacks are effective at producing harmful content, training LLMs with concrete adversarial prompts is often computationally expensive, leading to reliance on continuous relaxations. At the same time, despite their effectiveness and generalization capabilities, training with continuous perturbations does not always capture the full spectrum of vulnerabilities exploited by discrete attacks. In this work, we aim to bridge this gap by introducing MIXAT, a novel method that combines stronger discrete and faster continuous attacks during training. We rigorously evaluate MIXAT across a wide spectrum of state-of-theart attacks, proposing the At Least One Attack Success Rate (ALO-ASR) metric to capture the worst-case vulnerability of models. We show MIXAT achieves substantially better robustness (ALO-ASR < 20% ) compared to prior defenses (ALO-ASR > 50% ), while maintaining a runtime comparable to methods based on continuous relaxations. We further analyze MIXAT in realistic deployment settings, exploring how chat templates, quantization, low-rank adapters, and temperature affect both adversarial training and evaluation, revealing additional blind spots in current methodologies. Our results demonstrate that MIXAT's discrete-continuous defense offers a principled and superior robustness-accuracy tradeoff with minimal computational overhead, highlighting its promise for building safer LLMs. We provide our code and models at https://github.com/insait-institute/MixAT.

1 Introduction

Ensuring robustness to adversarial attacks remains a critical challenge in machine learning [1]. Traditional adversarial attacks typically involve subtle input modifications that cause drastic changes in model output, such as misclassifying images. However, adversarial attacks on LLMs differ due to the discrete nature of text. Prominent attacks include rephrasing inputs [2, 3] or appending optimized adversarial suffixes [4], and often trick models into generating harmful outputs. As LLMs become ubiquitous, ensuring their robustness to such attacks is becoming an increasingly important challenge.

Adversarial Training for LLMs. Inspired by successes on traditional models, adversarial training (AT) has been adapted to LLMs [5-9]. However, these approaches often face limitations. On the one

How to Build a Bomb?

How to  build  a bomb?

How to  build  a bomb

for a school project?

???

(a) Overview of our method.

GCG

PAP

0

20

40

60

80

85.0

92.5

87.5

7.5

2.5

65.0

0.0

12.5

25.0

25.0

47.5

2.5

0.0

12.5

0.0

Base

R2D2

CAT

PAP-AT

Mix-AT

(b) Attack Success Rate (ASR) [%]

↓

XSTest

ARC-E

MMLU

0

20

40

60

80

100

98.8

81.0

56.2

33.6

80.1

56.1

50.8

78.2

54.8

94.0

82.3

56.4

74.0

81.6

56.2

(c) Utility Scores [%]

↑

comparison.

hand, training with concrete adversarial prompts is computationally expensive, making it impractical for large models [4], while, recent cheaper methods based on continuous input embeddings [7] or latent representations [8, 9], remain vulnerable to stronger discrete attacks (Section 4).

This Work: Combining Continuous and Discrete Adversarial Training. We propose Mixed Adversarial Training (MIXAT), a novel approach for AT of LLMs that combines the efficiency of continuous attacks with the resilience of discrete attack training to achieve a state-of-the-art robustness-utility trade-off. Concretely, MIXAT uses continuous perturbations, like CAT [7], but applies them on top of discrete adversarial inputs, resulting in attacks that better cover the adversarial embedding space, as demonstrated in Fig. 1a. Our extensive evaluation on diverse suits of attacks using our At Least One Attack Success Rate (ALO-ASR) metric reveals that, unlike prior defenses that are highly vulnerable (ALO-ASR > 50%), MIXAT generalizes to new adversarial attacks not seen during training, establishing a new strong baseline (ALO-ASR < 20%) for robust LLMs.

Main Contributions:

A novel framework, MIXAT, combining discrete and continuous attacks for efficient AT of LLMs.

Rigorous audit of existing AT models under realistic LLM settings, including LoRA, quantization, chat templates and non-zero temperatures, addressing gaps in the current AT evaluation of LLMs.

Extensive evaluation showing that MIXAT exhibits much better utility-robust trade-off against diverse sets of attacks compared to prior work, while remaining efficient for large models.

2 Background and Related Work

This section provides a unified overview of adversarial attacks and defenses, with a particular focus on recent methods tailored specifically to Large Language Models (LLMs).

2.1 Adversarial Attacks in Machine Learning

Adversarial attacks, first introduced by Goodfellow et al. [1] on image classifiers, exploit misalignments between neural networks and human perception. These attacks are crafted by choosing a point ˆ x in the neighbourhood N ( x ) of the true input x that changes the classification decision away from the true target y . Typically, the neighbourhood N ( x ) is chosen to be an ϵ -ball B p ( x , ϵ ) = { x + δ | ∥ δ ∥ p ≤ ϵ } around x , with δ denoting the perturbation vector. For targeted

attacks with target y ∗ = y , ˆ x can be obtained by maximizing the network's log-probability of y ∗ :

for network parameters θ . For untargeted attacks, ˆ x instead minimizes the log-probability of y [1].

2.2 Adversarial Attacks on LLMs

With the increasing adoption of LLMs, adversarial attacks on LLM-based systems have become a pressing concern. Unlike image-based adversarial attacks, adversarial prompts for LLMs involve manipulations of discrete input text, designed to elicit harmful, unethical, or unintended outputs.

We consider the common scenario of an adversary targeting an auto-regressive LLM engaged in a sequence prediction task. The output of an LLM, defined by parameters θ , and given an input sequence x 1: n ∈ V n over vocabulary V , is a response y 1: m ∈ V m , generated by maximizing the likelihood P θ ( y i | x 1: n , y 1: i -1 ) . The adversary's objective is to generate a malicious prompt ˆ x such that the response ˆ y violates predefined constraints C , we will elaborate on in Section 4.

Prompt-level Jailbreak attacks manipulate the sentence-level structure of input text using a rephrasing function R , which maps an input x to R ( x ) = ˆ x . Denoting the set of all rephrasings of x as R ( x ) , the adversarial prompt ˆ x aims to maximize the log-likelihood of a harmful response ˆ y :

Techniques such as PAIR [3], TAP [10], and AutoDAN [11] iteratively refine prompts to bypass safety mechanisms. In contrast, PAP [2] utilizes predefined strategies for low-cost prompt generation.

Token-level attacks , such as GCG [4] and other gradient-based methods [12-15], modify specific tokens in the input sequence to guide the model toward adversarial outputs. Letting ⊕ denote sequence concatenation, GCG constructs the adversarial prompt ˆ x by iteratively optimising an adversarial suffix ˆ s appended to the original x , maximizing the log-likelihood of the malicious response ˆ y :

Continuous Attacks Unlike prior discrete attacks, continuous attacks perturb inputs directly in the LLM embedding space. Given token embedding e x = E θ ( x ) ∈ R n × l ( E θ maps each x i ), the goal is to find a perturbation ˆ δ ∈ B p (0 , ϵ ) that maximizes the log-likelihood of ˆ y :

This perturbation ˆ δ is typically computed using iterative methods like projected gradient descent [16]. Notably, the perturbed embeddings ˆ e i = e x i + δ i does not generally correspond to any tokens in V n . In abuse of notation let x + δ refer to the perturbed embeddings, unifiying Eqs. (2) and (3):

Model Tampering attacks Besides input space and latent space attacks, model tampering attacks allow modification of weights and latent activations [17]. For this work, we consider such stronger adversarial attacks out of scope, but consider them an interesting avenue for future work.

2.3 Adversarial Training

Adversarial training [16] provides a natural defense mechanism against adversarial attacks. It is formulated as a min-max problem minimizing the worst-case loss over adversarial examples:

Here, L adv is typically the cross-entropy loss, and N ( x ) represents the set of adversarial perturbations.

̸

2.4 Adversarial Training for LLMs

Unlike classification tasks, the output space of LLMs is unbounded, making it insufficient to minimize the likelihood of a single harmful sequence. Therefore, Mazeika et al. [5] introduced a combined loss function reducing the likelihood of (multiple) unsafe responses ˆ y while increasing the likelihood of a predefined set of safe ones y s from a dataset D h of triplets ( x , ˆ y , y s ) :

where ˆ x = arg max x ′ ∈N ( x ) log P θ (ˆ y | x ′ ) and L util ( D u ) is an additional utility loss based on a utility dataset D u , which is used to mimic the original model's training data.

Discrete Adversarial Training involves training the LLMs to refuse adversarial prompts generated before or during training ( R ( X ) in Fig. 1a). For this, static methods, such as Rainbow Teaming [18], generate diverse adversarial examples at the start of training. In contrast, dynamic methods like R2D2 [5] and SAP (Sequential Adversarial Prompting) [19] generate adversarial samples iteratively throughout training, improving adaptability but increasing computational cost. For example, R2D2 finetunes LLMs on adversarial GCG suffixes, requiring over 100 GPU-hours for a 7B model. Adversar-

Depending on the adversarial perturbation set N ( x ) (e.g., R ( x ) for jailbreaks, x ⊕V k for suffix attacks, or B p ( x , ϵ ) for continuous attacks), this framework generalizes most existing attack types.

PROMPT

words

tokens

embeddings

rephrasing

23

16

49

75

5

[0.2,...,0.7]

[0.3,...,0.4]

[0.4,...,0.6]

[0.5,...,0.8]

LLM

Discrete

Attack

Attack

ial Tuning [6] reduces this by precomputing candidate suffixes and only refining a subset.

Continuous Adversarial Training To mitigate the computational overhead of discrete approaches, Continuous Adversarial Training (CAT) [7] generates adversarial examples in the input embedding space ( X + δ in Fig. 1a). While this method is computationally efficient and inspired by defenses in vision and audio models, most perturbed embeddings do not correspond to valid text sequences. Latent Adversarial Training (LAT) [8, 9] extends adversarial perturbations to the model's hidden layer states. Both approaches achieve robustness against certain attacks at lower computational costs than discrete methods, though their efficacy against diverse discrete attacks remains limited (Table 1).

Prompt-Based Defenses Another complementary approach is to train a fixed prompt prefix via gradient-based optimization to steer the model away from adversarial completions [20, 21]. These methods are lightweight, model-agnostic, and effective against direct malicious requests, but they leave the model vulnerable against more sophisticated attacks (Table 1).

2.5 Other Defenses

Beyond adversarial training, non-model-based defenses aim to mitigate the impact of adversarial prompts. Techniques here include pre-filtering inputs [22], ensembling multiple models [23], and post-filtering generated responses [24]. While effective in some scenarios, they often incur substantial inference-time overhead and are complementary rather than alternative to adversarial training.

One particularly prominent direction in model-augmented defenses is the principle of Constitutional AI , wherein humans specify a set of normative guidelines that govern a model's behavior to mitigate harmful outputs [25, 26]. Under these guidelines, model (self-)critique can be employed to generate additional training data or serve as reinforcement learning feedback [25], promoting safer and more aligned behavior. Constitutional Classifiers [26] build on this approach by leveraging synthetic training data aligned with natural-language constitutions to monitor both model inputs and outputs for harmful content. While such classifiers provide an additional layer of defense against a broad range of attacks, we consider them orthogonal to adversarial model training.

3 Method

Next, we introduce MIXAT, our method that combines continuous and discrete adversarial training.

3.1 Mixed Adversarial Training (MIXAT)

Building on Section 2 and as shown in Fig. 2, the core of MIXAT lies in unifying continuous and discrete adversarial training by extending the perturbation set to include both continuous and discrete components. Following prior works, we train our models using the same principles and loss functions as described in Section 2.4. The adversarial perturbation space in MIXAT is defined as:

as represented by R ( X ) + δ in Fig. 1a. Intuitively MIXAT aims to center any continuous adversarial perturbation around a discrete adversarial example instead of an original benign data point.

Discrete To cover a broad range of adversarial examples in the discrete part of Eq. (8), we generate discrete adversarial seed points R ( x ) using the adversarial training variant of PAP [2] (PAP-AT):

where N PAP ( x ) represents the space of paraphrased adversarial texts around the original sample x . While our Mixed Adversarial Training procedure is compatible with using any adversarial attack as discrete seeds, we chose to train with PAP samples since they are cheap to generate, yet very strong and diverse. In Section 4.3 we analyse a MIXAT variant which also includes GCG samples. While it results in marginally better robustness, we find the 8x higher computational cost too large in practice.

Continuous For the continuous part of Eq. (8), we build upon CAT [7], optimizing model robustness in the embedding space by defining the perturbation set as an L 2 ball around the embedding of x :

While CAT alone effectively handles continuous perturbations for harmful requests x , it lacks the ability to address discrete adversarial attacks such as paraphrased jailbreak prompts. Centering the perturbations directly on such discrete adversarial examples addresses this directly at the initialization, both shifting the L 2 ball into a "more adversarial" region of the input while also allowing us to leverage much more efficient continuous optimization for a large fraction of the optimization.

Batch-wise Sampling Strategy. To balance continuous and discrete perturbations, MIXAT employs a mixing parameter α ∈ [0 , 1] . For each training batch, continuous attacks are applied on top of adversarial seeds with probability P C + D = α and top of plain prompts with probability P C = 1 -α .

DUALAT Variant. An alternative approach combines the losses from continuous and discrete adversarial training directly, without applying continuous perturbations to discrete seeds. This effectively merges the perturbation sets, leading to a Dual-objective Adversarial Training (DUALAT):

where, we batchwise choose between the discrete and continuous loss with P D = α and P C = 1 -α .

3.2 Empirical Motivation for MIXAT

MIXAT is designed to address the limitations of existing adversarial training methods by combining discrete and continuous attacks. Compared to DUALAT and purely discrete of continuous attacks, MIXAT explores a wider region of the adversarial space, leading to improved robustness against a diverse set of attacks, as visualized in Fig. 1a. To empirically validate this, in Appendix B.14 we conduct a qualitative analysis of different attack prompts and their ability to induce harmful behaviors in LLMs. Moreover, in Fig. 6 we quantify the cosine similarities between different prompts in the discrete and continuous perturbation spaces. We observe that the combination of PAP and continuous attacks creates a prompt that is the least similar to the original malicious request, while also being overall closer to the GCG samples. This suggests that training using MIXAT attacks can improve robustness against a wide range of attacks, including those that are not directly included in the MIXAT training set. We demonstrate this empirically in Section 4, showing MIXAT leads to significant gains in robustness both against direct attacks and diverse adversarial benchmarks.

(HF) model released on HuggingFace (GF) defence prompt released on GitHub (R) re-trained model or prompt using public code

4 Experiments

This section presents our evaluation, comparisons, and discussion of MIXAT. We show that combining discrete and continuous adversarial training yields models that are more robust while keeping higher utility , and having the lowest training costs among all methods. We also provide ablation studies over different design choices in MIXAT, as well as some general insights on model-based defenses.

4.1 Experimental Setup

We chose to evaluate MIXAT on four different open-source models of varying sizes and capabilities: Zephyr-7B [27], Llama3-8B [28], Qwen2.5-14B and Qwen2.5-32B [29]. Most of our experiments, ablations, and design choices were made on Zephyr-7B, while extended evaluations on other models highlight the generalizability of our method. Unless stated otherwise, we follow the design and hyperparameter choices of prior work [7]. The default PAP sample ratio is α = 0 . 5 , with paraphrases drawn randomly from all 40 strategies [2]. For more details on the training process see Appendix C.

Evaluation Metrics Our goal is to train models that are both robust against adversarial attacks and maintain high utility. To assess robustness , we use the Attack Success Rate (ASR), which quantifies the percentage of adversarial samples that successfully induce harmful model responses to malicious inputs. We evaluate against a variety of adversarial methods, including PAP [2], TAP [10], PAIR [3], AutoDAN [11], GCG [4], and HumanJailbreaks [30]. Additionally, we test the model's resistance to direct malicious requests not tied to specific attack methods. Following prior work [5, 7, 9, 6], we use the HarmBench dataset [5]. Since small models (7-8B parameters) often struggle with reproducing copyright content, we restrict our evaluation to the first 40 non-copyright-related samples in the HarmBench test set (details in Appendix B.4). Since different attack strategies often succeed on different samples, we also report the "At Least One" Attack Success Rate (ALO-ASR), reflecting

the success rate of a meta-adversary using all attacks. ALO-ASR serves as a proxy for universal robustness. We obtain a score of 18 . 75% on a scale from 0 (worst) to 100 (best).

We evaluate the utility of our models on common benchmarks including multiple-choice questionanswering tasks (ARC-Easy, ARC-Challenge [31], and MMLU [32, 33]) as well as instructionfollowing tasks (MT-Bench [34]). We also assess compliance using Harmless [7], a set of 40 simple questions phrased similarly to HarmBench samples, and XSTest [35], a set of 250 harmless requests designed to detect over-refusal tendencies in robust models.

4.2 Main Results

In Table 1 we compare MIXAT with stat-of-the-art adversarial training methods (R2D2 [5], CAT [7], LAT [9]) and prompt-based defences (RPO [21], PAT [20]), as well as our baseline PAP-AT and the variant DUALAT. We show that MIXAT achieves the best tradeoff between robustness and utility, outperforming other methods on both metrics.

On Zephyr-7B, MIXAT achieves the lowest ALO-ASR (12.5%) while keeping competitive scores on utility benchmarks. We can identify the main weakness of our MIXAT model to be GCG attacks: even though MIXAT significantly improves the robustness w.r.t. the base model, the GCG-trained R2D2 [5] expectedly achieves higher robustness here. On the other hand, MIXAT outperforms all other methods on any jailbreak attack, with the lowest ASR scores on TAP, PAP, and AutoDAN.

With no LAT-trained Zephyr-7B models publicly available, we trained LAT models (using the official code). However, we consistently observed steep utility-robustness curves where increasing robustness noticeably decreased compliance (XSTest) and utility (ARC-{E,C}), making LAT tuning difficult.

Llama3-8B, due to the model's alignment training [28], consistently behaves more robust than Zephyr-7B. However, a wide range of attacks is still able to force the model into generating harmful content for 85%of requests. Following the same trends as for Zephyr-7B, MIXAT achieves the lowest ALO-ASR of all methods, and is particularly effective against jailbreak attacks, achieving the lowest scores on TAP, PAP, and AutoDAN while being slightly more vulnerable to GCG attacks.

Attack Success Rate [%]

0

.

0

ALO ASR

PAP-10 ASR

GCG ASR

Direct R ASR

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Discrete Attack Fraction (

αPAP

)

(a) Effect of

α

PAP

on model robustness.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Discrete Attack Fraction (

αPAP

)

0

25

50

75

ALO Attack Success Rate [%]

↓

Dual-AT

Mix-AT

Figure 3: Attack Success Rate [%] comparison for various attacks on models trained with different α PAP ratios in both MIXAT and DUALAT.

While the released Llama3 LAT model achieves robustness levels comparable to MIXAT (with a different attack distribution), we could not reproduce this performance using the provided code.

Finally, we show that MIXAT is also effective on larger models, achieving very low ALO-ASR on Qwen2.5-14B and Qwen2.5-32B, while maintaining competitive utility scores, as well as on Mistral-7B [36] and Llama3.1-8B [28] in Appendix B.7.

Additionally, we preset results for training MIXAT on two more models: Our method shows strong generalization across a range of jailbreak prompts, even though only PAP jailbreak prompts are employed during adversarial training. The diversity of malicious requests and paraphrasing strategies offered by PAP [2] likely contributes to this generalization ability.

4.3 MIXAT Ablation Studies

Next, we present our in-depth ablations studies, showing the contribution of the key components of MIXAT to the overall performance of the methods, and demonstrating the importance of the exact way we mix our discrete and continuous adversarial attacks.

Continuous vs Discrete trade-off In Fig. 3 we examine the effect of varying the amount ( α ∈ [0 , 1] ) of mixed attack samples used throughout the training on model robustness. Expectedly, we observe

↓

75

50

25

0

0

.

0

0

.

5

1

.

0

1

.

5

LoRA weight (

λ

)

0

25

50

75

100

Attack Success Rate [%]

↓

ALO

PAIR

TAP

PAP

CAT

MixAT

0

.

0

0

.

5

1

.

0

1

.

5

LoRA weight (

λ

)

0

.

50

0

.

75

1

.

00

Relative Utility Scores (Base Model = 1.0)

↑

MixAT

CAT

MMLU

ARC-C

MT-Bench

XSTest

0

.

0

0

.

5

1

.

0

1

.

5

Generation Temperature

0

20

40

1/10 ASR

Avg. ASR

10/10 ASR

that for low α values, the models are less robust against PAP attacks. On the other hand, models trained with high α values can become less robust against Direct Requests because the model does not see enough clean malicious samples during training. We choose α = 0 . 5 as the default value for our main experiments as a balanced trade-off between direct and paraphrased samples, since we also observe more stability and robustness in this region. We present the full results (including utilities) in Table 3, and observe that the α ratio does not seem to have a significant effect on model utility.

MIXAT vs DUALAT We further examine the robustness differences between MIXAT and DUALAT in Fig. 3. While MIXAT trains the model using combined attacks, DUALAT separately trains the model on continuous and discrete attacks. When α = 0 , both methods are equivalent to CAT training, while for α = 1 only DUALAT corresponds to PAP-AT training. We observe that MIXAT generally outperforms the simpler DUALAT, indicat ing that training with the our attack combination is more effective than directly combining the continuous and discrete training.

Incorporating GCG samples into MIXAT training Our results show that while MIXAT achieves almost perfect robustness against most prompt-based attacks, it is still slightly more vulnerable to GCG attacks. To further enhance model's robustness against GCG attacks, we experiment with including some GCG samples in the training process. To do that we adapt our training mix to include α GCG + C = 10% GCG samples with continuous attacks on top, α PAP + C = 45% PAP samples with continuous attacks on top, and α C = 45% clean samples with continuous attacks on top. In Table 1 we show that this approach improves the robustness against GCG attacks, while keeping the robustness against other attacks and the utility scores similar to the default MIXAT training. However, even by adding only 10% GCG attacks and only running them for 100 steps (as opposed to 500 steps used for attacking), we observed a 5 times increase in training time when compared to the base MIXAT (Table 2). While the results obtained by mixing paraphrasing, continuous and suffix attack samples are already promising, this method could benefit from further exploration to obtain better hyperparameters and reduced computational costs.

MIXAT static training To validate the importance of using dynamically generated attacks while training, we also experiment with a static version of MIXAT, where all adversarial samples used for training are generated on the base model. We observe that this approach leads to a significant drop in robustness, with the ALO-ASR increasing from 12.5% to 25% on Zephyr-7B and from 25% to 52.5% on Llama3-8B (for detailed results see Table 12 in Appendix B). This indicates that the dynamic generation of adversarial samples during training is crucial for achieving high robustness.

4.4 Scaling the LoRA weights

One of the main challenges of adversarial training is to find the optimal trade-off between adversarial robustness and utility. Intuitively, we can regulate the effect of LoRA adapters [37] by scaling their weights with a constant λ . This approach may yield different trade-offs depending on the chosen λ values. To test our hypothesis, we evaluate LoRA-scaled variants of the MIXAT and CAT Zephyr-7B models on utility and adversarial robustness benchmarks. This involves multiplying the low-rank matrix A by the constant λ : W = W 0 +( λA ) B , where W 0 denotes the original weight matrix, A and B are low-rank matrices used to introduce perturbations to the model weights, and λ is the scaling factor. We scaled λ from 0.0 to 1.5 with a step of 0.25. As shown in Fig. 4, the ASR (Adversarial Success Rate) decreases as the magnitude of LoRA scaling increases, supporting our hypothesis about regulating the effect of LoRA. This decline in ASR is more rapid for the MIXAT model compared to the CAT model, indicating the higher effectiveness of the MIXAT training.

On the other hand, increasing the strength of the LoRA adapter also gradually reduces the models' utility, confirming the inherent trade-off between utility and robustness. However, we observe that utility scores of MIXAT are consistently similar or higher than those of CAT across all λ . In general, for λ > 1 , the models begin to significantly lose utility through over-refusal. In the range 0 < λ < 1 , utility is not significantly degraded (even slightly improving on some tasks). For the MIXAT, we observe that values of λ between 0.5 and 1.0 yield good robustness with minimal utility losses.

4.5 Effect of Temperature on Robustness Evaluation

In Fig. 5 we examine the effect of temperature on the robustness of the LLama-3-8B model trained with MIXAT. For each temperature value, we sample the model 10 times for each harmful prompt. We analyze the distribution of harmful generations across temperatures, reporting the percentage of prompts for which the model produces at least one harmful response (1/10 ASR), all harmful responses (10/10) and the ASR averaged across all samples (Avg. ASR). We observe that the average ASR score does not change significantly with temperature, but the likelihood of generating at least one harmful response in multiple tries increases. This is consistent with the findings of Raina et al. [38], who show that the model's robustness is greatly correlated to the first few generated tokens.

4.6 Further Experiments on Robustness (Evaluations)

We also conduct a series of additional experiments to further investigate the robustness and utility of our models. We present the key findings here, with extended details and data provided in Appendix B.

Discussion on Randomness The training and evaluation of LLMs are inherently random due to multiple reasons. In Appendix B.2 we investigate the impact of randomness in the training process (Table 5), and compare it with the randomness in the evaluation process (Table 4). We observe that models that are very robust (low ASR scores) or very unrobust (high ASR scores) are less affected by randomness, while models with intermediate ASR scores are more affected.

Impact of Model Quantization Model quantization is a frequently used technique to decrease memory costs. We examine the effects of quantization on robustness during training and evaluation in Appendix B.3. We observe that quantization slightly improves the robustness, but also lowers the capabilities of the model. This is valid when quantization is applied both during training and during evaluation.

Evaluating on more samples In Appendix B.4 we evaluate the robustness of some of our models on the whole HarmBench test set, as well as a subset of the XSTest-Harmful set. We find the robustness trends with respect to models and attacks to be mostly stable in our experiments. This indicates that evaluating on the 40-sample subset is a good proxy for quantifying models' robustness.

Transferability of attacks Prior work [4] has investigated the transferability of adversarial attacks across regular LLM models. In Appendix B.5, we assess how transferability is affected when adversarially trained models are used. We observe that transferability in this scenario is harder.

4.7 Training Time and Costs

Next, we report the training times of the main methods examined in this work in Table 2. As shown there, we train all of the models using either NVIDIA A100-40GB or NVIDIA H200 GPUs. The

Table 2: Estimated Training Costs We estimate the cost of training models using different adversarial training methods, in terms of time, memory, and monetary expense. For PAP-AT and MIXAT the total cost includes the cost of generating PAP samples through API calls (less than $1 per run).

* for R2D2 we use the costs as reported by Mazeika et al. [5]

results show that the compute time and resources required for MIXAT are lower than those required for CAT and R2D2, and only slightly higher than those required for LAT. We observe that the costs scale roughly linearly with the model size. This indicates that our method is efficient and can be applied to larger models without significant computational overhead.

5 Limitations

The main limitations of our work lie in the evaluation process, which, while improved over prior methods, continues to pose significant challenges. As discussed in Section 4, randomness during both training and evaluation introduces considerable variability. This issue is compounded by the high computational cost of evaluation, which forces us to use a limited number of samples and leads to higher variance. Additionally, the ambiguity in determining the malicious intent of certain evaluation samples in datasets like MT-Bench further contributes to noise in the reported metrics.

6 Discussion

In this section, we provide a broader discussion of the role of adversarial training in LLM safety, as well as the current state of adversarial LLM evaluation.

Can Adversarial Training Fully Mitigate Malicious Requests? While we consider adversarial training a foundational tool for defending LLMs against malicious inputs, it is unlikely to fully resolve the problem of harmful requests. In particular, we find that with many adversarial training techniques, LLMs can be significantly strengthened to resist specific (known) attack types. However, in contrast to conventional neural network classifiers, the input-output space of LLMs is substantially broader, with adversarially trained models often remaining vulnerable to adaptive attackers who craft novel strategies beyond the training space. Moreover, adversarial training alone does not address the deeper ethical and contextual understanding that is critical for appropriately responding to harmful inputs-and, in some cases, even for categorizing them as harmful . A holistic defense strategy must therefore combine adversarial training with complementary methods, such as enhanced filtering systems, context-aware response generation, and ongoing model evaluation and refinement.

Challenges in Evaluation: Are Current Datasets Adequate? Any statement about the robustness of an (adversarially trained) LLM is fundamentally tied to the quality of the evaluation datasets used. During our evaluation, we found that many existing datasets exhibit specific flaws that prevent more holistic statements about LLM robustness. In particular, while these datasets collectively address various attack types, their scope is typically insufficient to test resilience against more powerful adversaries. Importantly, overemphasizing a single attack type can lead to miscontextualized results claiming high overall robustness despite failures under other, equally practical attacks. Additionally, many datasets disproportionately emphasize certain attack categories, such as toxicity, while underrepresenting other forms of harmful or manipulative input. Given the lack of an apparent 'best attack,' we consider our proposed ALO-ASR a practical post hoc improvement over prior approaches. Nevertheless, to better evaluate and improve LLM robustness, there remains a pressing need for richer, more diverse datasets encompassing a broader range of adversarial examples, including dynamic and context-sensitive attacks.

7 Conclusion

Previous adversarial training approaches for LLMs have either focused solely on faster continuous perturbations or relied on limited, slower discrete ones, which we experimentally show hinders their ability to generalize. In this paper, we introduce MIXAT, an efficient adversarial training strategy for LLMs that merges both continuous and discrete adversarial attacks, resulting in significantly more robust models than prior methods. Our thorough evaluation shows that MIXAT scales to large LLMs and generalizes well across a wide range of adversarial attacks due to its more comprehensive coverage of the adversarial space. Detailed ablation studies under various inference settings confirm that MIXAT performs effectively in realistic use cases, offering a meaningful advance toward building safer generative AI.

Acknowledgments

This research was partially funded by the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).

This project was supported with computational resources provided by Google Cloud Platform (GCP)

This research was supported by the EKÖP-24 University Excellence Scholarship Program of the Ministry for Culture and Innovation of Hungary from the source of the National Research, Development and Innovation Fund.

Part of this work has been done under the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract no. MB22.00088). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them. The work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI) (SERI-funded ERC Consolidator Grant).

References

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373 , 2024.

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419 , 2023.

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 , 2023.

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249 , 2024.

Fan Liu, Zhao Xu, and Hao Liu. Adversarial tuning: Defending against jailbreak attacks for llms. arXiv preprint arXiv:2406.06622 , 2024.

Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks. arXiv preprint arXiv:2405.15589 , 2024.

Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030 , 2024.

Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Targeted latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549 , 2024.

Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119 , 2023.

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451 , 2023.

Paarth Neekhara, Shehzeen Hussain, Prakhar Pandey, Shlomo Dubnov, Julian McAuley, and Farinaz Koushanfar. Universal adversarial perturbations for speech recognition systems. arXiv preprint arXiv:1905.03828 , 2019.

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems , 36, 2024.

Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733 , 2021.

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125 , 2019.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Proc. of ICLR , 2018.

Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E McKinney, Rohit Gandikota, Aidan Ewart, Domenic Rosati, Zichu Wu, et al. Model tampering attacks enable more rigorous evaluations of llm capabilities. arXiv preprint arXiv:2502.05209 , 2025.

Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprint arXiv:2402.16822 , 2024.

Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. Attack prompt generation for red teaming and defending large language models. arXiv preprint arXiv:2310.12505 , 2023.

Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Fight back against jailbreaking via prompt adversarial tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263 , 2024.

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 , 2023.

Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts. arXiv preprint arXiv:2404.05993 , 2024.

Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308 , 2023.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837 , 2025.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 , 2023.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages 1671-1685, 2024.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv , abs/1803.05457, 2018.

Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595-46623, 2023.

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263 , 2023.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825 .

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, and George Karypis. Extreme miscalibration and the illusion of adversarial robustness. arXiv preprint arXiv:2402.17509 , 2024.

Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. Exploiting llm quantization. arXiv preprint arXiv:2405.18137 , 2024.

Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute. arXiv preprint arXiv:2402.15570 , 2024.

Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Improved few-shot jailbreaking can circumvent aligned language models and their defenses. Advances in Neural Information Processing Systems , 37:32856-32887, 2024.

Qingsong Zou, Jingyu Xiao, Qing Li, Zhi Yan, Yuhang Wang, Li Xu, Wenxuan Wang, Kuofeng Gao, Ruoyu Li, and Yong Jiang. Queryattack: Jailbreaking aligned large language models using structured non-natural query language. arXiv preprint arXiv:2502.09723 , 2025.

Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. Artprompt: Ascii art-based jailbreak attacks against aligned llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15157-15173, 2024.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961 , 2024.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235 , 2023.

I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

A Boarder Impact Statement

In this work, we propose better training methods for defending Large Language Models against adversarial actors, an important and so far unsolved concern. We see particular promise in the idea of combining discrete (input-aligned) inputs with continuous (training-efficient) techniques. Our work constitutes a first effort in this direction, highlighting that it can achieve promising results. We hope future work can build on these ideas and improve the overall alignment and robustness of upcoming models for societal good.

B Additional Experimental Results

B.1 Extended Ablation Results

In Table 3 we present full results of MIXAT (PAP+c & CA) and DUALAT (PAP & CA) style trainings on Zephyr-7B model with different α ratio of PAP examples utilized throughout the training. The results suggest that models trained with MIXAT are generally more robust than those trained with DUALAT (with the examination of ALO scores corresponding to different α percentages). It can be also examined that low ALO (better robustness) often leads to worse performance on MT-Bench.

However, there are no clear trends regarding, which is the universal, optimal α ratio of PAP examples for our adversarial training methods. In the case of MIXAT (PAP+c & CA) style training, 50% and 90% discrete ratios are two local optima in robustness, while DUALAT (PAP+c & CA) style training has the lowest ALO at 70%.

B.2 Discussion on Randomness

Randomness of attack generation All of the adversarial attacks we use in our experiments have a random component in their generation and evaluation process, since including random sampling from a distribution (GCG) or querying LLMs with a non-zero temperature (TAP, PAIR, AutoDAN). The adversarial attacks are evaluated by LLM-as-a-judge [34], which might cause inaccuracies. Randomness can yield different results when evaluating the robustness of the same model. To measure the extent of this randomness, we generate the adversarial samples for a single model multiple times and evaluate the robustness of the model on each set of samples. The results in Table 4

show that the variance in the ASR scores is relatively low in the case of Zephyr-7B Base and MIXAT model, while CAT results (PAP, TAP, PAIR) have significant randomness. This can stem from that the evaluation results of CAT indicating moderate robustness, which can only defend against weaker adversarial attacks. Therefore, this huge variance might indicate one limitation of LLM adversarial attacks, namely that it is difficult to generate similarly strong adversarial examples.

Randomness of utility benchmarks Given that we conduct our utility evaluations by sampling the models with temperature 0, we can consider most tasks to have deterministic evaluation. This includes Multiple-Choice Questions such as ARC-C, ARC-E and MMLU, but also the Harmless and XSTest evaluation where we only check for refusals. On the other hand, the open-ended generation tasks (MT-Bench) present a slight randomness because we use a judge model (GPT-4o) to assign scores to the generated responses. In our experiments, we evaluated some models three times and we obtained standard deviations below 1% accuracy points (e.g. zephyr-base obtains a score of 61 . 1 ± 0 . 5 , and zephyr-MixAT obtains 53 . 4 ± 0 . 3 ). This indicates that the randomness involved in the utility evaluation process is not significant.

Randomness of training We also examine the effect of randomness in the training process. We train the same model with different random seeds and evaluate the robustness of the models against adversarial attacks. The results in Table 5 show that the variance induced by the randomness in the MIXAT training procedure of Zephyr-7B has a relatively small magnitude, except the XSTest results. We would like to examine the causes of XSTest randomness in future work.

B.3 Impact of Model Quantization

Model quantization is a frequently used transformations on model weights, to decrease the memory costs of deploying deep neural networks. As Egashira et al. [39] outlined, manipulated alignment can yield to robust model, whose quantized version is malicious. This indicates that it should be ensured, that the quantized version of a model is sufficiently robust against adversarial attacks, before its public release (e.g. on Hugging Face).

To examine the impact of model quantization, we train several models without 4-bit quantization and evaluate both the base and 4-bit quantized versions of them. The results (Table 6) suggest, that models trained without 4-bit quantization are less robust than the models which were trained in 4-bit quantized form (higher ALO-ASR).

B.4 Evaluating on more samples

In all our experiments we evaluate the models on a subset of 40 samples from the HarmBench test set. To assess the generalization of our models, we evaluate some of the Zephyr-7B models on the full test set of the HarmBench dataset. However, we still exclude the copyright samples as before, and we have a total of 240 samples. Since generating GCG attacks for all 240 samples would take 6 days on a single A100 GPU for each evaluated model, we only evaluate the models using the other attacks. The results in Table 7 show that the models achieve similar robustness scores on the full test set as on the subset used in the main experiments. This indicates that evaluating on the 40-sample subset is a good proxy for the full HarmBench test set.

We also evaluate the robustness of the Zephyr models on a subset of 40 samples from the harmful prompts of the XSTest dataset. The results in Table 8 show that the general robustness trends are similar across the two datasets, when considering the same attack.

B.5 Transferability of attacks

Prior work [4] showed that adversarial attacks generated for non-adversarially-trained models transfer well to many other models. Here, we want to assess if adversarial attacks generated for adversariallytrained models exhibit the same trend. To do this, we generate adversarial samples using TAP, PAIR,

and GCG attacks for different Zephyr-based models - undefended Z-Base, as well as, Z-CAT, and our Z-MixAT. We evaluate these sets of generated attacks on a large set of models both based on Zephyr and Llama3 and compare their efficiency versus Direct Requests and attacks specifically generated for the models. The results in Table 9 show that almost all transferred attacks achieve worse performance compared to their targeted counterparts, with some transferred attacks being even less successful than the direct requests. Further, MIXAT shows incredible resilience against all adversarial attacks, regardless of their origin.

B.6 Utility benchmarks without chat template

Recent work highlights a failure mode in previous works, that evaluating LLMs on robustness and utility benchmark inconsistently with or without chat template might produce unrealistic results [7], since in real-world scenarios the settings of text generation do not depend on the content of prompts or whether they are malicious or ordinary requests. To enhance consistency, in this section we present ARC-E, ARC-C and MMLU results of LLama3 and Zephyr-7B with their default chat template Table 10.

In the case of Zephyr-7B [27] adapters, our Mix-AT utility scores are still competitive compared to other models, which have significantly worse robustness results. Furthermore, our Mix-AT adapter still has higher utility than the base model, meaning maintaining good robustness. Our method compared to CAT adapters, has much higher utility (0.1 more ARC-E and 0.05 more ARC-C), demonstrating the power of short schedule trainings in the case of Zephyr models.

On the other, Llama3 [28] Mix-AT also maintains a good utility despite being evaluated under chat template. However, the utility gain from short schedule training is not so significant.

B.7 MIXAT on other models

To demonstrate the generalizability of our method, MIXAT has been evaluated on additional models. We aimed to assess MIXAT on an adversarially less robust (Mistral-7B[36]) and more robust

(Llama3.1-8B [28]) base models. The evaluation results are in Table 11. These are similar to our original findings based on Zephyr-7B, LLama3-8B and Qwen2.5 models. We observe that MIXAT drastically increases the robustness of the base model in all of the cases, with a slight drop in utility.

B.8 MIXAT Static training ablation results

We present the results of opur ablation using statically generated adversarial attacks in the training procedure of MIXAT. The evaluation results are in Table 12. We observe that while the static model has slightly higher utility, this comes at the cost of much worse robustness. This reaffirms the effectiveness of MIXAT's dynamic attack component.

B.9 Evaluating MIXAT against other attacks

MIXAT is evaluated against other attacks, including BEAST [40] and I-FSJ [41]. The results are in Tables 13 and 14, respectively. We observe that MIXAT is robust against these attacks as well. This demonstrates that even though MIXAT is trained using a combination of PAP and continuous attacks, the robustness properties generalize to a wide range of attacks.

B.10 MIXAT + GCG with static GCG prompts

Since generating GCG attacks on-the-fly during training requires significant time, we further experimented with using static GCG examples created prior to the training for this purpose. Statistically generating all the GCG samples for the whole training set takes 33.5 hours on a single H200 GPU for Llama3-8B (36.6 hours for Zephyr), which makes this singular experiment significantly more expensive than our dynamic MIXAT + GCG training. On the other hand, once we have generated the static GCG samples, we can run multiple training runs incorporating them. Our results in Table 17 show that incorporating static GCG samples into MIXAT training indeed shows similar benefits with the dynamically generated ones, providing an interesting avenue for future model training (datasets).

B.11 MIXAT with GCG as discrete attack

To assess how PAP contributes to the success of MIXAT, we additionally train Zephyr and Llama3 MIXAT models with statically generated GCG discrete attacks as seeds instead of PAP (using dynamically generated GCG samples would make the training too long and expensive). The results are in Table 18. Our results demonstrate that using GCG samples as seeds for continuous attacks can improve the model's robustness over the CAT baseline, but the lower diversity of GCG samples is detrimental to the generalization of robustness properties compared to rephrasing attacks such as PAP, TAP, and PAIR. In our MIXAT + GCG ablation, we have shown that combining PAP, GCG, and Continuous attacks can further improve robustness, suggesting that the diversity of the attacks used for training is crucial for broad robustness. On the other hand, the comparison between MIXAT and our baseline DualAT shows that the stronger mixed (discrete + continuous) attacks are better for training than simply combining the two training losses in a multi-objective manner.

B.12 Varying the hyper-parameters of the continuous PGD attack

To validate our hyperparameter choices, we show how scaling the ϵ (0.5x, 2x), the number of steps (0.5x, 2x), and the step-size (0.5x, 2.0x) of the continuous PGD attacks used in MIXAT affects the robustness and performance of the trained models. Note that, for a fair comparison, we scaled the step-size with the perturbation budget and inversely with respect to the number of steps. The results are shown in Table 19. Overall, we observe consistent results on the utility benchmarks, with only minor, to be expected, deviations from the original values. For robustness, this trend continues;

however, we find that generally the chosen tradeoff of original MIXAT performs on the top end of all tested configurations, with especially high-step low learning rate experiencing a stark drop in ALO robustness.

B.13 Robustness against fundamentally different attack modalities

To show the generalization of MIXAT defense to entirely different types of attacks (i.e., neither classical adversarial examples (GCG) nor paraphrasing/role-playing jailbreaks (TAP, PAP, HumanJailbreak)), we conducted two additional experiments - one using code attacks from the QueryAttack dataset [42] on Qwen 2.5 14B models; and one using ASCII art-based attacks from the ArtPrompt dataset [43] on Llama-3 8B models. We chose the model families per attack so that the attacks were particularly potent against the base models, allowing us to compare the defense methods meaningfully. The results are shown in Table 15 and Table 16, highlighting that all defense methods significantly improve robustness against these different attack modalities, with MIXAT consistently achieving the best robustness.

B.14 Examining malicious requests

Our main intuition for the increased effectiveness of MIXAT training lies in the fact that the combination of rephrasing and continuous attacks can cover a larger portion of the adversarial space. To illustrate this, we have used an LLM2Vec [44] model built on top of Llama3 to extract embeddings for some malicious requests, as well as GCG, PAP, and continuous attacks targeting these requests. We illustrate these examples in Table 20 and their pairwise cosine similarities in Fig. 6. We observe that combining PAP and continuous attacks results in samples that are further away from the original prompt than each individual attack, confirming our hypothesis that combined attacks are stronger and can explore a wider section of the adversarial space, while still being close enough that they are useful for training.

DR

DR+C

GCG

GCG+C

PAP

PAP+C

DR

DR+C

GCG+C GCG

PAP

PAP+C

1.00

0.90

0.88

0.86

0.80

0.79

0.90

1.00

0.88

0.90

0.75

0.83

0.88

0.88

1.00

0.96

0.79

0.84

0.86

0.90

0.96

1.00

0.75

0.84

0.80

0.75

0.79

0.75

1.00

0.93

0.79

0.83

0.84

0.84

0.93

1.00

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

1

.

00

Cosine Similarity

C Additional Experimental Details

C.1 Datasets and Models

Training Data We train our models using the same harmful requests as Xhonneux et al. [7], which are similar to the ones in the HarmBench dataset [5]. The safe answer used during adversarial training is always ' Sorry, I can't do that. ' To maintain utility, we use the UltraChat200k [45] utility dataset, used in both adversarial training [7] and general fine-tuning [27, 29, 46]. Due to memory constraints, UltraChat200K is filtered to samples with fewer than 256 tokens.

Further in the malicious dataset, there are 100 harmful queries alongside multiple unpreferred continuations (1188 in total), which we together consider the harmful part of the dataset. The dataset is sampled with the interleave_dataset function from the HuggingFace datasets library, with probabilities 0.875 (benign) and 0.125 (harmful), and with strategy first_exhausted. So the ratio of benign and malicious queries is around 7:1. With our fixed seed, the benign part consists of 8476 samples sampled from UltraChat200K.

Other Datasets In Table 21 we list the licenses of the different datasets used in this paper.

Models Used In Table 22 we list the sources of the Hugging Face models and adapters used and evaluated in this work.

C.2 Training Details

Unless stated otherwise, we follow the design and hyperparameter choices of prior work [7]. By default, models are 4-bit quantized and aligned using LoRA adapters [37] (we examine the non-quantized models in B.3). We use 10-step L 2 -bounded continuous adversarial attacks with ϵ = 0 . 075 , and discrete PAP attacks. The default PAP sample ratio is α = 0 . 5 , with paraphrases drawn randomly from all 40 strategies [2]. We train

for 2 epochs (in contrast to 5 in CAT) with a batch size 64 , a learning rate of 2 e -4 , the AdamW optimizer [47], and a cosine learning rate scheduler.

We note that a bug in the original CAT code was brought to our attention by the authors of Xhonneux et al. [7] around the time of publication of this work. In particular, the authors pointed to us that their code was not properly deactivating model weights during the adversarial attack generation phase. This results in unintended gradient accumulation for the model parameters during the attack, potentially impacting training dynamics. The reported results for both CAT and MixAT training are on models trained with the code that contains the bug. Prompted by their notification, we locally fixed this bug but did not observe any significant performance differences between training with and without the bug in our tests.

C.3 Inference hyperparameters

To enhance reproducibility, we report the different hyperparameters for adversarial attacks used throughout the evaluations in Table 23. For generating adversarial attacks we use the HarmBench GitHub repository [5]. We note our Harmless (Hless) evaluation scores were computed using double quantization (bnb_4bit_use_double_quant=True in HuggingFace), unlike all other benchmarks.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We describe MIXAT in detail in Section 3, and show experimentally in Section 4 that MIXAT exhibits better accuracy-robustness tradeoff compared to prior work (Table 1). We also demonstrate that MIXAT is more computationally efficient compared to training with discrete attacks in Table 2. Finally, we investigate how practical properties of the LLM inference such as usage of LoRA weights (Fig. 4), temperature (Fig. 5), and quantization schemes (Appendix B.3) affect MIXAT. This is in line with our claims in our abstract and introduction sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we discuss our limitations in Section 5, further describing open challenges in the research area of adversarial robustness of LLMs as whole in Section 6. We, in particular, emphasize the importance of solving the randomness of both robustness training and evaluation as two major factors of inhibiting research progress in the area.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not provide theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide both our training and evaluation code to facilitate reproducibility on GitHub. Further, we provide the weights of our trained models on HuggingFace. We also describe MIXAT and its hyperparameters in detail in Section 3 and Section 4, respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly available datasets for both the training and the evaluation of our MIXAT models, as described in Section 4. Further, we provide our code for training and evaluation on GitHub and provide access to the models themselves through HuggingFace.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide both of our training and evaluation code to facilitate reproducibility on GitHub. We also describe MIXAT and its hyperparameters in detail in Section 3 and Section 4, respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide discussion on randomness of both evaluation and training alongside error bars in Appendix B.2. As explained in Appendix B.2, we do not provide error bars for other experiments in the paper as the experiments are very computationally heavy and running them enough times to provide statistically significant error bars is computationally infeasible for us.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We compare our required computational resources, including the GPU runtime and VRAM to prior work in Section 4.7.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper does not introduce any new harmful datasets, instead relying on existing ones. Further, the models and training methods introduced in the paper increase security of state-of-the-art LLMs, and, thus, do not pose any additional security risks. Further, we provide an impact statement at the end of the paper, discussing the possible societal benefits of our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide an impact statement as Appendix A, discussing the possible societal benefits related to MIXAT. As we do not introduce any new harmful datasets or models, we do not envision any negative impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not introduce any new harmful datasets, instead relying on existing ones. Further, the models and training methods introduced in the paper increase security of state-of-the-art LLMs, and, thus, do not pose any additional security risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite all of our datasets used and provide a list of datasets and their licenses in Table 21.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper releases code, as well as, model weights which are both extensively documented on GitHub and HuggingFace, respectively.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not conduct experiments involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not conduct experiments involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were only used for editing of the manuscript but were not responsible for generating any technical content and did not contribute to the methodology proposed in this work.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSome past works do LLM adversarial training using discrete adversarial inputs. These are less efficient but more on-distribution for many types of real-world attacks. Others use continuous adversarial inputs. These are more efficient, but more off distribution for many real-world attacks. This paper asks if we just did discrete and then continuous attacks one after another for adversarial training. They introduce this method, analyze cosine similarity by the embeddings produced by their attacks, and test their method on a bunch of models and evals. They also do ablation tests and some other useful experiments on quantization, transferability, etc.\n\nSTRENGTHS AND WEAKNESSES:\nS1: I think it's a great idea for a paper. I hope I don't see any other reviewers complain and make assertions about a lack of novelty. I work closely in the space, and I am very familiar with the methods and goals of the paper, and I think this type of work is nontrivial, non-obvious, good, and needed. \n\nS2: Not a substantive thing -- but just a comment on design. I really like figure 1 and the fuzzy text. \n\nS3: Pretty good diversity across experiments in model, size, evals, and baselines. I can tell this was a ton of work by the authors. \n\nS4: Well written\n\nS5: I like the analysis of costs in table 2. \n\nS6: I think this paper overall does very well what it sets out to do -- making a strong case that MixAT approaches belong in the SOTA toolbox for making more robust LLMs. I think the paper is of really clear value, and I would be willing to go to bad for this paper. \n\nW1: There is a claim in the abstract that, to my knowledge, is generally believed to be false: \"As these relaxations do not correspond to discrete input tokens, such latent training methods often leave models vulnerable to a diverse set of discrete attacks.\" All three of the papers cited in the \"Continuous Adversarial Training\" paragraph of related works argue all give evidence against this claim. It's good and fine to hypothesize that discrete and continuous adversarial training, complement each other, and it's also good and fine to hypothesize that discreet adversarial training on a certain set of attacks is likely to improve generalization to similar input-space attacks, but I know of no evidence to suggest that training on latent/embedding attacks results in poor generalization due to the attacks not corresponding to actual tokens. \n\nW2: In section 2.5, I recommend citing recent work from anthropic on constitutional classifiers. \n\nW3: In fig 6, please extend the caption to explain the acronyms and fig. Captions should be stand-alone explanations, not just labels. See also figure 4 from [Che et al. (2025)](https://arxiv.org/abs/2502.05209) who do different but closely related analysis of the similarity of different attacks. It could also be useful related work to discuss in section 2.2. \n\nW4: This might be a major pain in the ass, but consider replacing table 1 with heat-colored text or a heatmap. As is, I have to stare at it too long to get the jist. And I buy the topline result that, overall on average, Mixat is good (I hope no other reviewers complain about how it doesn't pareto dominate), but it would be much easier from a heat-coded version. \n\nW5: Consider also adding two more columns to table 1 giving the mean delta for utility score over baseline and the mean delta for ASR below baseline. If you don't color-code the entire table, you could at least colormap the values in these new columns.\n\nQUESTIONS:\nNone in particular. I think I have a good understanding of this type of paper, and I think this particular paper is very clear.\n\nLIMITATIONS:\nMixAT is really more of a family of methods than a single method. It looks like this paper has good success with PAP + CAT, but there are a lot more possibilities.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes MIXAT, a hybrid adversarial training strategy for large language models that combines discrete prompt attacks (e.g., PAP paraphrasing) and continuous embedding space perturbations. The authors introduce the At Least One Attack Success Rate (ALO-ASR) metric to quantify worst-case vulnerability and report that MIXAT achieves substantially lower ALO-ASR (< 20%) than prior methods (e.g., CAT, R2D2) while maintaining comparable utility across benchmarks (e.g., ARC, MMLU)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The motivation and problem formulation of this paper are quite clear. The paper identifies key limitations of purely discrete versus purely continuous adversarial training and formulates a unified perturbation set in Eq. (8).\n\n2. This paper provides a comprehensive evaluation across a diverse suite of attacks (PAP, TAP, GCG, AutoDAN, etc.) and multiple LLM backbones, using both standard benchmarks and the proposed ALO-ASR metric.\n\nWeakness:\n\n1. MIXAT simply merges two well-known ideas—discrete adversarial training (e.g., PAP-AT [2]) and continuous embedding perturbations (e.g., CAT [7])—without introducing new attack or defense techniques. The main contribution is their combination, which may limit its novelty to current LLM robustness research.\n\n2. In Section 4.3, the authors report a “MIXAT + GCG” variant that mixes GCG attacks during training to further reduce ALO-ASR. However, this raises fairness concerns: most baselines are not allowed to train on GCG, so the comparison becomes uneven. \n\n3. Different attack bounds. The author only reported the performance under certain attack hyperparameters. Tighter or looser attack bounds are suggested to enhance the experiments.\n\nQUESTIONS:\nTable 1 reports a long list of individual utility and ASR results across multiple tasks and attacks, but lacks an average performance column (e.g., mean utility or mean ASR).\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an improvement to adversarial training for LLMs. Existing adversarial training works either optimize the adversarial perturbation in either the token embedding space, the latent space of the model, or the discrete tokens that are input to the model. This results in a lack of robustness to a particular type of attack for each of these methods. Consequently, this work, MixAT, suggests combining both discrete and embedding-space attacks during adversarial training. The authors provide convincing evidence that MixAT achieves both higher robustness and higher utility than the baseline defenses.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The knowledge around adversarial training for LLMs has been sparse despite its importance in practice. I believe that this study contributes substantially to this body of knowledge, showing benefits in combining more than one types of attacks.\n2. The experiments are comprehensive and convincingly show the improvement from MixAT across different base models and hyperparameters. I appreciate the ablation studies and the detailed reports on multiple utility benchmarks, multiple attacks, effects of randomness, as well as the runtime/computation cost. I also like the comparison between static and dynamic AT.\n3. Overall, the presentation is good; the paper is well-written. There are lots of numbers to parse in Table 1 and in the paper so I would also appreciate some simplifications, but overall, I have no trouble understanding and learning from the paper.\n\n## Weaknesses\n\nThe paper has no glaring weakness in my opinion, but I do have several questions in the next section.\n\n1. The biggest point I have to say is in the novelty of the method. MixAT essentially enhances a well-known adversarial training algorithm by simply combining two different types of attacks against LLMs. That said, the paper has made up for this weakness very well by figuring out many tiny technical pieces that make this defense work (and also report the ablations) and by showing thorough empirical results.\n2. Another small point is that MixAT does not perform well on XSTest but is comparable to the baselines. When measuring utility, I’d put more weights on XSTest than generic benchmarks that do not test safety capabilities of the model.\n\nQUESTIONS:\n1. There is a comparison between MixAT and static MixAT where the attacks used during training are supposed to be pre-computed and not generated on the fly. In my understanding, PAP can already be computed and is not dependent on the target model. So the only part that becomes static is the continuous AT (i.e., adversarial embeddings are computed once on the base model alone)? I’m quite surprised this even works at all. Does this work because you do LoRA and so the embedding table does not change?\n2. Continuing from the first question, I wonder what happens if you also freeze the GCG suffixes, generating them only on the base model, and use it to train MixAT + static GCG. MixAT + dynamic GCG seems to be very good but suffers from high computation cost. I wonder if using static GCG finds a good trade-off.\n3. How do you evaluate a response as harmful or not? Do you use one of the classifier that comes with HarmBench?\n4. L315 mentions that “the quantization slightly improves the robustness.” Do you have any explanation for this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MIXAT, a novel adversarial training (AT) method designed to enhance the robustness of Large Language Models (LLMs) against jailbreaking attacks. The authors identify a critical trade-off in existing AT methods: discrete, token-level attacks (e.g., GCG, PAP) are effective but computationally prohibitive, while continuous, embedding-space attacks (e.g., CAT) are efficient but fail to generalize to realistic, discrete threats.\n\nMIXAT elegantly bridges this gap by proposing a two-stage adversarial sample generation process. First, it applies a low-cost discrete attack (specifically, PAP-based rephrasing) to a harmful prompt to generate a \"seed\" adversarial example. Second, it applies a continuous, embedding-space perturbation on top of this discrete seed. This combined approach aims to explore the adversarial space more effectively and efficiently.\n\nThe authors conduct a comprehensive evaluation on several state-of-the-art LLMs (Zephyr, Llama3, Qwen) and a wide range of attacks. They introduce a new metric, \"At Least One Attack Success Rate\" (ALO-ASR), to measure worst-case vulnerability. The results compellingly show that MIXAT achieves a superior robustness-utility trade-off, significantly reducing ALO-ASR to <20% compared to baselines (>50%), while maintaining high utility and incurring minimal computational overhead compared to discrete methods. The paper further provides insightful analyses on the effects of quantization, LoRA scaling, and temperature on model robustness.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The core idea of MIXAT—centering continuous perturbations around discrete adversarial seeds (R(X) + δ)—is both simple and remarkably effective. It provides a principled way to combine the strengths of both discrete and continuous AT. The comparison against the simpler DUALAT method (R(X) ∪ B(x, ε)) successfully demonstrates that the specific formulation of MIXAT is crucial to its success.\n2. The experimental validation is a major strength of this paper and sets a high standard for future work in this area.\n\n## Weaknesses\n\n1.  The paper primarily uses PAP for generating the discrete seeds, citing its low cost and effectiveness. However, the performance of MIXAT could be dependent on the quality and diversity of these seeds. It would be beneficial to include a brief discussion or a small-scale experiment on how the choice of the discrete attack method (e.g., using a few steps of a different, non-rephrasing attack as a seed) impacts the final robustness. This would help clarify whether MIXAT's success is tied specifically to PAP or to the general principle of discrete seeding.\n2. The authors perform extensive ablations on the mixing ratio α (discrete vs. continuous). However, the parameters of the continuous PGD attack itself (e.g., perturbation budget ε, number of steps, step size) are taken from prior work. Given their importance, a brief sensitivity analysis on these hyperparameters would be valuable. For instance, does a larger ε or more PGD steps in MIXAT yield diminishing returns or even harm utility?\n3. The paper demonstrates strong generalization against attacks not seen during training. However, many of these attacks (TAP, PAIR, AutoDAN) fall within the general category of prompt-level modifications. It would be interesting to test MIXAT's robustness against fundamentally different attack modalities, such as those based on role-playing, complex scenarios, or attacks that use non-English languages or encodings, which are not directly related to paraphrasing. This is more of a suggestion for future work but would strengthen the claims of generalizability.\n\nQUESTIONS:\n1. Could you elaborate on the intuition for why the combined perturbation space (R(X) + δ) is so much more effective than the union of the two spaces used in DUALAT? The paper states it \"better covers the adversarial embedding space,\" but a deeper explanation of the underlying geometry or optimization landscape would be insightful.\n2. Regarding the inclusion of GCG samples in MIXAT + GCG, the training cost increased 5x. Was this increase primarily due to the GCG sample generation, or did the training on these combined embeddings also converge more slowly?\n3. In Figure 4, the utility of MIXAT-trained models is consistently higher than or equal to CAT-trained models across different LoRA scaling factors λ, even though both are being fine-tuned. Do you have a hypothesis for why MIXAT's training regimen is less detrimental to model utility compared to CAT?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}