Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence
Authors: Shaopeng Fu
Liang Ding
Jingfeng Zhang
Di Wang
Venue: NeurIPS 2025 poster ()
Keywords: In-context Learning
Adversarial Training
Jailbreak Attacks
Large Language Models
Learning Theory
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We find that it is effective to defend "long-length" jailbreak attacks via efficient "short-length" LLM adversarial training, supporting by both theoretical and empirical evidence.

Abstract: Jailbreak attacks against large language models (LLMs) aim to induce harmful behaviors in LLMs through carefully crafted adversarial prompts. To mitigate attacks, one way is to perform adversarial training (AT)-based alignment, i.e., training LLMs on some of the most adversarial prompts to help them learn how to behave safely under attacks. During AT, the length of adversarial prompts plays a critical role in the robustness of aligned LLMs. While long-length adversarial prompts during AT might lead to strong LLM robustness, their synthesis however is very resource-consuming, which may limit the application of LLM AT. This paper focuses on adversarial suffix jailbreak attacks and unveils that to defend against a jailbreak attack with an adversarial suffix of length $\Theta(M)$, it is enough to align LLMs on prompts with adversarial suffixes of length $\Theta(\sqrt{M})$. Theoretically, we analyze the adversarial in-context learning of linear transformers on linear regression tasks and prove a robust generalization bound for trained transformers. The bound depends on the term $\Theta(\sqrt{M_{\text{test}}}/M_{\text{train}})$, where $M_{\text{train}}$ and $M_{\text{test}}$ are the numbers of adversarially perturbed in-context samples during training and testing. Empirically, we conduct AT on popular open-source LLMs and evaluate their robustness against jailbreak attacks of different adversarial suffix lengths. Results confirm a positive correlation between the attack success rate and the ratio of the square root of the adversarial suffix length during jailbreaking to the length during AT. Our findings show that it is practical to defend against "long-length" jailbreak attacks via efficient "short-length" AT. The code is available at https://github.com/fshp971/adv-icl.

Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence
Shaopeng Fu' Liang Ding Jingfeng Zhang Di Wang"*
'King Abdullah University of Science and Technology The University of Sydney The University of Auckland shaopeng- fužkaust edu. sa, liangding Liangding.Liam@gaae www.mmmmmmmm.....mm.mmmmmmmm...m com jingfeng zhangdauckland. nz, di. wanggkkauss edu.
Abstract
Jailbreak attacks against large language models (LLMs) aim to induce harmful behaviors in LLMs through carefully crafted adversarial prompts. To mitigate attacks, one way is to perform adversarial training (AT)-based alignment, i.e. training LLMs on some of the most adversarial prompts help them learn how behave safely under attacks. During AT, the length of adversarial prompts plays critical role the robustness of aligned LLMs. While long-lengte adversarial prompts during AT might lead to strong LLM robustness, their synthesis however is very resource-consuming, which may limit the application of LLM AT. This paper focuses on adversarial suffix jailbreak attacks and unveils that to defend against jailbreak attack with an adversarial suffix of length +
enough align LLMs on prompts with adversarial suffixes of length @(VM). Theoretically, we analyze the adversarial in-context learning of linear transformers regression tasks and prove a robust generalization bound for trained transformers. The bound depends on the term 6 D(VMes/MMnnn the numbers of adversarially perturbed in-context samples during training and testing. Empirically, conduct AT popular open-source L.MSs and evaluate their robustness against jailbreak attacks of different adversarial suffix lengths. Results confirm a positive correlation between the attack success rate and the ratio of the square root of the adversarial suffix length during jailbreaking to the length during AT. Our findings show that it is practical defend against "long- length" jailbreak attacks via efficient "short-length" AT. The code is available at https: //gi gigghub.b... com/fshp971 1/ addv-icl.
linear
where Mininn and Mies are
Introduction
Large language models (LLMs) [5, 51, 28, 65) are widely adopted in various real-world applications assist human users [55, 67, 57, 56, 24], but their safety is found to be vulnerable toward jailbreak attacks [60]. With carefully crafted adversarial prompts, one can "jailbreak" the safety mechanism of
LLMs and induce arbitrary harmful behaviors [73, 7, 30). To tackle the challenge, recent studies [63, 36, 68, 6] propose performing safety alignment through adversarial training (AT) [32] to enhance LLMs robustness against jailbreaking. standard AT for LLMs would train them on jailbreak prompts synthesized by strong jailbreak attacks to learn to refuse these harmful instructions [36].
In such AI, the length of synthesized adversarial prompts used for model training is critical the final jailbreak robustness of LLMs. [3] and [64] have shown that longer adversarial prompts enjoy stronger
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

jailbreaking abilities. Thus, it is reasonable to deduce that performing AT with longer adversarial prompts can help LLMs achieve stronger robustness to defend against "long-length" jailbreak attacks. However, synthesizing long-length adversarial prompts in adversarial training is resource-consuming since it requires solving discrete optimization problems in high-dimensional spaces, which thus needs lots of GPU memory and training time. This may limit the application of AT in L.MS' safety alignment and further raises the following research question: How will the adversarial prompt length during AT affect trained LLMs' robustness against jailbreaking with different prompt lengths?
We study this research question by analyzing suffix jailbreak attacks, where each jailbreak prompt is constructed by concatenating a harmful instruction with synthesized ad adversarial suffix. Our main finding is: To defend against a suffix jailbreak attack with suffix length of (MM), is enough to adversarially train LLMs on adversarial prompts with suffix length of only O MMM. other words, we show that it is possible to defend long-length jailbreak attacks via efficient short-length AT.
Our finding is supported by theoretical and empirical evidence. Theoretically, we leverage the in-context learning theory [53, 69] to investigate how linear transformers learn linear regression tasks from in co-context task samples under AT. To better simulate suffix jailbreak attacks in real-world LLMs, our analysis introduces new in-context adversarial attack. Concretely, for any in-context task sample, this attack will adversarially perturb the last several in-context training points to maximize the squared prediction error that linear transformers made on the in-context test point. Under our theoretical framework, we prove robust generalization bound for adversarially trained linear transformers. This bound has positive correlation with the term O Mes Munin), where Muain and Miet are the number of perturbed in-context points in training and testing in-context task samples, respectively.
Empirically, we conduct AT with the GCG attack [73], one of the most effective jailbreak attacks, under various adversarial suffix lengths on five popular real-world LLMs and evaluate their robustness against jailbreak attacks with different adversarial suffix lengths. We use the jailbreak attack success rate (ASR) to express the robust generalization error of trained LLMs and find that this ASR has a clear positive correlation with the ratio of the square root of test-time adversarial suffix length to the AT adversarial suffix length. Such a correlation empirically verifies our main finding. We also find that AT with an adversarial suffix (token) length of 20 is already able to reduce the ASR of jailbreak attacks with an adversarial suffix (token) length of up to 120 by at least 30% in all experiments.
2 Related works
Jailbreak attacks. Jailbreaking [60] can be seen as adversarial attacks [49, 14] toward LLMs, which aim to synthesize adversarial prompts to induce targeted harmful behaviors from LLMs. Many efforts have been made on token-level jailbreak attacks, i.e., searching adversarial prompts in the token space of LLMs, which can be achieved via gradient-based optimization [48, 16, 73, 26, 45, 71], heuristic greedy search [44, 17, 22], or fine-tuning prompt generators from pre-trained LLMs [38]. Other attempts include word-level adversarial prompt searching [30] or directly prompting LLMs to generate adversarial prompts [7, 29]. Our work focuses on token-level jailbreaking since it make it easier for to control the adversarial prompt length for our analysis. More recent studies have found that increasing the length of adversarial prompts by adding more harmful demonstrations [3, 54, 61] or synthesizing longer adversarial suffixes [64] can make jailbreaking more effective. These works motivate us to investigate the problem of defending against "long-length jailbreak attacks.
Adversarial training on LLMs. To defend against jailbreak attacks, large body of studies focus on aligning LLMs to refuse responding jailbreak prompts [37, 42, 40, 41, 8). More recent works have started to adopt adversarial training (AT) [32] to align L.M.s. [36] trained L.MSs on (discrete) adversarial prompts synthesized by GCG attack [73], in which they cached the intermediate synthesized results to reduce the heavy cost of searching adversarial prompts from scratch. Mcanwhile, various studies [63, 6, 46, 68] conduct AT with adversarial examples found in the continuous embedding space rather than the discrete text space since searching in the continuous embedding space is more computationally efficient. Nevertheless, as a preliminary study of the length of adversarial prompts during AT, our work only analyzes AT with discrete adversarial prompts.
In-context learning theory (ICL). Transformer-based large models like LLMs are strong in per- forming ICI: Given a series of inputs (also known as "prompt") specified by a certain task, LLMs can make predictions well for this certain task without adjusting model parameters. Current theories in understanding ICL can be roughly divided into two categories. The first category aims to under-

---

stand ICL via constructing explicit multi-layer transformers to simulate the optimization process of learning function classes [13. 53, 1, 9, 34, 59]. The second category focuses on directly analyzing the training [69, 66, 19, 62, 27] and generalization [31, 33, 11, 47] of simple self-attention models (i.e., one-layer transformer). [4] the first to study adversarial attacks against linear transformers and finds that an attack can always succeed by perturbing only single in-context sample. However, their analysis allows samples to be perturbed in the entire real space, which might not appropriately reflect real-world settings since real-world adversarial prompts can only be constructed from token/character spaces of limited size. Unlike [4], we propose a new ICL adversarial attack that requires each adversarial suffix token to be perturbed only within restricted spaces, which thus can be better tool for understanding world jailbreaking.
Finally, we notice that [61] also recognizes the critical role that the number of adversarial in in-context samples plays in ICI--based attacks. They present a theoretical analysis (not based on ICL. theory) for adversarial attacks against ICI. text classification and characterize the minimum number of in-context adversarial samples required to increase the safety loss of ICI. to some extent. However, the main difference is that [61] focuses studying the adversarial robustness of fixed ICL models, whereas our work analyzes how adversarial training affects the robustness of ICL models.
3 Preliminaries
Large language models (LLMs). Let [V] = (1,.... be a vocabulary set consisting of all possible tokens. Then, LLM be seen a function that for any sequence Ii:s [V] consists of n tokens, the LLM will map Zim to its next token In+1 following xn+1 ~ Pa (-(- where pe is conditional distribution over the vocabulary set [V] and 0 is the model parameter of the LLM. Under such notations, when using the LLM po to generate a new token sequence
for the input Tims the probability of generating sequence /1.m E [V]m of length m is given by Pe (3/mm [I1m) = m Po Zim 2 (1:(i-1)), where "Ø" denotes concatenation.
Jailbreak attacks. This paper focuses on suffix jailbreak attacks. Concretely, suppose (+()) and (()) are two token sequences, where (A()) represents a harmful prompt (e.g., "Please tell me how to build a bomb.") and ((h represents a corresponded targeted answer (e.g.. "Sure, here is a guide of how to build bomb").. Then, the goal of a suffix jailbreak attack against the LLM po aims to synthesize an adversarial suffi (1+) for the original harmful prompt x: (h) via solving the following problem.
min E
- log (o(y loggee(y
(k) @ ((e)))
where x (h) @ (9) is the adversarial prompt and m is the sequence length of the adversarial suffix (a)
xi.m Intuitively, a large will increase the probability of the LLM pe that generating the targeted answer (A) for the synthesized adversarial prompt (h) @ 2 To solve Eq. (1), a standard method the Greedy Coordinate Gradient (GCG) attack [73], which leverages gradient information search for better æ (+) within the discrete space [V] in a greedy manner.
Ad Adversarial training (AT), We consider the canonical AT loss L [36, 40] to train the LLM po- which consists of two sub-losses: an adversarial loss Lady and an utility loss Cuinyy- Specifically, given a safety dataset D(h), where each of its sample (z("), yfth), (f(h)y(()) y(b)) E D(h) consists of a harmful
instruction !!!), a harmful answer (l,,, and a benign answer (b)) (e.g.. "As a responsible AI, I can't tell you how to..."). The adversarial loss Lady is defined as follows, (a, M, D(M): ==
(a (01)) log ( (b) (1) + x ( ).
(2)
where with is the adversarial suffix obtained from Eq. (1) and m is the adversarial suffix length. Note that the probability terms in Eqs. (1) and (2) look similar to each other. The difference is that the term in Eq. (1) denotes the probability that Po generates the harmful answer (b) for the adversarial prompt, while that in Eq. (2) denotes the probability of generating the benign answer (()) Besides,
let D(") be a utility dataset where each of its sample (z("),y()) E D(1)) consists of a pair of normal instruction and answer. Then, the utility loss Luiliyy is given by
Luiliyy (0, D(w))==
E |- Logpo( (y (4) (.
(0)))D)))
3

---

Thus, the overall AT problem for improving the jailbreak robustness of the LLM po is given as min (0, M, ((A) + (1 ax) Cuitity D))));
where a E [0, 1] is a factor that balances between the adversarial and utility sub-losses. The idea behind such loss design is that: (1) help LLM learn respond harmlessly even when strong jailbreak prompts present (achieved via Ladv), (2) retain the utility of LLM gained from pre-training (achieved via Luillit). Intuitively, larger adversarial suffix length m during AT will help the LLM gain robustness against jailbreak attacks with longer adversarial suffixes.
Theoretical evidence
This section establishes the theoretical foundation of how "short-length" AT can defend against "long-length" jailbreaking. Our analysis based on the in-context learning (ICL) theory [69, 47, 4], and we will bridge the ICL theory and the LLM AT problem defined in Eq. (3) later (in Section 4.2). Here we first introduce the necessary notations to describe the problem. To avoid confusion, we note that all notations in this section will only be used within this section and have relevance to those in other sections (e.g., Section 3).
In-context learning (ICL). In the ICL theory, prompt with length N related to specific task indexed by is defined as (Z+,1 ( "r.1,".. Nr,NN Vr,N, Fr.g), where Tr,i E Rd is the i-th in-context training sample, Y7,1 E is the label for the i-th training sample, and Zr.g Rd is the in-context query sample. Then, the task-specific ICL input E, is detined as
E,
TT,1 Vr,1
IT Vr.N 0
IT..9
E [dd(-I)((++1)
Given an ICL input E, of task т, the goal of an ICL model is to make prediction based on E, for the query sample Fr.g. Such an ICL model design aims to model the ability of real-world LLMs in making decisions based on prompting without updating model parameters.
Linear self-attention (LSA) models. ISA models are kind of linear transformer that has been widely adopted in existing theoretical ICL studies. [2] empirically show that LSA models share similar properties with non-linear ones and thus useful for understanding transformers. We follow [69] to study the following single-layer LSA model,
E B. E
S.SA,o(E)) E + BWWVE E.
E (dd+1))(N+1)
N
where 0 == (WV, WKQ is the model parameter, W ((d+1)x(d+1) is the value weight matrix, WKQ E ((d++))(d++)) is a matrix merged from the key and query weight matrices attention models. E, R(++1)x(N+1) is the task-specific ICL input, and N is the prompt length. The prediction Dq.0 for the query sample Zr,g is given by the right-bottom entry of the output matrix of the L.SA model, iee, 0g Fez) == o Fr (+)((4))(N++)) We further follow [69] denote that
W (w=2)
W
W
R(d+1)x(d+1).
Wiza
where E (V, KQ), WH E Raxd, w wiz E Rdx1 and whn E RR. Under this setting, the model prediction Vq.0 can be further simplified as follows,
E.
KQ RQ ) (w21
Ng = (EsAo(E-) = ((w)) why
Er.gg
(5)
Other notations. For any n E N+ we denote [n] == (1, 11.... ,.). For any A E R we denote ||Al||.0o == maxisism || 112, ||A]]
2a be the operator norm, and | be the Frobenius norm. For any A E paxx, we denote Tr(AA) == En Ai, We use standard big O notations O(.) and ((-).
4.1 Problem definition for adversarial ICL.
We now formalize the AT problem in ICL with the previously introduced notations. We focus on the linear regression task and introduce a novel ICI. "suffix" adversarial attack, where in-context adversarial points are appended to the end of ICL inputs, analyze the robustness of LSA models.

---

In-context linear regression. For any task indexed by т, we assume that there is task weight w, R drawn from W, (0, La). Besides, for any in-context training point Zr,i (1 N) and the query point 2r,g (see Eg. (4)), we assume that they are drawn from Zr,i. ~ (0, A), where A E Rdxd is a positive-defintt covariance matrix. Moreover, the ground-truth labels of training points Zr,; and the query point Fr.q are given by Vr,i = w. F7,5 and y7,4 w, Zr.qq
ICL suffix adversarial attack. Our novel adversarial attack against ICL models is launched via concatenating (clean) prompt embedding matrices with adversarial embedding suffixes. Specifically, for an ICL input E, of length N (see Eq. (4)), we will form its corresponding adversarial ICL input FT. E (N+++)) by concatenating E, with an adversarial suffix of length M as follows,
X Y
X + Dr ysf
Irra
E ==
(6)
Training Data of Length N
Adverssrial Sultis of Length M
Query Sample From E,
where X, := (Ir,11 ... NT,N) E RdxN and Y, == (Ur.1 ...
Vr,N) E RIxN denote the N original a ( E RRdxM Y = (w ... w E
training samples and labels, and X = (2
RIXM, and A (5+,1 suffix labels, and adversarial perturbations. The clean suffix samples X and labels Y here follow the same distribution as those in-context data in the embedding E,, i.e., T ~ N000 A) and Ve. w.. = hold for every E [M]. For the adversarial perturbation matrix 4,, we require each perturbation 8r,i is restricted within a ball-sphere as ||o,,ill2 S e, where 0 is the perturbation radius. This aims to simulate that in jailbreak attacks, and each adversarial token searched within a token vocabulary set of limited size.
6r,M) E RdxM denote the new M clean suffix samples, clean
The goal of the ICL adversarial attack is to add an optimal suffix adversarial perturbation matrix A, to maximize the difference between the model prediction 8, (E:thr) based on the adversarial ICL input Eqdy and the ground-truth query label Vr,q. We adopt the squared loss to measure such a prediction difference, which thus leads to the robust generalization error for the model PLSA as
Rady (0, () = max
[|A lla. 2 10.00 (E - Vr.gl" 12,
where M is the length of the adversarial suffix and the expectation E.|] is calculated over the randomness of Wy, X,, X and Tr.g. As we aim to understand how the adversarial prompt length in AT would affect the robustness of L.M,, Eq. (7) will also only foeus how the adversarial suffix length M in ICL adversarial attacks would affect the robust generalization error Radv (6, M).
Adversarial in in-context learning. Following previous studies minimax AT [32, 20, 43, 12, 58], here we adopt minimax AT loss to train the LSA model. Concretely, we first use the aforementioned ICL adversarial attack to synthesize adversarial prompts and then update the LSA model based on these adversarial prompts to help the model gain robustness against them. We further assume that the adversarial suffix length is fixed during AT, which thus leads to the following ICL AT problem,
min Lady (0) = min Radly (0, Mmmin) min E FIATIR
max
2 110g (BT.aan) Vv.a 1
(8)
where Ladv (0) = Rady (0, Mimin) is the AT loss in ICI. and Minnn E N+ is the fixed adversarial suffix length during AT. We will perform AT with continuous gradient flow, and further following [69] to make the following assumption on the LSA model parameter initialization.
Assumption 1 (c.f. Assumption 3 in [69].. Let > 0 be a parameter and θ E Radxd be any matrix satisfying |000T F and OA + Odxd- We assume.
(axx 0ax1) WKQ = O1xd
(0OOT 0 1x
0ax1
W (0)
KQ do not contribute to the model prediction 0-)0 Assumption 1 Recall in Eq. (5), WY2 kQ and w/2 thus directly sets them to be zero at initialization. To ensure symmetric initialization, it further sets w (0) and w2z (0) to zero. We will see how Assumption helps simplify the analysis of ICL AT.
.QQ
5

---

4.2 Bridging ICL AT and LLM AT
We now discuss similarities between the ICL AT problem defined in Eq. (8) and the LLM AT problem defined in Eq. (3) to help motivate why ICL AT can be a good artifact to theoretically study LLM AT.
Firstly, in-context inputs (i.e., E, defined in Eq. (4)) for LSA models are similar to prompt inputs for real-world LLMs. If we replace each token in an I L.MM prompt with its one-hot encoding defined over the token vocabulary space, then these one-hot encodings would be similar in-context samples 2, in Eq. (4) since both of them are now "feature vectors". Besides, we note that each in-context label y: in Eq. (4) can be seen as the "next-token prediction label" in real-world LLMs. The main difference is that in LLMs, the i-th token in a prompt can be seen as the i-th input token and the (i 1) next-token prediction label simultaneously, while in LSA models, the i-th in-context input and the (i 1)-th in-context label are explicitly separated into two terms I; and Vi-1-
Secondly, the search for adversarial in-context samples (see Eq. (6) in the ICL suffix adversarial attack is similar to the search for adversarial tokens in suffix jailbreak attacks. We note that each adversarial token in jailbreak prompts be seen as replacing the "padding token". Thereby, from the point of view of one-hot encoding, searching for an adversarial token can thus be seen as applying an f2-norm adversarial perturbation within a radius of v2 to transform the one-hot encoding of the padding token to that of the adversarial token. This process is the same as the search for adversarial in-context samples in the ICL suffix adversarial attack defined in Eq. (7), which would perturb each in-context suffix sample 2,,, within f2-norm ball-sphere under given radius e >> 0.
Thirdly, motivations behind ICI. AT and LLM AT are also similar to each other. Both of the two AT problems aim to enhance models' robustness via training them some synthesized adversarial inputs. The adversarial inputs syntheses in ICL AT and LLM AT are also similar, as both of them aim to make targeted models behave wrongly via manipulating suffixes of input prompts. The difference is that suffix jailbreak attacks are targeted adversarial attacks aimed at inducing LLMs to generate specified harmful content while our ICL attack is an untargeted adversarial attack aimed at reducing the utility of linear regression prediction made by LSA models.
4.3 Training dynamics of adversarial ICL
We now start to analyze the training dynamics of the minimax ICL AT problem formalized in Eq. (8). The main technical challenge is that to solve the inner maximization problem in Eq. (8), one needs to analyze the optimization of the adversarial perturbation matrix A.. However, the matrix A, along with the clean data embedding E, and the clean adversarial suffix (x y=*x) are entangled together within the adversarial ICL input Endy, T. which makes it very difficult to solve the inner maximization problem and further analyze the ICL AT dynamics.
To tackle such challenge, we propose to instead study the dynamics of closed-form upper bound of the original AT loss Cal (0), Formally, we will analyze the following surrogate AT problem:
min cady (0) == min }.
(9)
where čadv (0) = Et= (0) the surrogate AT loss, Lelean
Xx
Y
Nala
ysft
FT.4
0
and
Erlean
EvllenT T Minin Miii
WxkQ !1 (w2) )
2
(0) 2 2 (ww)
w22)
Ir.q
yr.o
N + Misin
(2( = 2e 'M2.
100 E
(N + Murin)
2ea Muain
X ydfx
(0) = (N + MMrann
E
| KQ xT. [1((w])) T
wky)
113
2ca Musin
Xafx saf
W.Q (w
(0) = (N + M
l 112
E
Iy 112
The surrogate AT loss (0) in Eq. (9) is the closed-form upper bound for the original AT loss Cady (0) in Eq. (8), as illustrated in the below Proposition I (see Appendix A.2 for the proof).

---

Proposition (Uniform upper bound for Cad (0)). For the AT loss function (0) defined in Eq. (8) and the surrogate AT loss function (a)v(0) defined in Eq. (9), for any model parameter
0 = (WV, Wk4)or of the LSA model fLSA,6. we uniformly have that: L I LLdv(
This result indicates that when are training the LSA model via solving the surrogate AT problem Eq. (9). we are also reducing the model training loss in the original AT problem Eq. (8). Thus, solving the surrogate AT problem will also intuitively improve the robustness of the model.
Based on our previous analysis, we now turn to study the training dynamics of surrogate AT defined in Eq. (9). To better describe our results, we define two functions r()): N + Raxd and v/(+) : R, both of which depend on the adversarial suffix length M, as follows,
T(M) N + M 11 A+
Tr(A) N M La Rdxd, (M) (N + M)² E R,
M²Tr(A)
(10)
N + M
where N is the prompt length of the original ICI. input E, (see Eq. (4)) and A is the covariance matrix of in-context linear regression samples. The closed-form surrogate AT dynamics of the ISA model fLSA.e is then given in the following Theorem 1 (see Appendix A.3 for the proof).
Theorem 1 (Closed-forn Surrogate AT Dynamics), Suppose Assumption I holds and fLSA. o is trained from the surrogate AT problem defined in Eq. (9) with continuous gradient flow. When the a in Assumption 1 satisfies a < a-I[T(Mmm)A (e (MMa)) after training for infinite long
time, the model parameter O will converge 0, (M = (M W.QQ (Murin)), satisfying:
KQ W. = Odxd, and KQ KQ W.Y W.21 0dx1. W2,22
w w2.12 2221
w WX WK (M(aan)) + e (M
Remark 1. When the no-m adversarial perturbation radius is zero, the closed-form AT solution 0, derived in Theorem degenerates to that obtained without AT (see Theorem 4.1 in [69]). Thus, a sufficient large adversarial perturbation is key to helping the LSA model fLSA,8 obtain significant adversarial robustness. This will be further justified in the next section.
4.4 Robust generalization upper-bound
With the closed-form AT solution 0, (Miiin) in Theorem 1, we now analyze the robustness of the trained ISA model. All proofs in this section are presented in Appendix A.4. We study how L.SA model ad adversarially trained under a fixed adversarial suffix length Musin can defend against the ICL adversarial attack with a different adversarial suffix length Mest. That is, we aim to analyze the magnitude of the robust generalization error Rah (0, (Muiin), Mest) for the converged robust model parameter f₂_ (Musin). Here, we prove an upper- for it the following theorem.
Theorem 2 (Surrogate AT Robust Generalization Bound). Suppose all conditions in Theorem I hold and 0_ (Musin) is the surrogate AT' solution Theorem 1. We have
Rudl (0.(MM)))) Musa)= S 2TT [A3 (TreaA + e Vreata) (Tuninn + -.orrain Ta)
+ A).
22
where Murin is the adversarial suffix length in the ICL adversarial attack, and I Tirain == I I'Mmain).
I Vest (MMaa), 4lrain v (MMain)) ), and thuss (Mus)) are functions in Eq. (10).
We further adopt Assumption 2 to help better understand our robust generalization bound.
Assumption For adversarial suffix lengths during AT and testing, we assume that Munin: Muss S O(N), where N is the original ICL prompt length. Besides, for the 12-norm adversarial perturbation radius, we assume that @(Vd), where d is the ICL sample dimension.
In the above Assumption 2, the assumption made on adversarial suffix lengths means that they should not be too long to make the model "forget" the original ICL prompt. Besides, the assumption made on the perturbation radius ensures that it is large enough to simulate the large (but limited) token vocabulary space of real-world LLMS to help model gain robustness.
Corollary 1. Suppose Assumption 2 and all conditions in Theorem 2 hold. Suppose ||A < O(1).
Then, we have the following robust generalization bound,
d2 N
M
Ruby (Mrra)))) Miea) S ((d)
+0
N22
M train
7

---

REAST
Zhuh, NECDAN
AaUi Succesa hus (90
Unmad-dD di--dddddd
Queer 2-t.
ASACC Succeen from
5
15
23
08
15 2.5
25 VMan MMear
0.5
1.0
28
15 20 25"
.Han / Moah
<Reum/
/Pa
-Phur:
</eam Moan
Figure I: Scatter plots of ASR to the ratio v MMes/Maain For each pair of base model and attack, 48 points are plotted. A high ASR indicates weak jailbreak robustness.
Corollary is our main theoretical result, which clearly show that for an adversarially trained LSA model, its robust generalization bound depends on the term O AMaa//Mrinn where Musin and Mist are the number of adversarially perturbed in in-context samples during training and testing. In other words, for ICL adversarial attack with an adversarial suffix length (MM). maintain the order of the robust generalization bound, it is enough to perform surrogate AT with only an adversarial suffix length e / M). Such an observation is useful in practice, since one can thus leverage "short-length AT, which requires less GPU memory and training time, to defend against "long-length" jailbreakings.
Empirical evidence
In this section, we follow Eq. (3) to perform AT on LLMs and investigate the relationship between adversarial suffix lengths during L.L AT and jailbreak attacks.
5.1 Experimental setup
daasesss We adopt five pre-trained LLMs, which are: Vicuna-7B-v1.5 [70]. Mistral-7B- Instruct-v0.3 [21], Llama-2-7B-Chal [52], Llama-3-8B-Insrruct [15], and Qwen2.5--BB-Inttruu [65]. For data in AT, we use the training set from Harmbench [36] as the safety dataset and Alpaca [50] as the utility dataset. For data the robustness evaluation, we construct a test set of size 100 that
consists of the first 50 samples from the test set of Harmbench [36] and the first 50 samples from AdvBench [73]. For data in the utility analysis, we use the benchmark data from AlpacaEval [10].
Adversarial training. We leverage GCG [73], a token-level jailbreak attack, to synthesize (suffix) jailbreak prompts, in which the adversarial suffix length Musin is fixed to one of {5, 10,20, 20, 30, 40, 50) during AT. To reduce computational complexity of tuning LLMs, LoRA [18] is applied to all query and key projection matrices in attentions. In every AT experiment, we follow Eq. (3) perform AT with Adam. Please refer to Appendix B.2 for omitted settings.
Jailbreak attacks. We use both suffix and non-suffix jailbreak attacks evaluate the adversarial robustness of trained LLMs. Specifically, five token-level suffix jailbreak attacks are adopted. which GCG [73], BEAST [44), AmpleGCG [26]. Zhu's 'ss AutoDAN [71], and GCQ [17]. The adversarial suffix token length Mea is varied within {5, 10, 20, 40, 60, 80, 100, 120). Meanwhile, two non- suffix jailbreak attacks are leveraged, which are PAIR [7] and DeepInception [25]. Please refer to Appendix B.3 for full implementation details of all used jailbreak attacks.
Evaluations. We focus on evaluating the jailbreak robustness and the utility of trained LLMs. For the robustness evaluation, we report the Attack Success Rate (ASR) of jailbreak attacks. An LLM-bbssed judger from [36] is used to determine whether a jailbreak attack succeeds or not. For the utility evaluation, we use the AlpacaEval2 [10] to report the Length-contruld of targeted models against reference model Davinci003 evaluated under the Llama-3-70B model. two models are equal, while an LC- WinRate of 100% means that the targeted model is consistently better than the reference Davinci003. Please refer to Appendix B.3 for the detailed settings of model evaluations.
WinRate (LC-WinRate)
An LC- WinRate of 50% means that the output qualities of the
5.2 Results analysis
Correlation between the suffix jailbreak robustness and the ratio / Misa/ Mirain We plot the ASR of models trained and attacked with different adversarial suffix lengths in Figure This results in 48 points for each pair of base model and jailbreak attack. The Pearson correlation coefficient (PCC)
8

---

Table 1: PCCs and p-values calculated between ASR and ratio MMsa Muain A high PCC (within [-1, 1)) means strong correlation between ASR and the ratio. p 5.00 10-2 means that the observation is considered statistically significant.
GCG Abark
DEAST Attack
AmpleOCG Attack
Zhu's AmoDAN
GCQ Albok
Model
POD(T)
p-)ale(()
PCCYf)
p-value())
POCKT)
p-vatuel))
PCC(T)
p-valne())
PCC(T)
p-value(
Vicuna-BB
0.93
4.7 x 10-21
0.63
1.4 x 10-6
0.19
1.9
10
0.51
2.5 10"
0.82
1.4% x 10
Mismnl-BB
0.86
4.0 10 15
0.29
4.4 10
0.74
1.5 10
0.49
x 10
0.70
2.6 x
L-7m
0.88
9.0
10-17
10
0.67
1.7 10-7
0.37
1.0
10
0.13
3.8
10
0.71
2.1 x 10-8
1oma-3-88
0.76
2.8 10-10
0.26
7.7 10-2
-0.07
6.2
10
-0.12
4.1 10
0.0
9.7 10-1
Qwen2.5-70
10
0.58
10
-0.24
1.0
10
0.72
1.1 10
Mound
Maurt0
Maun22
Maunio
Maant0
Muunto
Muu-100
1uu7120
CCc
BE4ST
Arpredoo
2245 AUD0OAN
oog
Atrck rtan door
Longhh
Lenge
Lenget
Longh
Figure 2: ASR versus Mirain Vicuna-7B-vl.5 under jailbreaking with different Mist- Munin means that AT is not performed on the evaluated model. A low ASR indicates a strong robustness.
and the corresponding p-value between the ratio MMea and the ASR are calculated in Table 1, where bold p-values indicate that observations are statistically significant (i.e., p < 0.05), while underlined ones indicate they are not significant.
When the jailbreak attack used during AT is the same as that used during robustness evaluation (i.e., GCG), one can observe from Figure that a clear positive correlation between the ratio /Muss/ /Mess/Mainn and the ASR for all evaluated base models. Further, high PCCs (> 0.7) and low p-values (< 0.05) in Table 1 also confirm that the observed positive correatatn is statistically significant.
Besides, when the jailbreak attack is BEAST and GCQ, which is different from that used during AT, the significant positive correlation between the ratio /Mest/ MMraa/Maain MMrain and the ASR can only be observed from some of the base models. This may be due the fact that AT with only single jailbreak attack may not help the model generalize well to unseen attacks. Therefore, it might be necessary to adopt multiple attacks when performing AT-based alignment on LLMs. Nevertheless, from Figure 1, find that for those models where the correlation is not significant (i.e., Mistral-7B. and Llama-3-8B), GCG-based AT can still suppress the ASR no more than 50%, which indicates that can still help models gain a certain degree of robustness against unseen attacks.
Finally, for AmpleGCG and Zhu's AutoDAN attacks, we notice that the correlation between the ratio v /Mes//Maain Mrrain and the ASR cannot be observed on most of the base models. However, this is simply due to AT being too effective in defending against these two attacks: from Figure 1, one observe that AT effectively reduces ASRs of AmpleGCG and Zhu's 's AutoDAN to nearly zero in most cases.
Relationship between adversarial suffix lengths in AT (i.e., Mirain) and suffix jail- breaking (i.e., Mist). We plot curves of the ASR on Vicuna-7B versus the adversarial suf- fix token length during AT in Figure 2. Results on remaining base models are presented in Fig- ure 4 in Appendix B.4. From these figures, we find that as the adversarial suffix token length during AT increases, AT can effectively reduce the ASR of all analyzed attacks. Furthermore, when the AT adversarial suffix token length is set to 20, AT is already able to reduce the ASR by at least 30% under all settings. All these results demonstrate the effectiveness of defending against long-length jailbreaking with short-length AT.
Table 2: Time cost (hrs) of LLM AT with different adversarial suffix lengths.
Adversarial Suftix Token Length Merin
Model
5
10
20
30
40
50
Vicuno-28 78
10.2h
11.3h
13.8h
16.th
18.26
20.40
Mistrul-7l
8.5h
12.0h
14.3h
1amms-2-78
9.00
11.0h
13.2h
15.5h
18.th
20.06
Llama-s-88
9.7h
10.8h
13.1h
15.3h
17.7h
20.2h
Qwen2.5.7B
9.3h
9.5h
11.7h
13.9h
16.4b
18.4h
Time cost of LIM AT with different adversarial suffix lengths Mirin- We then present the time costs of performing LLM AT in Table 2. From the table, we find that when the adversarial suffix

---

Table ASR(%) %) of suffix jailbreak attacks versus models adversarially trained with different adversarial suffix length M A low ASR indicates a strong robustness.
Advensoiial Suriis
Token I Lngth Minia AT
Model
O (No AT)
5
10
20
30
40
50
Vicuna-78 7B
84.0
53.0
48.0
42.0
50.0
44.0
55.0
PAIR
Qwen2.5-7B
71,0
20.0
17.0
25.0
19.0
24.0
26.0
Despinception
Vieuna 7B
Qwan2.5-78
76.0
39.0
150
0.0
0.0
0.0
0.0
length Mysin during AT is as long as 50, the time cost of AT can reach around 20 hours, which is around 30% to 60% longer than that when Muain is set to 20 or 30. Meanwhile, according to Figure in this section and Figure 4 in Appendix B.4, AT with short adversarial suffix length of 20 or 30 can already enable trained LLMs to achieve strong jailbreak robustness. These results demonstrate the advantages of using short-length AT instead of long-length AT.
Robustness of jailbreak attacks beyond suflix attacks. We also calculate the ASR of two non-suffix jailbreak attacks, PAIR and DeepInception attacks, against LLM AT in Table 3. From the table, one can observe that: (1) For the DeepInception attack, LLM AT with a short adversarial suffix length (Muain 20) can already suppress ASR to 0%. (2) For the PAIR attack, while LLM AT with a short adversarial suffix length can reduce ASR from 84% to
Vicuna-BB
Llaaa2278
Qwen225-15 5-7B
50
800
LC-Wintare (1
around 50% against the Vicuna-7B model and from 71% to around 25% against the Qwen2.5-78 model, further increasing the suffix length does not help much to improve LLM robustness against PAIR. These results suggest that the mechanisms behind suffix-based and non-suffix-based jailbreak attacks might have different properties.
10 20 30 Suftx Length Mea-
30
Figure 3: Utility analysis based on LC-WinRatt against the referenced Davinci003 model. high LC-WinRate indicates strong model utility.
Utility analysis. Finally, we plot the LC-WinRate of models trained under different adversarial suffix token lengths and the original model (i.e., Mirin = 0) in Figure We find that while AT reduces the utility of models, they can still achieve WinRates close to or more than 50% against the reference model Davinci003. This means that these adversarially trained models achieve utility comparable to Davinci003.
Conclusion
We study the AT problem in LLMs and unveils that to defend against a suffix jailbreak attack with suffix length of ((M), it is sufficient to perform AT on jailbreak prompts with suffix length of (VM)). The finding is supported by both theoretical and empirical evidence. Theoretically, define a new AT problem the ICL theory and prove a robust generalization bound for adversarially trained linear transformers. This bound has a positive correlation with ((VMes) (aria/) Empirically, we conduct AT on world LLMss and confirm a clear positive correlation between the jailbreak ASR and the ratio v /Mest Minin Our results indicate that it is possible to conduct efficient short-length AT against strong long-length jailbreaking.
Acknowledgements
Di Wang and Shaopeng Fu are supported in part by the funding BAS/1/1689-01-01 and funding from KAUST- -Center of Excellence for Generative AI, under award number 5940.
References
[1] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Conference Neural
10

---

Information Processing Systems, 36:45614-456333 2023.
[2] Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Linear attention is (maybe) all you need (to understand transformer optimization). In International Conference Learning Representations, 2024.
[3] Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong. Jesse Mu, Daniel J Ford, et al. Many-shot jailbreaking. In Conference Neural Information Prucessing Systems, 2024.
[4] Usman Anwar, Johannes Von Oswald, Louis Kirsch, David Krueger, and Spencer Frei. Adver- sarial robustness of in-context learning in transformers for linear regression. arXiv preprint arXiv:2411.05189 2024.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In Conference Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[6] Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030, 2024.
[7] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
[8] Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, and Chuan Guo. Aligning LLMs to be robust against prompt injection. arXiv preprint 240.0214,,. 2024.
[9] Xingwu Chen, Lei Zhao, and Difan Zou. How transformers utilize multi-head attention in in-context learning? A case study on sparse linear regression. arXiv preprint arXiv:2408.0455 04532, 2024.
[10] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled AlpacaEval:: A simple way to debias automatic evaluators. arXiv preprint D.XXi:2404.04444 2024.
[11] Spencer Frei and Gal Vardi. Trained transformer classifiers generalize and exhibit benign overfitting in-context. arXiv preprint arXiv:2410.01774, 2024.
[12] Shaopeng and Di Wang. Theoretical analysis of robust overfitting for wide DNNs: An NTK approach. In International Conference on Learning Representations, 2024.
[13] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Conference on Neural Information Processing Systems, 2022.
[14] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint 1rXiv:1412.6572, 2015.
[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
[16] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. In Conference Empirical Methods in Natural Language Processing, 2021.
[17] Jonathan Hayase, Ema Borevković, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query- based adversarial prompt generation. Conference on Neural Information Processing Systems, 2024.

---

[18] Edward Hu. yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[19] Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint 1rX00:2310.05229, 2023.
[20] Adel Javanmard, Mahdi Soltanolkotabi, and Hamed Hassani. Precise tradeoffs in adversarial training for linear regression. arXiv preprint wrXi:2002..0047, 2020.
[21] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
[22] Haibo Jin, Andy Zhou, Joe D. Menke, and Haohan Wang- Jailbreaking large language models against moderation guardrails via cipher characters. In Conference on Neural Information Processing Systems, 2024.
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[24] Tong Li, Shu Yang. Junchao Wu, Jiyao Wei, Lijie Hu, Mengdi Li, Derek F Wong, Joshua R Oltmanns, and Di Wang. Can large language models identify implicit suicidal ideation? an empirical evaluation. arXiv preprint arXiv:2502. 17899, 2025.
[25] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191, 2023.
[26] Zeyi Liao and Huan Sun. AmpleGCG: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs. In Conference Language Modeling, 2024.
[27] Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. In International Conference on Learning Representations, 2024.
[28] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenggi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint 4rXiv:2405.04434, 2024.
[29] Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao. Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak L.M.s. arXiv preprint arXiv:2410.05220 2024.
[30] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations, 2024.
[31] Yue M Lu, Mary I Letey, Jacob A Zavatone- Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint 175. 2024.
[32] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[33] Roey Magen, Shuning Shang, Zhiwei Xu, Spencer Frei, Wei Hu, and Gal Vardi. Benign overfitting in single-head attention. arXiv preprint arXi::4410.07746, 2024.
[34] Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In International Conference Learning Representations, 2024.
12

---

[35] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan, PEFT: of-ee----trhtrrrre parameter-effiiiet fine-tuning methods. https: 2022.
/ //g thub. cww www.cc/gggigaa////e///ooo/oocee/
[36] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendryeks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.004249, 2024.
[37] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Conference on Neural Information Processing Systems, 0127770-277744 2022.
[38] Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Adyprompter: Fast adaptive adversarial prompting for LLMs. arXiv preprint arXiv:2404.1687, 2024.
[39] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008.
[40] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma. Subhrajit Roy, Ahmad Beirami, Prateck Mittal, and Peter Henderson. Safety alignment should be made more than just few tokens deep. arXiv preprint arXiv:2406.05944 2024.
[41] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuming aligned language models compromises safety, even when users do not intend to! In International Conference on Learning Representations, 2024.
[42] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Conference on Neural Information Processing Systems, 36, 2023.
[43] Antonio H. Ribeiro, Dave Zachariah, Francis Bach, and Thomas B. Schön. Regularization properties of adversarially-traine linear regression. In Conference on Neural Information Processing Systems, 2023.
[44] Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one GPU minute. In International Conference on Machine Learning, 2024.
[45] Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Günnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source LLMs through the embedding space. In Conference on Neural Information Processing Systems, 2024.
[46] Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Madfield-Menele and Stephen Casper. Latent adversarial training improves robustness to persistent harmful behaviors in LLMs. arXiv preprint 2xXi::2407.15549 2024.
[47] Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? In International Conference on Machine Learning. 2024.
[48] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
[49] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13,,,6 2014.
[50] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An mnslooition-oolooinng L.LaMA model. https: / /gi thub. coom/tatsu-a htab/stanfors ord, alpaca, 2023.
13

---

[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume I Lample. Llama: Open and efficient foundation 2023.
language models. arXiv preprint arXiv:2302.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (xXiv:23070092880 2023.
[53] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 144444..
PMLR, 2023.
[54] Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao. Adversarial demonstration attacks on large language models. arXiv preprint arXiv:2305. 14950, 2023.
[55] Liangyu Wang, Jie Ren, Hang Xu, Junxiao Wang, Huanyi Xie, David Keyes, and Di Wang. 7o2: Scalable zeroth-order fine-tuning for extremely large language models with limited gpu memory. arXiv preprint arXiv:2503. 12668, 2025.
[56] Liangyu Wang, Junxiao Wang, Jie Ren, Zihang Xiang, David E Keyes, and Di Wang. Flashdp: Private training large language models with efficient dp-sgd. arXiv preprint arXiv:550..0115, 2025.
[57] Liangyu Wang, Huanyi Xie, and Di Wang. Distzo2: High-throughput and memory- efficient zeroth-order fine-tuning Ilms with distributed parallel computing. arXiv preprint arXiv:2507.03211, 2025.
[58] Yunjuan Wang, Kaibo Zhang, and Raman Arora. Benign overfitting in adversarial training of neural networks. In International Conference on Machine Learning, 2024.
[59] Zhijie Wang, Bo Jiang, and Shuai Li. In-context learning on function classes unveiled for transformers. In International Conference on Machine Learning, 2024.
[60] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Conference on Neural Information Processing Systems, 2023.
[61] Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv: 230.206308687 2023.
[62] Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In International Conference Learning Representations, 2024.
[63] Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in LLMs with continuous attacks. In Conference on Neural Information Processing Systems, 2024.
[64] Zhao Xu, Fan Liu, and Hao Liu. Bag of tricks: Benchmarking of jailbreak attacks on LLMs. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[65] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.1511 15115, 2024.
[66] Tong Yang. Yu Huang. Yingbin Liang, and Yuejic Chi. In-context learning with representations: Contextual generalization of trained transformers. In Conference on Neural Information Processing Systems, 2024.
[67] Junchi Yao, Jianhua Xu, Tianyu Xin, Ziyi Wang, Shenzhe Zhu, Shu Yang, and Di Wang. Is your llsed multi-agent a reliable real-world planner? exploring fraud detection in travel planning, arXiv preprint arXiv:2505.16557 2025.
14

---

[68] Lei Yu, Virginie Do. Karen Hambardzumyan, and Nicola Cancedda. Robust LLM safeguarding via refusal feature adversarial training, arXiv preprint arXiv:2409.20089, 2024.
[69] Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55, 2024.
[70] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Sto- ica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[71] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. AutoDAN: Interpretable gradient-based adversarial attacks large language models. In First Conference on Language Modeling. 2024.
[72] Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko,
J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In Advances in Neural Information Processing Systems, 2024.
[73] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307. 15043, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claim made by the abstract and introduction is that: short-length AT effectively help LLMs defend against long-length jailbreak attacks, which is supported by both theoretical and empirical evidence. The theoretical evidence is justified in Section 4, while the empirical evidence is justified in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5.2 discusses the limitation of using only single jailbreak attack during AT to defend against unseen attacks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are stated as Assumption 1 and Assumption 2. All proofs are presented Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary details to reproduce experimental results in this paper are provided in Section 5.1 and Appendix B. The experimental code is also provided in the supplementury material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
17

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Experimental code and detailed instructions provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n ctitps..cc/pbbi//gugu guwdd//u//////eeeeee /CodeSubmiee ss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All necessary details to reproduce experimental results in this paper provided Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neurri: cc cctppppiccccpccttcticcccccccccce ita.iddeeeedeiieneeeneene
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See the README. md file and the LICENSE file in the submitted experimental code for details.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See the README. md file in the submitted experimental code for details.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tprr//s.u...eeeee / for what should or should be described.
22

---

A Proofs
This section collects all the proofs in this paper.
A.1 Technical lemmas
This section presents several technical lemmas that will be used in our proofs. Lemma A.1 (c.f. Lemma D.2 in [69]). If x E RRdx1 is Gaussian random vector of d dimension, mean zero and covariance matrix A, and A E Rdxd is fixed matrix. Then
E E/xr' Axx 11 M(A MAA+ATJA -
Lemma A.2. If x E [dxx is Gaussian random vector of d dimension, mean zero and covariance matrix A, and A Rdxd is fixed matrix. Then
E[x Ax] (r(AA).
Proof. Since
d Arg = Σ (AA )1+- = TY(A),
E|x T Ar)
A
Σ i
Ar, E[z,ey]
E 1
which completes the proof.
Lemma A.3. For any matrices A E Raxm and B E Tg we have
Tr(AB) = Tr(BA).
Proof. Since
Tr(BB) = Σ (AB)..
Ar =
B BB = (BA), TT(A),,
1=1
j= F=
which completes the proof.
Lemma A.4 (From Lemma D.1 in [69]; Also in (39]). Let X E Raxm be variable matrix and E Roxn and B Raxm be two fixed matrices. Then, we have
OxTx(BXT) = B mm
dx (+(XXX"))) = (AXB+ A AB Raxm.
Lemma A.5 (Von Neumann's Trace Inequality; Also in Lemma D.3 in [69]). Let A E Raxm and B mm be two matrices. Suppose (a1 (A), mi (A)) and (o, (B), ... min (B)) all the singular values of A and B, respectively. We have
m.
min
Tr(AB) <
(B)) <
Σ
||
||B||2 = min(n (nn m) ||Al/2 [B]|2-
A.2 Proof of Proposition
This section presents the proof of Proposition 1.
Proof of Proposition 1. For the AT loss L(0) defined in Eq. (8), we have that
[ cady (0) = adv (0, Murin) = E max
l ( (Eady
1 Vr.gl
IIA 2.00
1 ((wx)
Eady T.uly. M.Mii
WKQ
12)
E
All (w T)
-rr.
Vrrgg
(A.1)
| 12 Se 2
N Mirain
23

---

Then, the term E
E.MM. Mmn Mun M.Maan can be decomposed as follows,
E
E T "r,Muin
((x-)
X Y
X + ys
TT.1
X, Y
X 4, ysk
IT.q
0
0
X
Xae
IT.99
X
X
IFTa
OIXN =OIxx
Odxx
D
0dx11
Oaxx OIxN
0ax1 0
Y
0
Y
Y
01x Muss
MixMan
O
X
X
Ira
OaxN 0IxN
A
Oax1
(Oax 0IXN
0ax
X
X yaix
Ir.
+
+
Y
O
Main
Mxx
Y
X
Xafa
Fr.
X
X
Traa
Ar
A.
T
Y
Y
ysix
O x Mexn
01x Musin
X
T
X
yslk
0 x Minic
+
01
Mon
y
which further means that
Eady
Muni TT.Man
wKQ (wx21)7
(ww)) T
wx22
IT.9
N + Muain
X
X
IT.q
Xx
X
ZT.8
T
Y
ysfx r
Y,
y
WKQ
=(w!))
V. Wazz
(w Re )
N
Mirain
X ystx
A.
A,A7
+ (w
-w IT. + ((w21)
w
W. Ir
N + Musin
N + Musin
x stx
T
W.QQ (w (
+ V
Z7.g.
(A.2)
N
Mrain
Inserting Eq. (A.2) into Eq. (A.1) and applying the inequality that la + b12 2 (a2 + 12), Ladr can thus be bounded as
X,
X
X,
x ystx
Fr.
Y
yilk
Y
W k
Cadv (0) s 2- ((x
wx22
R (w (T
IT
yr.0
N + Mirain
A
2
(WY))
wrQ xT.
2
max
||L 21,.00 Sr
+ Mirin
=0))
s sfx
A
((xh))
W KQ
2
+ 2 R 7 ||L, 5c
max
W222)
Ir.9
N
M
==>=(0)
Xxfx yatx
(Wx)).
W.Q (w (
2
2 E max
IT.9
(A.3)
||A Se
N + Musin
Ag())
24

---

We then bound terms (1))), A2(0), and As(0) in Eq. (A.3) seprately. For the term (1) in Eq. (A.3).
we have
A1) == (N Muain) )2
E max
(wx)))
Mun
W.k
FT.q
2
I|A, Se
1=1
Admi
Mesi
$ (N Musin) )2
2 max
Σ ((w2)) To 0r,]²
18 Wi
k
IT
||! 11,.00 Se
by Cauchy- Schwarz Inequality
Munic
Minic
S (N + Masin) )2
max
[(w2]) T 8r,11
Σ
max lor.illese
16] W vn. 12
Musi
Musi
I la c]p
I KQ
xrgll2 c??
(N + Murin) )2
2c M
| 12 BIlW
kQ
Fr. 12
(A.4)
(N Murin) )2
11
For the term A2(0) in Eq. (A.3), we have
A2( = (N Musin )2
2
E
max
((w))
wx22
Musi
J.fEx V7.5
8T.
W
KQ
IT.9
72
T
IA 112, Se
Masin ((w)))
Manin
W
E
max
U222
(N M M
|| 10 Se
by Cauchy- Cauchy-Schwaazz Inequality
Misi
au T. y
Music
E ((w))
E
max
11 xraa) 12
(N Minnn )
118.,
Musi
Musis
2
(ww)) T
wK22
WlllW KQ
xT 6|?
(N M 12
2c Miain
Minic
]
E
KQ
Rrall 1
((wX))
(A.5)
(N Muan)?
For the term As(0) in Eq. (A.3), we have
Moan
K.
12
A0( = (N + )
E max
(wh))
Sr
(
I7.9
IIA
Se
i=1
Mini
Mrm
W.kQQ (w
E max
[ww T S0,12
ITT1
(N Murin )2
IIA 112.00
i=1
by Cauchy Schwarz Inequality
2
Music
max [(w21] (Tok 8r
Misic
1 V
W:KQ
27
(N Murin }
(w (
Munic
Music
fo T.s yots
Dr1g 12
IIwbklle ej2
W1
(N + M y2
(w )T)
2c Mirain
Mrin
ala yalk
T
W.KQ
I ll
Σ
(w RQ,
Ir.g
(A.6)
(N Murina
mm
25

---

As result, by inserting Eqs. (A.4), (A.5), and (A.6) into Eq. (A.3), we finally have that
X
X
X,
X
Iz.a
0
V
yst
0
Y
Ystx
wKQ
Cudy (0) 2- 2- (
wx2)
IT
N
Muain
2e Menin
+ (N + Manin)
1 V. 12
E (WW
Fr.q
26 Murain
Musi
E WW KQQQ IT.
I3 ((w)))
wxa)
(N + MMrinn²
2e2 Mirai
Musin
w.Q
2
+ (N + Musin)
11 12:
IE
(wez
( )
IT
(A.7)
The right-hand of Eq. (A.7) is exactly the surrogate AT loss (udv(() in Eq. (9), which thus completes the proof.
A.3 Proof of Theorem 1
This section presents the proof of Theorem 1, which is inspired by that in [69]. Specifically:
KQ 1. we first prove that terms w and 2221 stay zero during the surrogate AT (Lemma A.6) via continuous gradient-flow, which thus can simplify the surrogate AT loss Lade (0) defined in Eq. (9) (Lemma A.7).
2. We then calculate a closed-form solution 0, for the surrogate AT problem based on the simplified _ (f) (Lemma A.8), which is exactly the solution given in Theorem 1.
3. Finally, prove that under the continuous gradient flow, the LSA model starts from the initial point defined in Assumption can indeed converge to the closed-form solution 8, (Lemma A.12), which thus completes the proof of Theorem 1.
We now start to prove the following Lemma A.6.
Lemma A.6. Suppose Assumption I holds and the LSA model fLSA,0 is trained via minimizing surrogate AT loss Ladv (0) in Eq. (9) with continuous gradient flow. Then, for any continuous training time 2 0, we uniformly have that wh (t) = K.Q (t) = 0dx1-
Proof. When the LSA model fiSA.o trained with continuous gradient-flow, the updates of w/ and KQ with respect to the continuous training time 2 0 are given by
221
O,w (t) = - uy Eudy
(a))( (0),
0 2211 (t) = 0
KQ KQ(()
kg andy (8).
21
Meanwhile, since Assumption assumes that w (0) = w (0) = 0ax1s therefore, to complete the
proof, we only need to show that d,w (t) = 0 w. 21 (t) = 01xd as long as w (t) = w KQ (t)
KQ
0ax1 for any t > 0. In other words, below need to show that w w 0ax1 indicates Owy Zadv (0) 0 x fadv (0) 0 xxd-
Toward this end, adopt the notation in Eq. (9) to decompose the surrogate AT loss Σ(0) follows,
judy [udy( (t) = (0) + (0) (g) ( + 2())),
26

---

where
X,
Xstx
ZT.g 0
X,
Xsxx
IT
(0)) 2F ((wh)) 7
Y
y
Y
yof
W
2222)
(u=2) (T)
2r.9
yr
N
Murai
(A.8)
2c M urain
(2( (N + Manin) )a
/ 12 l|
Q
x 2
(A.9)
2ca Marii
Xxa y
(0) (N + Munin) )2
W KQ
Trgll 11 II((wN))T T wx2)
12
(A.10)
2c Musin
Xsfx
T
wkQ
(0) (N + Manin)
11
E 12
Tr y
klle
Fr,
(A.11)
In the remaining of this proof, we will show that when whi (1) Owy (0) = Owke (1)) = 01xd, (2) Owy, (0) = @wx = 01xd. (3) Owy (a(0) =
w/21
KQ
Oax1 holds, one has:
@wxely (0) = 01xa: and (4) @wy (0) = @w (0) 01 xd which thus automatically indicates that Owy Ladv (0) ww Ladv (0) 01xd-
Step 1: Show that w 21 KQ 0ax1 indicates owy (0) @w 01xd- Such claim
can be directly obtained from the proofs [69]. Specifically, when setting the (original) ICL prompt length from N to (N + Mman). the ICL training loss L in [69] is equivalent to our & (0) defined in Eq. (A.8). Therefore, one can then follow the same procedures as those in the proof of Lemma 5.2 in [69] to show that the continuous gradient flows of WVl and WKQ Please refer accordingly for details.
are zero when Assumption holds.
Step 2: Show that w
KQ = 0axi indicates owy, (0) = 2 (0) = 01xd- Since the term W21
"2 KQ does not exist in the expression of (2()) in Eq. (A.9), we directly have that 0 K kg 82 (0) 01xd- W22 Besides, for the derivative owy, (0),,, based on Eq. (A.9) we further have that
2e MMMMa Maann
Ouy, (2(
= -wyy
| 112
||
KQ 11
2r.g
1o -0x11
(N + Mrain) )2
w =Da1
4e Maain
E
W KQ
Tr. 122 (wh))
(N + MMrain) )
-0x0
4eaMa Mrann
KQ x 112 0.x1
01xd=
(N + MMran)?
which justifies our claim in Step 2.
Step 3: Show that w = 22
KQ
0ax1 indicates 0ny, (z( = Ô 2/
(g) = -0xxa- We first
rewrite (f) that defined in Eq. 10) as follows,
(s( (N + Murai)) 1
26 Musin
E |i
xT 1l II((w)))
wxr
X yalx
[2
2ca Muain
Minin
To T.d
LE || KQ
xr Σ E ((wni)
why
((wh))
w//
(N + Murin )²
y
yo
2ca Mixin
Maan
I1) ((wh))
TF. |I
KQ
((wx))
wha
wx2)
xr
(N + Mirain
(A.12)
27

---

Then, for any i E [M]
we have
six
I stx
(2 (2e T
(wi,) 2etss) T (wo x..,y)) TT)
E
E We a
I an w
MT.
w +
A
A- Oax1 W. AwOT
1)
A 01xd
Oax1 Tr(Taa)
A
Oax1 Tr(A))
(A.13)
O1xd A
Ew,
Olxd
by Lemma A.2/
Finally, by inserting Eq. (A.13) into Eq. (A.12), Ez(
can thus be simplified as follows,
E (0 (N + Manin)
2c Musin
Ke
Tr [12| ((w)))
Ir.
wxy
Minis
Σ
A
Oaxi
((wh))
w//2) T
Oixd
2c Mfriin
KQ IE II
m1 Bo
((w)) Awyy + TT(A)(u)2)P)
(A.14)
(N + Muain) )2
According to Eq. (A.14), (f) does not depend u121 KQ which means that 0 (x) 01xd- On "ht
the other hand, based on Eq. (A.14), when w O, the derivative of (z( with respect to w is calculated as follows,
2c 2 Masin
0uy E3(
-0 -wxy (N + M )2
E
KQ
T.
[
((w))) Aux + (V)))
w//=
2c R.MMaain
-ux (wx2))
Awwy =x==0
(N + MMrrin) )2
4e Ma
[(w2))) TA]
(N + -Mrran) )2
=0
4e2 M wain
KQ [|
0 A
01xd:
(N + MM
which justifies our claim in Step 3.
Step 4: Show that w
u21
KQQ 0ax1 indicates Ouy, (0)
0 (ggel((0)
01xd. When
w w22
KQ Oax1, based on the expression of (0) given in Eq. (A.11), the derivative of (0)
with respect to whi is calculated as follows,
2c Mrain
X yaix
w.k (wwt (wuttet))
Ouy, (4(
Omeyi (N + Misin)
|1021 112
E
Ir
in wil
KR 00x1
=Y=w1
Kg =0ax1
4e2 Muran
X
= -[(N + Mirii)) j
E ||
yalx
Ir. 11 ((12))
w KQ -0x1
4c2 Mirin
Xatx EEl
y ystx
11
21.9122- 0xx1 = 01xd-
(N + M
02x1
Besides, for the derivative of (0) with respect to KQ we also have that
2c Musin
Xatx ssix
T
W.Q
((00)
I 1012
w 12
E
||
11
x 1
212
w/i = RD
KQ=0dx1
(N + Murii) )
(w
w =
xQ
=04x1
2ca Manin
Xala Ell
W-kQ
(N + MMrin)2
1l V
2
xa
p ysix
(1021 RQ
)
w 2
K2 =0xx1
26 Munin
x
W-KQ (
(N + Misin 2
||0ax1 113
θ E
xo
y
Fr.
=0xd
KQ =0ax
10
28

---

The above two equations justify the claim in Step 4. Step 5: Based results from previous Steps to 4, eventually have that
Duy L (0)
Duy (2) (3())
01xd 01xd:
w w1
Kg =0xx1
w = =0axs
Kg 21
Eadv
=0xx1
= =0 + (4)
K2 =0x11
0xx
01xd-
i=1
The proof is completed.
With Lemma A.6, we can then simplify the surrogate AT loss Lady (0), as shown the following Lemma A.7. Lemma A.7. Under Assumption I, the surrogate AT loss Ludv (0) defined in Eq. (9) can be simplified as follows,
Eudv (0) - 2Tr (T
(T 2T ) 2b Ma) (W (WX
44T (w:2 W KQ A²) A 2Trr(A),
Eq. (10). N+M N+M and (MM) = ((TT(A)
where I(M)
N-M+1 A
Tr(A]
(N+ are same functions as that defined in
Proof. When Assumption holds, by applying Lemma A.6, one can substitute terms wxi, and W in the surrogate AT loss Eutv with the zero vector Odx1, which thus simplifies [m) as follows,
KQ
X
Ir.
X
X
IT.y
Y
y
Y
ysf
W.K
Eadv (0) 2E 20
w22)
N
Merain
0
Ir.q Vr.
2e Merain
(xsfx y
+ 0 +++
|i Fr.9 12 -1
wik))
[2]
+
(N Murain) )²
Y,X, + ystx X
2 W022
V
WKO FT.q
Vt.g
N + Musin
(B((0)
2c Mirin
E IIw xnall
(A.15)
(N + Muain)
BB(())
For the term (0) in Eq. (A.15), have that
(0) = 2 E W32
Y,X + (X=
W.K
Fr.9 Vra
2
N + Miain
E
: (X,XT + X (X:R)T
W22 w.y.H.
WKK
-Fr. - w Ir.
2
+ Mirain
X + Xx:
X X, + X xxxx(xx (X:R)T
W2z
WKQ
xr.g
FT.9
w.w.F
wa W.K.
2r. Ir.9
N + Musin
+ Murain
X + X (X:st)
X X,XT + Xxxx (X::O)
w wyy WKO I:.g D7. ]
V w
2 E
Fr.
Ir.g
Ia
N + Musin
N + MMain
Er (X,XT + X (X (X,XX + Xxxx (X*())
= -2 TT.g
(w w. K
wyy W. -rrig
(N + Mirain)2
E, (X,X7 + X
44 x 7.
(WY, W.Q) 27,99 + IT. 200]. (A.16)
N + Muraio
29

---

For E, (Xxx7 (x;") (X;
in Eq. (A.16), we have
7
(x*x)
= E X
x 1
+
E
x
E X
X
+
E X
/ (x T x
'
Jr F f 3. 3
+ E
+
E
Ir Tr, T
7,3
E
E
(a (1
Ty t
A2
+ Munin A NA + NA Marin
ISijENiifj
A2
+
E
Isi.SMMam (#)
N
22A Tr(A)A) + +(N2 N) ^2 MNMMaain A²
=
E
by Lemma A.I
Misin
+ E
22 + T(AAAA + (M2 train - Mrain) A
by Lemma A.1 (N2 + N Mtain + Mirain + 2N Mimin) A2 + (N + Mminn) Tr(AA) A
= (N + Murin) ((N + Miain + +1) A2 + Tr(A) A) = (N + Muain) I (MMain) (A.17)
For Er [x,X] (X;M)))) in Eq. (A.16), have
E X + X (X:f)) J]
E
Iy Ty T T.i
+ E Σ x (a
NA + Misin
(N + Mirain)
(A.18)
Inserting Eqs. (7. and (8.18) into Eq. (A.16) leads to [(Manin) Wyy.K 11 .TT9
Bi (0) 2
T.. (WXWKK))
IT A wyy WKO [rn]] + 2. R. TT #To0
2 Tr (wbs (wbb TT r(Mmin)A woys w.k -A
by Temma A.2
- -4. Tr A why w KQ A +2 Tr(A)
11
by Lemma
2 2 Tr ((Mrnn)A ) (w w A2)
by
4. Tr (Wo A +2 Tr(A).
(A.
by Lomma A.3
30

---

Besides, for the term (2( in Eq. (A.15), we have that
2x Miain
B2( (N + Musia)
E |W (W Ww
TTgllé
11
y
T
2e Minin
(N + Miran) )2
(2 E FT
2r.g
E2 w
X x (x:t))
w-
2xa Minin
(w
T A by Lemma A.2
E w Mirain wy
(N + Murin)2
2c Muain
= (N + Muin)
(2 Tr Tr[wwo... A (WKO)
Tr
Misin la
by Lemma A.3
Lernnna A.2
= =2c². M Tr(A)
(N + Mz) Tr (w W KQ At) A whe)
train
=222 (MMaii))) Tr ( k. Ad)
(A.20)
Finally, by inserting Eqs. (A.19) and (A.20) into Eq. (A.15), we have
(0)
= 2- Tr [(Maain )A (w K
WHWKAA (w w KQ A ) ]] - 4 Tr (w (KOAW -A
+ 2 Tr(A) + 26 (Mran))) Tr (w w A A (
2 Tr (T(Maain) A ( La) (w w.o ( (W
4- Tr
A + Tr(A),
which completes the proof.
Based on the simplified surrogate AT loss, the closed-form global minimizer 6, for the surrogate AT problem is then calculated in the following Lemma A.8.
Lemma A.8. Suppose Assumption I holds. Then, 0, == (W. W.O) is minimizer for the surrogate AT loss Ladv (f) in Eq. (8) if and only if wY w = (MMaaan + @xpp(main)) Ta) A.
Proof. For the simplified surrogate AT loss proved in Lemma A.7, we rewrite it follows,
Zadv (a)
2Tr (T(M )A + e² (Musnn) (w WKA (wh ( A
4TT
(10 W.KKKKKKZ 11) A3 + 2Tr(A)
2 Tr (I -ainn + Vrrranaaa) (w w K (I rrann + e² (urraiaaa -1)))110 A
(wyy D.y.WKKAK (Tuain + Ed -2 }
Tr (TTaminn A + 2 Ta) 111 + Tr(A),
(A.21)
where T Irain = T(Mrain) and vhtmni = (MMrin))
Notice that the second and third terms in Eq. (A.21) are constants. Besides, the matrix (T Urain A T in the first term in Eq. (A.21) is positive definite, which means that this first term is non- negative. As result, the surrogate AT loss Ludv (B) will be minimized when the first term in Eq. (A.21)
becomes zero. This can be achieved by setting
W.2.W.KK W.KQA A - (T(M A + (amin Ta) A = 0,
31

---

which is
W. W.kip (T A + (Mrinn))) La) -a) AA.
The proof is completed.
We now turn to prove an PL-inequality for the surrogate AT problem. The proof idea follows that in [69]. Specifically, we will first prove several technical lemmas (i.e.. Lemma A.9, Lemma 10 and Lemma A. I), and then present the PL-inequality in Lemma A.12, which can then enable the surrogate AT model in Eq. (9) approaches its global optimal solution.
Lemma A.9. Suppose Assumption I holds and the model fLSA.0 is trained via minimizing the surrogate AT loss Ladv (0) in Eq. (9) with continuous training flow. Then, for any continuous training time 2 0, we uniformly have that
(w (1)) (H)WRR()(WWT)) (O (
Proof. Since the model is trained via continuous gradient flow, thus θ, W, (I) can be calculated based on the simplified surrogate AT loss proved in Lemma A.7 as follows,
0 W
KQ (t) kro (0)
= --2 0 w. Tr
(T (Maan)))) )A + e² (Main) (w WR
AD)) (w K2 A )
+ 4 8 x Ir
(2b2 WHW A²]
2 (w )2. 0 w
Tr (T(Miin)) ) + 6² 3bb ...k A (WK2)7] + Aw A2
by Lemma A.4
- ( (T(Mmin)) Lh WKQ A +4wYgA?
(A.22)
by Lemma A.4
Similarly, for d:wly((), we have
(,w -0ny Eally (0)
= --2 0,y Tr
(T(Munin )A + 2 (MM(()n)) (w WKKA3 AD). (mb/ KKQ
AB)T]
11
11
+ 44.00 Jwy Tr
(wby W KQ A²) A 11
=-4wk Tr
(T(Musin)) )A + e² Musa))a) (WKRAt) (W)
+4. Tr (WKKN A?]
(A.23)
Combining Eqs (A.22) and (A.23), we thus have
Tr a KQ (OWWKQ()((KF (()) (1))
- -w2 2. Tr (T(Mmin) + (. (W:KAA) Ab). (WAOAA)
+ +ww% Tr A (
= (0.wYx(t)) (t),
which further indicates that
T W (1)
=Trfaww (1) (W (WH (1) + Tr Tr[wwKK -0
= -(Orbb)(t) (x)(() + (t) (0) = (t)3)
(A.24)
32

---

Finally, according to Assumption 1, we have that when the continuous training time is t = 0,
Tr W KQ (W) | o² = (0)
Combine with Eq. (A.24), thus have that
Tr (W( w))(()², V2 0.
The proof is completed.
Suppose Assumption holds and the model fLSA.0 o is trained via minimizing the surrogate AT loss Ladv (0) in Eq. (9) with continuous training flow. Then, if the parameter in Assumption I satisfies
Lemma A.10.
V<1 d. d.II(T ) + eb (P
Vd
M
we have w/2(1) holds for any continuous training time ( 0.
Proof. According to the simplified AT loss calculated in Lemma A.7, we know that w (t) = 0.
then Ladv (0,) = (Tr(A). Besides, under Assumption 1, we have w (0) > 0. Therefore, if we can show that aady (0.) # 2Tr(A) for any / 2 O, then is proved that w)y(() (t) > 0 for any / 2 0.
To this end, we first analyze the surrogate AT loss Lade (0.) at the initial training time = 0. By applying Assumption 1, we have
Zady (0p)
2Tr (T(Mmin) A + + Ia) (1) (
-4TH (O) A + 2Tr(A)
2Tr (I(Mmai)) A + (M (o) (a"00 ( (
-4Tr (0200 A²) A 2Tr(AA)
- Tr (T(Mmin) )A + (M Ia)A 1OO 1OO 46 1101 (Tr(A)
20 d. [I(T((mmnn)) A + e (Mrain) 1a) 11-1110--11000000 0OO -4 2 2TT(A)
||000
by
20 d. (M(m((Mmmm)) )A + 2 (M ) 1a) ||000 A/Ir |100 ||
-101 + (Tr(A)
2 d. [I A + (Maain M || 40 A + 2Tr(A)
20.. || (d o² [I[T (Maaaaa 2 (Marai))) 14)A 112 - 2) + 2Tr(A).
(A.25)
By Assumption 1, we have |A0| > 0. Thus, when (d- 11 (MMrni )A + 2 ( La)A 111-11110 112
2) < 0, which
2 V d. [[ (Mrran)
V
we will have (adv(Op) < Tr(A).
Finally, since the surrogate AT loss īadv (9)) ) minimized with continuous gradient, thus when the above condition holds, for any / > 0, we always have that addv (0.) < Çadv [ad)(On) < Tr(A).
The proof is completed.
33

---

Lemma A.11. Suppose Assumption 1 holds and the a in Assumption satisfies o <
2
Then, for any continuous training time t 2 0, we have
V (EIII((Ma))AA tf Ma)a 2
(t))² マ 0, where
0². |A P (2 d. (T + epp I (22 0.
2d Ad| l
Proof. By applying Eq. (A.25) in Lemma A.10, we have that for any 2 0,
20 100 (d o² - (Maan))) + i Maain) Ma)A -112 - 2) + +Tr(AA)
(ad 2 (od((()))
2Tr (T(Muai)) )A (M (w22 WH
4TT (w W K2 ^ A + 2Tr(A))
(I A 2 (M (M Lo) (wlg A A
4Tr
U22 W wwKkKA A2 2TT((A)
2 4d 2 ||A || 11 +2TH(A),
I Lemma A.5
which indicates
2o². |l (d- 0² ((T(MMMn))))))) + (i Ta)A -2) -4d |wwbaal ||AB|12 ||W KQ ||,,
thus
1//2 1| Ii || 2 0² -2- ||0 (2 d 2. [I )A 22 e Musia )d)AA 1122)
(A.26)
2d A212
Besides, by combining Lemma A.9 and Lemma A.10, we know that
(1) V Tr|Wh KQ (()WK (0) [IHR(OIE |[WH&(()))) (f)l.F.
(A.27)
Finally, inserting Eq. (A.27) into Eq. (A.26), we thus have
o² |
(2 - d. o² [[T(Ma)))A + (Main ( (22) > 0.
2d Ad
The proof is completed.
Lemma A.12 (PL-inequality). Suppose Assumption holds and the LSA model fLSA.0 trained via minimizing the surrogate AT loss Ludv (0) in Eq. (9) with continuous training flow. Suppose the o Then for any continuous truining
Assumption satisfies a V (e
time >> O, uniformly have that
Lady (0.)112 pr. [Cud((()) min Ladr
(a)((0)) (0)),
where
8R
p:= [[ A A + +2 2a)rainna)
- ||A
+1
is defined in Lemma A.11, and Vee(.) denotes the vectorization functioo
Proof.
From Eq. (A.22) in Lemma A.9, we have that 0.wKK( KQ (t) --4 (2xy)2 (T A (Mimmi)) W.KQ A 4wy2A2 =-4wky ( (TMMmnin + 25 Ia) D(O.) Ab.
34

---

where
((O.) 2222 whw KQ A (T(M )A + 2
Rdxd
(A.28)
As result, the gradient norm square |10 Lady goyy (0+)1 can be further lower-bounded as follows, ||0g Euly (0)) = (dwy, Endy (0,)) + L (00))
1 Lodv (0+)l1?
||4 w (T(Mriin)) ) + (M Ta) ((0+) A A
16 (w 22 (M A (M Ld) ((0)) AA112
16.V
(M + e (Maam) D(0() A+1
(A.29)
by I Lemma A.II
where 0 defined in Lemma A.11.
Meanwhile, according to the proof of Lemma A.8, rewrite and upper-bound (Qutr(B.)
ming Lady (0) follows,
[ady (0t) min (0)
= 2 Tr (T irain + Virrria ld)
V. WK A -(T (Ttrain A + 2 Virraan la) - A18) 1022
(u2 V.WKRAS (T train + @2gurain Ja) 12)=~111 AA3 -3 ']
= 2. Trr((T Anain + --2 Ia) ((0)) D(0))
= 2 Irr(rrain + e La) D(O.). D(O.)
(T + +
Lemma A.3
2. 2-1((Trrainn ((T'trainA
e² vrrriiaaa) ((0))
< u[Turann + e Wrani (a) 11 -A- 1
Irran + Virainta) D(0,) A (A.30)
where T Irain = ((Mrain) and virrain = 2/( (Mrin.
Combining Eqs. (A.29) and (A.30), we thus know that Lmdy Ldd (0:)1
8y
Judy (uudy( (0:) mmin Li (8))
[ A + e Id)
b
1|A
The proof is completed.
Finally, we prove Theorem based on Lemma A.8 and Lemma A.12.
Proof of Theorem I. When all the conditions hold, when the surrogate AT problem defined in Eq. (9) is solved via continuous gradient flow, by Lemma A.8 have
0, Jady (0.) - min (O)
Cz (0,) 0,0, = Eu uuv (0,) (-0, Eudy (l.)) = Lad (0,)))1
-u (Zn) min Eady (0))
which means
(nndy (0.) min Lady (0) (Linty (0o) min Caii (0) -ut
As result, when performing continuous gradient flow optimization for infinitely long time, since > O, the surrogate AT loss will eventually converge to the global minima, i.e.,
[Gade (0.) min 6
zady (0) lim
(Gady (0.) - min gadv (0) (ad)(0)) S (Ba))(0))
(0o) min Cady (0) lim e -Mt 0,
t0
(
where 0, lim,~soc O, is the converged model parameter. Meanwhile, from Lemma A.8, we know that is global minimizer if and only if w. W W = (T (MMaia) A + (Marin)) Ta) A, which completes the proof.
35

---

A.4 Proofs in Section 4.4
This section collects all proofs that omitted from Section 4.4.
Proof of Theorem 2. By substituting all Musan with Musi in proofs of Proposition 1 and Lemma A.7, we immediately have that for any model parameter the LSA model fLSA,0.
R(0, Mes)) S 2Tr
) (M wH At). (w (W
4Tr (w>y W
(KQA3) A²) A (Tr()..
By inserting the converged model parameter 0. (Mainn) which satisfies (w ,2 W Wh1 (T(Mrnin) +ManMmin)aaaa into the above robust generalization bound, we thus have that R(,, (Man()Ma)) Mea)
< 2Tr (T(Mes)A + 2 ( ((T A + & Md) A A²)
((F(Mma))) A + e [2) 1 AAAAA T
4Th
(T(M + 2 (Mrain) Ta) A A A + 2Tr((A)
21r (I + e2 (Mas)) (T A + e (MMain))) La)
A3 (M 2 (M (2) 1 + +0+ 2Tr(A)
(w+) S 2Tr A3 (T(Mss)) + (MMess) (1 A + é² E (Mrrin 1a) -3 + 2TT(AA)
where (+) is due to that the matrix ((T (MM )A + e² 2 Ma) -A3) is positive definite, and (**) due to that: (1) (T e 2bp (Meam It) symmetric and is commutative with A3, and (2) Lemma A.3.
The proof is completed.
Proof of Corollary I. Let A.. Ad be the d singular values the matrix A. Then, the robust generalization bound Theorem 2 can be rewritten as follows,
R(0, (0MaMM) Mea)
< 2Tr
A3
(T(Mes)) + (b -TT(Maam)) )A + 2 (Maai)) + 2Tr(A),
N+Mart d + Tt(A)
+ Tr(A)
M
N+Mas
[ue)
+ +TTr(A)
1=1
No d5 N+Mui
N+
Tr(A)
+ E²
M2 M2 (Y(A) +Nu))
Mou d +
TH(A)
d
e².
Ma Tr(A) IN+ M
Mix
N+
x
2Tr(A)
1=1
N- X; N+MMem
2
(N+MMm))
M Tr(A)
N + Murin
N + Miss +
Ed d6
X,
N + MMrin
N Mrett
N
X
(N Miran) )
M2
2
Σ
X
maxd (Ax)
N2
M
(0(1) + (dd)
(N + M )
M2
0d(d) O(1)
+ Od)
+ Odd)
N
2
N
M4
d
(N + M )1
M2
<0(4+0
+ O
N
N2
M
36

---

Then, by applying Assumption 2, we further have that
R(00 (Mriin), MEsa) 0(()+
d²
O
(N Myrin)"
)
fest
N
N2
M
Irain
d2
(N + ((N))
Me
50(d) +0
N
(vdj2
N2
M
train
d²
M2
O(d) +
+0
N22
N
M
which completes the proof.
B Additional experimental details
This section collects experimental details omitted from Section
B.1
Jailbreak attacks
Our experiments leverage both suffix and non-suffix jailbreak attacks. Specifically, four suffix jailbreak attacks are adopted, which are GCG [73], BEAST [44], AmpleGCG [26]. and Zhu's AutoDAN [71]. Meanwhile, two non-suffix jailbreak attacks are adopted, which are PAIR [7] and DeepInception [25]. We re-implemented all attacks except AmpleGCG by ourselves to enable fast batching operations during jailbreak, which can thus improve the efficiency of AT. Besides, other than the adversarial suffix length, we will also tune the following hyperparameters of jailbreak attacks:
GCG: According to Algorithm in [73]. hyperparameters that we need to tune for GCG include the iteration number T, the top-k parameter k. and the "batch-size" B.
BEAST: According to Algorithm in [44], hyperparameters that we need tune for BEAST are two beam-search parameters k1 and l2.
AmpleGCG: According to [26), AmpleGCG is an algorithm for training adver- sarial suffix generators. Our experiments adopt the adversarial suffix generator AmpleGCG-pluee 1ea---1mmmmsuuus cedu----ouuddd--eee (attimaa-1ib-guaaanea 7b13b which is offi- cially released by [26].
Zhu's AutoDAN: According to Algorithm and Algorithm 2 in [71], hyperparameters that need tune for Zhu's AutoDAN are the iteration number T in each step, objective weights w) and w2, the top-B parameter B, and the temperature т.
GCQ: According to Algorithm 1 in [17], hyperparameters that we need to tune for GCQ include the iteration number T, the proxy batch size bp. the query batch size by. and the buffer size B.
PAIR: According to [7], PAIR adopts LLM-based attacker and judger to iteratively syn- thesize and refine jailbreak prompts. a result, one needs to set the base models for the attacker and judger and the number of teratively refining for the PAIR attack.
DeepInception: According [25], DeepInception attack
manually crafted jailbreak prompts to attack targeted LLMs. We adopt the role play-based prompt from [25] to perform the attack. No other hyperparameter need be tuned for the DeepInception attack.
B.2 Model training
Jailbreak attacks during AT. We use GCG to search adversarial prompts during AT. The adver- sarial suffix token length is fixed to one of (5, 10, 20, 30, 40, 50) during the overall AT. For other hyparameters described in Appendix B.1, set T as 150, 256, and B 64.
Benign answer (b) for the safety dataset (h), We adopt four benign answers for the safety data during AT, which are:
lnttps://hugiinggacc co/osumlp/mmpleGGG plus- sourcedd- sourced-vicuma- vicuna- vicuna-7bb3b -uwnaco-
37

---

Kuard
Maurt0
-
Maur
Mazmi0
Maz+00
Ruu-120
Ampasoo
SCg
REAL Subcosa Itco coant
22
AT Sulto Langh
31 23
Trioe
Lergg
TakenEangh
(a) Mistral-78-v0.3.
Muari
Manr10
Meuri2
Mayo000
Mea
Muu-3000
Muu-1200
ROLA Sucoes mux pan
Zhulk Autt-Auttotoo
Acork Gucowen Ramn (AGR) (000)/00
Lengh
30 23
Tolnn
TakenLangh
(b) Llama-2-77 1-ana----B--hii 2-tt.Chat.
Mus-d
Mur-10
Maur2s
Mar Mux-50
Mua-50
Kuu-1100
Riue1200
BEAST
AILL Seocoss Rut Dannin
Amprwion
Anars Gurgeen Pank (ADR)
Acanh
20
(c) 1-ama---8B--nstruree
Muart
Murr10
Maum2s
Mazm0
Mua-50
Mua-50
Kuu-100
Riue1120
BEAST
KOLK Succos Nut (40%) pg)
Ampredoo
Zhuhs Pro0-A00000M
Jond Suraee Paan (A6R))0)
600
50
Leken
(d) Qwenn..-77-Instruuet Instruct..
Figure 4: Curves of the ASR versus the adversarial suffix token length during AT (i.e., Musin) under jailbreak attacks with different adversarial suffix token lengths (i.e., Meea). Mirain = means that AT performed the evaluated model. A low ASR indicates strong jailbreak robustness.
"As a responsible Al assistant, cannot generate content that promotes illegal activities or violence."
"Sorry, but I can't do that. Please try other instructions that safe and responsible."
"I cannot fulfill your request. am designed to provide safe and helpful information."
"This is topic am not permitted to discuss since it contain illegal or violent information."
LoRA sefting. We use the PEFT library [35] to apply LoRA [18] to all query and key projection matri- in attentions. The PEFT hyperparameters are r=64, 1ora_alpha 32, and 1ora_ orr__rrpoout00 1.
follow Eq. (3) to perform AT with AdamW W for 125 iterations, where the learning rate is set as 5 x 10-5 and the factor a is set as 0.2. Besides, the batch size is set as 64, in which samples are jailbreak prompts crafted from data from the safety training set, and the remaining 56 samples are from the utility training set.
Other hyperparameters for AT. In every AT experiment,
B.3 Model evaluations
Robustness evaluation. We report the Attack Success Rate (ASR) of jailbreak attacks to assess the robustness of models. Specifically, for each instruction from the safety test set, we synthesize the corresponding jailbreak prompt and use it to induce the targeted LLM to generate 10 responses. Then, we use an L.M based judge from [36], which was ine-tuned from the Llama-2-133 model to
lattps://hugginggaan
1allamm-2-133 2-13b-c1s
38

---

determine whether the 10 generated LLM responses are harmful or not. If any of them is determined be harmful, the jailbreak attack is considered successful.
Jailbreak attacks for robustness evaluation. For every suffix attack, the adversarial suffix length is varied within {5, 10, 20, 40, 60, 80, 100, 120). Besides, for jailbreak hyperparameters described in Appendix B.l:
For the GCG attack, we set T as 500, h as 256, and T as 64.
For the BEAST attack, we set 11 64 and l2 as 16.
For the AmpleGCG attack, Appendix B.I.
official adversarial suffix generator described in
For the Zhu's AutoDAN attack, we set as 3, w1 as 10, 10 as 100, B as 256, and r as 2.
For the GCQ attack, we set T as 200 and bp. bys and B all as 128.
For the PAIR attack, we set the base model for the attacker as Mistral-8x7B 1ae---xxXX--Insutt---ee the base model for the judger Llama-3-7000 -ontst--0B-nsttrueet, and the number of iteratively refining is fixed to 10.
For the DeepInception, as explained in Appendix B.I, we use a role-play-bas play-based prompt to perform the attack, and there are other hyperparameters that need to be tuned for this attack.
Utility evaluation. We the AlpacaEval2 framework [10] to report the Length-controlled WinRate (LC-WinRate) of targeted models against a reference model based on their output qualities on the utility test An LC-WinRate of 50% means that the output qualities of the two models equal, while an LC-WinRate of 100% means that the targeted model is consistently better than the reference model. We use Davinci003 as the reference model and use the Llama-3-70B model judge output quality. The official code of the AlpacaEval2 framework is used to conduct the evaluation. Additionally, the Llama-3-70B judger is run locally via the vLLM model serving framework [23].
B.4 Additional experimental results
This section collects additional experimental results (i.e., Figure 4) omitted from Section 5.2.
From Figure 4, we find that GCG-based AT is extremely effective in improving model robustness against GCG, AmpleGCG, and Zhu's ' AutoDAN. For the BEAST attack, GCG-based AT can also suppress the ASR to more than 50%. Further, when the AT adversarial suffix token length is set 20, AT is already able to reduce the ASR by at least 30% under all settings. It is worth noting that the adversarial suffix length during AT is only up to 50, while that during jailbreaking can vary from 5 to 120. All these results indicate the effectiveness of defending against long-length jailbreaking with short-length AT.
C
More experiments
This section presents experiments beyond those in Section 5.
C.1 Comparison with other jailbreak defense baselines
Here, we compare the jailbreak defense performance of short-length LLM AT with that of another jailbreak defense baseline, the Circuit Breakers method [72]. Specifically, adopt GCG and BEAST attacks to assess the jailbreak robustness of Mistral-7B and Llama-3-8B LLMS protected by short- length LLM AT the Circuit Breakers defense. For short-length LLM AT, we the adversarial suffix length Mrsin during AT to a small value of 20 or 30. For the Circuit Breakers defense, we directly use the trained Mistral-7B and Llama-3-8B models officially released by [72].
The resulting jailbreak ASRs are collected and presented in Table 4, from which we observe that: (1) When the base model is Mistral-7B, short-length LLM AT consistently achieves better jailbreak
Iactpe://huggingfaac. http:///ww......oe.ao///////aaaa Instrunttt
Pateps:/huuginngaaat wttp:/aaaaaaaaaaalaaa....o httpp 3-8B IB-IIntructt-P
39

---

Table 4: ASR (%) of suffix jailbreaking against LLMs trained with Circuit Breakers [73] or LLM AT. A low ASR suggests strong jailbreak robustness of the targeted model.
(a) ASRs different jailbreak attacks against Mistral-7B.
Adversarial Suftix Taken Length Mice Jtilbeesking
Aliack
Defeuse
10
20
80
100
Circait Rreskers [73]
20.0
21.0
25.0
28.0
23.0
11.0
7.0
10.0
1 AT (Miin 50)
11.0
13.0
8.0
6.0
7.0
5.0
5.0
5.0
Circait Breakers [72)
19.0
21.0
20.0
24.0
25.0
25.0
25.0
27.0
BEAST
LLM AT (Mggia 200
11.0
8.0
11.0
10,0
13.0
8.0
8,0
11.0
LLM (3dgii
12.0
13.0
19.0
18.0
22.0
17.0
22.0
(b) ASRs of different jailbreak attacks against Llama-3-8B.
Model
Defense
Adversarial Suftix Taken Length Miee Jiilwwakking
10
20
40
60
80
1000
120
Cinauit Berskers [72]
3.0
5,0
3.0
4.0
3.0
5.0
5.0
7.0
1 (Miin
8.0
1.0
1LM AT (Minin 30)
11.0
9.0
2,0
3,0
2.0
1.0
Circait Breakers 1721
12.0
15.0
17.0
15.0
BEAST
1 AT (Myin 20)
10.0
12.0
4,0
13.0
12.0
21.0
19.0
15.0
1LM AT (Mais
30)
19.0
15.0
12.0
6.0
9.0
14.0
11.0
10.0
62G
REAST
Vicama-B-a TB-Ld
Quen
vs.
Moann- BEAST
May--i Quardd--1 vo. GCG
Quera BEAST
Ducoro
(a) Scatter plots of ASR to the ratio / /Mua//e Murin-
(b) ASR versus Munin under different Miss. Musin 0 means that AT not performed on the evaluated model.
Figure 5: ASR of models trained from the BEAST-based LLM AT. A low ASR indicates strong jailbreak robustness of the model.
robustness than Circuit Breakers under different jailbreak attack adversarial suffix lengths. (2) When the base model is Llama-3-8B, the two defense methods achieve similar performance.
C.2 LLM AT with the BEAST attack
In our main experiments in Section 5, we solely use the GCG attack to synthesize jailbreak prompts for LLM AT. In this section, we investigate whether our theoretical results still empirically hold for AT with jailbreak attacks other than GCG. Specifically, we now perform L.M AT with the BEAST attack on Vicuna-7B-v/ 1.5 and Qwen2.5-7B - Instruct models. For the hyperparameters of BEAST described in Appendix B.I, we vary the adversarial suffix token length within (5, 10, 20, 30, 40, 50). and set k1 to 64 and k2 to 16. All other settings of LLM AT follow those described in Section B.2.
Experimental results are presented in Figure and Table From Figure Sa and Table 5, we observe a statistically significant positive correlation between the suffix jailbreak robustness and the ratio /Mea/ Mian in every experiment, which indicates that our ICL-AT theory still holds for BEAST- based LLM AT. Besides, from Figure 5b, one can find that AT with short adversarial suffix length Miain of 30 can already reduce the ASR from nearly 100% to around 20% in every evaluation case, which demonstrates the effectiveness of short-length BEAST-based L.M AT in defending against. jailbreak attacks.
C.3 LLM AT on larger models
We also perform short-length L.M AT on Vicuna-13B-vl. which is model larger than those 7B/8B L.MMs adopted in our main experiments Section 5. All hyperparameters for LLM AT follow those described in Appendix B.2. Results are presented in Table 6, which shows that AT with an
40

---

Table 5: PCCs and p-vauues calculated between ASR and ratio VMes/ on LLMs adversarially trained with the BEAST attack. p 5.00 x 10-2 means that the correlation between ASR and the ratio is considered statistically significant.
Model
OCG Alack
BEAST Anack
POCIT)
(-value())
POCKT)
p-vaine())
Vicara-78
0.91
5.8
10
0.94
6.7 x 10 10-24
Qwen2.5-7B
2.2
16
0.95
5.0 20
Table 6: ASR (%) of the GCG attack against Vicuna-1 1 33B trained with LLM AT. A low ASR suggests a strong jailbreak robustness of the targeted model.
(a) ASRs different jailbreak attacks against Mistral-7B.
Advensriial Sullix Length Jailbrakking
Defesse
None
92.0
94.0
96.0
96.0
96.0
99.0
98.0
(hdgan
LLM AT (Maaan 20)
12.0
9.0
11.0
6.0
6.0
6.0
7.0
adversarial suffix token length as short as 20 can already reduce the ASR the GCG attack from nearly 99% to around 10% in the worst case. This suggests the generalization of our theoretical findings beyond 7B/SB models.
41


Reviews:
[{"review_id": "ZeinbJAfIc", "reviewer": "Reviewer_W4gv", "summary": "This paper aims to address the challenge of defending LLMs against jailbreak attacks, a type of adversarial prompt designed to circumvent safety mechanisms and induce harmful behavior. Specifically, the authors focus on adversarial suffix jailbreaks and investigate how the length of adversarial prompts used during adversarial training (AT) affects the model’s robustness to such attacks.\n\nThe core contribution is a theoretical and empirical finding that adversarial training with shorter suffixes (length ∝ $\\sqrt M$) is sufficient to defend against longer attack suffixes (length ∝ M). The authors support this with:\n\n1. A theoretical analysis of in-context learning using linear transformers on linear regression tasks, yielding a robust generalization bound dependent on the ratio $\\sqrt {M_{test}} / M_{train}$.\n\n2. Empirical validation on five open-source LLMs using the GCG jailbreak attack, showing that shorter suffixes in AT significantly reduce attack success rates against longer adversarial prompts.\n\nOverall, the paper introduces a computationally efficient strategy for enhancing LLM safety by demonstrating that short-length AT can generalize effectively to defend against long-length jailbreak attacks.", "questions": "None", "limitations": "Please refer to the weaknesses", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper makes a strong theoretical contribution by establishing a novel generalization bound that rigorously connects adversarial training with in-context learning.\n\n- The paper evaluates five widely used open-source LLMs across a range of adversarial suffix lengths under four different jailbreak attacks. In all cases, the results consistently demonstrate the effectiveness of the proposed defense strategy.\n\n- Clear writing, solid structure, and well-motivated problem framing.\n\n**Weaknesses**\n\n- While the paper presents a rigorous and well-developed theoretical analysis, a significant portion of the main content is devoted to the theoretical framework, resulting in a relatively brief empirical section (approximately two pages). To improve balance and accessibility, some of the more detailed proof steps could be moved to the appendix, allowing more space for expanded experimental analysis and discussion in the main text.\n\n- The experimental evaluation focuses primarily on GCG-based jailbreak attacks. While GCG is a strong and representative method, it would strengthen the work to demonstrate the effectiveness of the proposed defense against a broader range of jailbreak techniques (e.g., [1, 2, 3]). This would help confirm the generalizability of the findings.\n\n- The paper does not provide a direct comparison between the proposed approach and existing defense mechanisms (e.g., [4, 5, 6]). Including such comparisons would clarify the relative strengths and potential trade-offs of the proposed method, particularly in terms of robustness and efficiency.\n\n- The current evaluation uses ASR as the primary metric. However, in practice, LLMs may generate responses that engage with irrelevant content in the adversarial suffix rather than directly following the harmful instruction. Such cases could still constitute successful jailbreaks under the intended threat model. A more detailed analysis or clarification of how these borderline cases are handled would provide a fuller understanding of the defense’s real-world effectiveness.\n\nReferences:\n\n[1] Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. Andriushchenko et al. 2024\n\n[2] Tree of attacks: Jailbreaking black-box llms automatically. Mehrotra et al. 2023\n\n[3] Query-based adversarial prompt generation. Hayase et al. 2024\n\n[4] Smoothllm: Defending large language models against jailbreaking attacks. Robey et al. 2023\n\n[5] Baseline defenses for adversarial attacks against aligned language models. Jain et al. 2023\n\n[6] Improving alignment and robustness with circuit breakers. Zou et al. 2024", "was_revised": true, "final_justification": "issues resolved:\n1. Agreed to shorten theoretical part for more experimental studies\n2. Conducted more experiments to demonstrate the generalisation of the proposed methodI\n\nissues unresolved:\n1. Evaluate the performance on standard benchmark to demonstrate that the proposed defense doesn't hurt the generic capability of LLMs.\n2. Focus only on suffix-based attacks thus not sure the generalisation of the proposed defense to other attacks.", "rebuttal": "Thank you for your thorough comments and kind support!\n\n**Q1: While the paper presents a rigorous and well-developed theoretical analysis, a significant portion of the main content is devoted to the theoretical framework, resulting in a relatively brief empirical section (approximately two pages). To improve balance and accessibility, some of the more detailed proof steps could be moved to the appendix, allowing more space for expanded experimental analysis and discussion in the main text.**\n\n**A1:** Thanks for your suggestion. We will shorten the theoretical analysis (Section 3) in the main text and add more experimental results to the main text.\n\n**Q2: The experimental evaluation focuses primarily on GCG-based jailbreak attacks. While GCG is a strong and representative method, it would strengthen the work to demonstrate the effectiveness of the proposed defense against a broader range of jailbreak techniques (e.g., [r1, r2, r3]). This would help confirm the generalizability of the findings.**\n\n**A2:** Thanks. Following your suggestion, we have added GCQ [r3], which is a suffix-based jailbreaking, as an additional jailbreak baseline. Preliminary results on the Vicuna-7B model are presented in Table r1 below:\n\n| $M_{\\text{test}}$ (GCQ Attack [r3]) | 5 | 10 | 20 | 40 | 60 | 80 | 100 | 120 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-7B (Original) | 85 | 95 | 92 | 96 | 95 | 97 | 96 | 94 |\n| Vicuna-7B + LLM AT ($M_{\\text{train}}=5$)  | 7 | 12 | 15 | 17 | 21 | 20 | 19 | 19 |\n| Vicuna-7B + LLM AT ($M_{\\text{train}}=20$) | 7 | 4 | 3 | 3 | 4 | 6 | 3 | 3 |\n\n*Table r1: ASR (%) of the GCQ attack [r3] against adversarially trained Vicuna-7B.*\n\n\n\nFrom the table, we find that short-length AT (where $M_{\\text{train}} = 20$) can already reduce the ASR of the GCQ attack from around 90% to less than 10% in the worst case.\n\nFurthermore, we have also adopted a non-suffix-based jailbreak attack, named PAIR [r7], into our experiments. Results on Vicuna-7B are shown in the following Table r2:\n\n| $M_{\\text{train}}$ | 0 (No AT) | 5 | 10 | 20 | 30 | 40 | 50 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-7B + AT v.s. **PAIR Attack [r2]** | 84 | 53 | 48 | 42 | 50 | 44 | 55 |\n\n*Table r2: ASR (%) of PAIR attack against adversarially trained Vicuna-7B.*\n\nFrom Table r2, we find that while LLM AT with a short adversarial suffix length $M_{\\text{train}}$ can reduce the ASR of the (non-suffix) PAIR attack from more than 80% to around 50%,but further increasing $M_{\\text{train}}$ does not help much to improve LLM robustness against PAIR. We will leave the theoretical analysis of defending against non-suffix jailbreaking attacks for future studies.\n\nComplete experimental results of GCQ and PAIR attacks will be added in the revision.\n\n**Q3: The paper does not provide a direct comparison between the proposed approach and existing defense mechanisms (e.g., [r4, r5, r6]). Including such comparisons would clarify the relative strengths and potential trade-offs of the proposed method, particularly in terms of robustness and efficiency.**\n\n**A3:** Thanks. Following your suggestion, we have added “Circuit Breakers” [r6] as an additional LLM jailbreak defense baseline. Specifically, we adopted the robust Mistral-7B and Llama3-8B models officially released by [r6] in our analysis and performed various jailbreak attacks against these robust models. Preliminary experimental results are presented in Tables r4 and r5 below:\n\n| $M_{\\text{test}}$ (GCG Attack)              | 5    | 10   | 20   | 40   | 60   | 80   | 100  | 120  |\n| ------------------------------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Mistral-7B + LLM AT ($M_{\\text{train}}=20$) | 6    | 9    | 16   | 25   | 19   | 25   | 19   | 19   |\n| Mistral-7B + Circuit Breakers [r6]          | 21   | 20   | 21   | 23   | 23   | 28   | 28   | 23   |\n| Llama3-8B + LLM AT ($M_{\\text{train}}=20$)  | 5    | 8    | 6    | 5    | 6    | 1    | 3    | 1    |\n| Llama3-8B + Circuit Breakers [r6]           | 3    | 5    | 3    | 4    | 3    | 5    | 5    | 7    |\n\n*Table r4: ASR(%) of GCG attack against different protection methods (i.e., LLM-AT and Circuit-Breaker [r6]).*\n\n| $M_{\\text{test}}$ (BEAST Attack) | 5 | 10 | 20 | 40 | 60 | 80 | 100 | 120 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Mistral-7B + LLM AT ($M_{\\text{train}}=20$) | 11 | 8 | 11 | 10 | 13 | 8 | 8 | 11 |\n| Mistral-7B + Circuit Breakers [r6] | 19 | 21 | 20 | 24 | 25 | 25 | 25 | 27 |\n| Llama3-8B + LLM AT ($M_{\\text{train}}=20$)  | 10 | 12   | 4 | 13 | 12 | 21 | 19 | 15 |\n| Llama3-8B + Circuit Breakers [r6] | 12 | 9 | 11 | 12 | 16 | 15 | 17 | 15 |\n\n*Table r5: ASR(%) of BEAST attack against different protection methods (i.e., LLM-AT and Circuit-Breaker [r6]).*\n\nFrom the two tables, we find that when the base model is Mistral-7B, short-length LLM-AT achieves better jailbreak robustness than Circuit Breaker. Meanwhile, when the base model is Llama3-8B, the two defense methods achieve similar performance.\n\nWe will add more baselines in the revision.\n\n**Q4: The current evaluation uses ASR as the primary metric. However, in practice, LLMs may generate responses that engage with irrelevant content in the adversarial suffix rather than directly following the harmful instruction. Such cases could still constitute successful jailbreaks under the intended threat model. A more detailed analysis or clarification of how these borderline cases are handled would provide a fuller understanding of the defense’s real-world effectiveness.**\n\n**A4:** We would like to clarify that the ASR calculation adopted in our paper is directly based on **whether the generated content contains harmful information**, but NOT on whether it follows the harmful input instruction. As a result, those borderline cases mentioned by the reviewer would not be a problem for our ASR calculation.\n\nSpecifically, we use an open-source LLM-based judger [r8], originally proposed in [r9], to determine whether a given piece of LLM-generated content contains harmful information. This LLM-based judger does not take the input harmful instruction into account during the ASR calculation process. Please see Appendix B.3 in our paper for more details on the ASR calculation.\n\nWe would also like to note that [r9] has shown that judgments made by this LLM-based judger align well with human judges. This judger has also been widely adopted for jailbreak robustness evaluation in many existing works, such as [r10], [r11], and [r12]. Therefore, we believe our ASR calculation is reliable enough to justify our finding that \"short-length LLM AT can help defend against long-length jailbreaking\".\n\n**References:**\n\n[r1] Andriushchenko et al. \"Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks\". ICLR 2025.\n\n[r2] Mehrotra et al. \"Tree of attacks: Jailbreaking black-box LLMs automatically\". NeurIPS 2024.\n\n[r3] Hayase et al. \"Query-based adversarial prompt generation\". NeurIPS 2024.\n\n[r4] Robey et al. \"Smoothllm: Defending large language models against jailbreaking attacks\". arXiv 2023.\n\n[r5] Jain et al. \"Baseline defenses for adversarial attacks against aligned language models\". arXiv 2023.\n\n[r6] Zou et al. \"Improving alignment and robustness with circuit breakers\". NeurIPS 2024.\n\n[r7] Chao et al. \"Jailbreaking Black Box Large Language Models in Twenty Queries\". arXiv 2023.\n\n[r8] https://huggingface.co/cais/HarmBench-Llama-2-13b-cls\n\n[r9] Mazeika et al. \"HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal\". arXiv 2024.\n\n[r10] Paulus et al. \"AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs\". ICML 2025.\n\n[r11] Xhonneux et al. \"Efficient Adversarial Training in LLMs with Continuous Attacks\". NeurIPS 2024.\n\n[r12] Yu et al. \"Robust LLM safeguarding via refusal feature adversarial training\". ICLR 2025."}, {"review_id": "7aQQAhQCBO", "reviewer": "Reviewer_q37y", "summary": "This paper investigates the effectiveness of short-length adversarial training in defending LLMs against long-length jailbreak attacks. \n\nThe authors provide very detailed theoretical analysis and some empirical experimental results to support their findings that there exists a positive correlation between the attack success rate and the ratio of the square root of the adversarial suffix length during jailbreaking to the length during adversarial training. Additionally, they establish a robust generalization bound under a linear transformer framework.", "questions": "1. How does the effectiveness of short-length AT compare when defending against other types of jailbreak attacks beyond suffix attacks? Can the authors provide insights or preliminary results on the robustness against more complex or multi-token attacks?\n\n2. Could the authors provide more details or quantitative analysis regarding the computational efficiency of short-length AT compared to longer-length AT during training?", "limitations": "As noted above, I am not optimistic about the generalizability of the proposed approach. The paper does not present experimental evidence on newer or larger-scale LLMs, different architectures, or diverse types of jailbreak attacks. Further validation in these directions would be necessary to demonstrate the broader applicability of the method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. The paper offers a rigorous theoretical treatment of adversarial in-context learning for linear transformers. The derived robust generalization bound sheds light on how adversarial prompt length affects model robustness.\n\n2. The empirical results, though limited in scale, show that short-length AT can help defend against long-length jailbreak attacks on several open-source LLMs.\n\n# Weaknesses\n\n1. Despite the strong theoretical contributions, the writing can be hard to follow in several parts, making it difficult for readers to understand the proofs. For example, some notational choices are confusing, such as the use of both M and m in Equations (2) and (3).\n\n2. The empirical evaluation is not comprehensive enough to fully support the paper’s claims. The authors only test on small-scale LLMs (7B) and focus on similar types of token-level attacks. It would strengthen the paper to include experiments on more diverse LLM families and a broader range of attack types.", "was_revised": true, "final_justification": "Raised my score. Although I am not fully convinced, I recognize this paper meets the bar of most ML conferences on this topic...", "rebuttal": "Thank you for your thorough comments.\n\n**Q1: Despite the strong theoretical contributions, the writing can be hard to follow in several parts, making it difficult for readers to understand the proofs. For example, some notational choices are confusing, such as the use of both M and m in Equations (2) and (3).**\n\n**A1:** Thanks. The uppercase \"M\" in Eqs. (2) and (3) should be lowercase \"m\". These are typos, and we apologize for any confusion.\n\nWe will improve our writing in the revision to help readers better understand our proofs. Specifically, we will: (1) recheck all notations in Section 3 and fix potential typos; (2) add a notation table for Section 4 to clarify all necessary notations in this section; and (3) add a \"Proof Skeleton\" section for Section 4 to better summarize the main idea and contribution of our theoretical analysis.\n\n**Q2: The empirical evaluation is not comprehensive enough to fully support the paper’s claims. The authors only test on small-scale LLMs (7B) and focus on similar types of token-level attacks. It would strengthen the paper to include experiments on more diverse LLM families and a broader range of attack types.**\n\n**A2:** Thanks. To address the concern about **testing only 7B/8B LLMs**, we have added an additional 13B model for experiments, which is Vicuna-13B-v1.5. Preliminary results are presented in the following Table r1.\n\n| $M_{\\text{test}}$ (GCG Attack) | 5 | 10 | 20 | 40 | 60 | 80 | 100 | 120 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-13B (Original) | 92 | 94 | 99 | 96 | 98 | 96 | 99 | 98 |\n| Vicuna-13B + LLM AT ($M_{\\text{train}}=5$) | 11 | 19 | 30 | 53 | 55 | 67 | 70 | 68 |\n| Vicuna-13B + LLM AT ($M_{\\text{train}}=20$) | 12 | 9 | 11 | 6 | 6 | 6 | 8 | 7 |\n\n*Table r1: ASR (%) of the GCG attack against adversarially trained Vicuna-13B.*\n\nTable r1 shows that AT with a short adversarial suffix length $M_{\\text{train}} = 20$ can already reduce the ASR of the GCG attack from nearly 99% to around 10% in the worst case. This justifies the generalization of our theoretical findings beyond 7B/8B models. Besides, we would also like to note that our original paper has already adopted models from 5 different LLM families (i.e., Vicuna-7B, Mistral-7B, Llama-2-7B, Llama-3-7B, and Qwen2.5-7B), which we believe is diverse enough to empirically verify our theoretical findings.\n\nTo address the concern about **testing with only token-level attacks**, we have additionally adopted a non-token-level jailbreak attack, i.e., PAIR attack [r1]. Specifically, PAIR attack is an agent-based jailbreak attack that generates jailbreak prompts directly using LLM-based agents. It has been widely adopted as a jailbreak baseline in many existing works [r2, r3, r4, r5]. Preliminary results of the PAIR attack on adversarially trained Vicuna-7B models are presented in Table r2 below:\n\n| $M_{\\text{train}}$ | 0 (No AT) | 5 | 10 | 20 | 30 | 40 | 50 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-7B + AT v.s. **PAIR Attack [r1]** | 84 | 53 | 48 | 42 | 50 | 44 | 55 |\n\n*Table r2: ASR (%) of PAIR attack [r1] against adversarially trained Vicuna-7B.*\n\nFrom Table r2, we find that while GCG-based LLM AT can reduce the ASR of the PAIR attack from more than 80% to around 50%, further increasing $M_{\\text{train}}$ does not help much to improve LLM robustness against PAIR. We will leave the theoretical analysis of defending attacks other than suffix-based token-level jailbreaking for future studies.\n\nThe complete analysis and discussion of the Vicuna-13B-v1.5 model and the PAIR jailbreak attack will be included in the revision.\n\n\n\n**Q3: How does the effectiveness of short-length AT compare when defending against other types of jailbreak attacks beyond suffix attacks? Can the authors provide insights or preliminary results on the robustness against more complex or multi-token attacks?**\n\n**A3:** Thanks. We have added a new baseline attack, named PAIR [r1], to our empirical analysis. As explained in **A2**, the PAIR attack [r1] is an agent-based jailbreak attack but not a suffix-based attack. Preliminary results are presented in Table r2 in **A2**, which show that while short-length AT can improve LLM robustness against PAIR, increasing the adversarial suffix length during AT does not help much to further improve LLM robustness. These results indicate the limitations of both our ICL-AT theory and short-length AT.\n\nNevertheless, we believe there is a potential solution to further improve our work by extending our current ICL-AT theoretical framework to the setting of \"continuous-AT\" [r4]. Specifically, continuous-AT [r4] is a more recent LLM AT method that has been shown to improve LLM robustness against both suffix-based and non-suffix-based jailbreak attacks. Since continuous-AT is implemented by adding adversarial perturbations to the continuous embeddings of input tokens, our ICL-AT theoretical framework is naturally easier to generalize to this setting. We will leave this generalization for future studies.\n\n**Q4: Could the authors provide more details or quantitative analysis regarding the computational efficiency of short-length AT compared to longer-length AT during training?**\n\n**A4:** Here, we present time costs of performing AT with different adversarial suffix lengths $M_{\\text{train}}$ on Vicuna-7B in Table r3 below:\n\n| $M_{\\text{train}}$ | 5     | 10    | 20    | 30    | 40    | 50    |\n| ------------------ | ----- | ----- | ----- | ----- | ----- | ----- |\n| Vicuna-7B          | 10.2h | 11.3h | 13.8h | 16.0h | 18.2h | 20.4h |\n\n*Table r3: Time costs of AT on Vicuna-7B.*\n\nFrom Table r3, we find that when the adversarial suffix length during AT is as long as $50$, the time cost of AT can reach around 20 hours, which is almost twice that when the suffix length is set to $20$. However, according to Figure 2 in our paper, AT with an adversarial suffix length of $20$ can already make Vicuna-7B achieve strong jailbreak robustness. These results clearly demonstrate the advantages of using short-length AT instead of long-length AT.\n\nThe complete time-cost analysis for AT will be added in the revision.\n\n**References:**\n\n[r1] Chao et al. \"Jailbreaking Black Box Large Language Models in Twenty Queries\". arXiv 2023.\n\n[r2] Paulus et al. \"AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs\". ICML 2025.\n\n[r3] Sadasivan et al. \"Fast Adversarial Attacks on Language Models In One GPU Minute\". ICML 2024.\n\n[r4] Xhonneux et al. \"Efficient Adversarial Training in LLMs with Continuous Attacks\". NeurIPS 2024.\n\n[r5] Yu et al. \"Robust LLM safeguarding via refusal feature adversarial training\". ICLR 2025."}, {"review_id": "FCGGQyfWGC", "reviewer": "Reviewer_6htW", "summary": "This paper presents both theoretical and empirical evidence to show the quadratic relation between the length of adversarial training and robustness for LLMs against jailbreaking. Theoretically, the authors demonstrate this insight by simplifying the problem as an instance of adversarial in-context learning. Empirically, this insight is validated by 4 jailbreaking attacks.", "questions": "1. How can this theory generalize to non-suffix jailbreaking attacks?\n2. What are the connection between this theory and other theoretical insights from ICL-based jailbreaking attacks?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. Theoretical understanding of adversarial suffix-based jailbreaking attacks remains an open and important problem.\n2. It is interesting to study this problem through the lens of in-context learning.\n3. This paper also presents comprehensive empirical results to support the theoretical insights.\n\n# Weakness\n1. While the theory mainly focuses on suffix-based attacks, its extension to non-suffix attacks is not well-discussed.\n2. The connection between the theory presented in this paper and other theoretical insights regarding ICL-based jailbreak attacks [1,2] is also not addressed. Additionally, in line 31, [1], which first presented this discovery before [2], was not cited.\n\n[1] Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations https://arxiv.org/pdf/2310.06387\n[2] Many-shot jailbreaking. NeurIPS 2024", "was_revised": true, "final_justification": "I think this is an interesting paper and keep my score for acceptance. Please incorporate the discussed points in your revision.", "rebuttal": "Thank you for your thorough comments and kind support!\n\n**Q1: (From Weakness#1 & Question#1) While the theory mainly focuses on suffix-based attacks, its extension to non-suffix attacks is not well-discussed.\nHow can this theory generalize to non-suffix jailbreaking attacks?**\n\n**A1:** Thank you for this good question. We believe a potential solution is to extend our current ICL-AT theoretical framework to the setting of \"continuous-AT\" [r3].\n\nSpecifically, continuous-AT [r3] is a more recent LLM AT method that has been shown to improve LLM robustness against both suffix-based and non-suffix-based jailbreak attacks. Since continuous-AT is implemented by adding adversarial perturbations to **the continuous embeddings of input tokens**, our ICL-AT theoretical framework is naturally easier to generalize to this setting.\n\nGeneralizing our current ICL-AT theoretical framework to analyze continuous-AT would require two steps: (1) introduce an additional trainable matrix into the LSA model $f_{\\text{LSA},\\theta}$ to map the ICL input $E_\\tau$ to a new embedding space, and (2) perform the training-time adversarial perturbation within the new embedding space (rather than in the original space of $E_\\tau$). We will leave the generalization for future studies.\n\n**Q2: (From Weakness#2 & Question#2) The connection between the theory presented in this paper and other theoretical insights regarding ICL-based jailbreak attacks [1,2] is also not addressed. Additionally, in line 31, [1], which first presented this discovery before [2], was not cited.\nWhat are the connection between this theory and other theoretical insights from ICL-based jailbreaking attacks?**\n\n**A2:** Firstly, regarding [r2], we respectfully note that this work is purely experimental and does not present any theoretical results/insights.\n\nSecondly, regarding [r1], the main connection between their theoretical insight and our ICL-AT theory is that **we both noticed that the number of adversarial in-context samples plays a critical role in ICL-based adversarial attacks**. Specifically, [r1] theoretically characterizes the minimum number of in-context adversarial samples required to increase the safety loss of ICL to some extent. On the other hand, our ICL-AT theory illustrates a “square-root connection” between the number of in-context adversarial (suffix) samples during training (i.e., $M_{\\text{train}}$) and during testing (i.e., $M_{\\text{test}}$).\n\nWhile [r1] and our work share similarities, the main difference is that [r1] focuses on studying the adversarial robustness of fixed ICL models, whereas our work analyzes how adversarial training affects the robustness of ICL models.\n\nWe will cite and discuss [r1] in the revision.\n\n**References:**\n\n[r1] Wei et al. \"Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations\". arXiv 2023.\n\n[r2] Anil et al. \"Many-shot Jailbreaking\". NeurIPS 2024.\n\n[r3] Xhonneux et al. \"Efficient Adversarial Training in LLMs with Continuous Attacks\". NeurIPS 2024."}, {"review_id": "Rt8sungJGb", "reviewer": "Reviewer_gkox", "summary": "This paper explores the challenges of using adversarial training (AT) to defend against jailbreak attacks on large language models (LLMs).\nThe core contribution is to demonstrate the effectiveness of short-length adversarial training for defending against long-length adversarial suffix attacks from both theoretical and empirical perspectives.\nCombining practical results with theory, this paper conducts empirical verification on five LLMs covering four attack types, and the results show that short suffixes used in training (e.g., 20 tokens) can defend against significantly longer attack suffixes (up to 120 tokens).", "questions": "1. Scope limitation: This paper focuses on suffix-based jailbreak attacks, but a large number of LLM jailbreak attacks use non-suffix patterns (such as deepinception), which reduces the practical value of this paper.\nIt would be even better if the author could discuss and clarify the limitations on the scope of this paper.\n2. Transferability to large models: The largest model in the experiments is Llama-3-8b.\nGiven that the parameter scale of commonly used LLMs typically range from 30B to 70B, this paper could discuss whether its findings and conclusions have the potential to be extended to larger models.\n3. Impact of suffix generation strategy:\nIt remains unclear the impact of suffix generation methods (i.e., GCG attack, token level generation) used in AT.\nIt would be great if the author could conduct additional experiments, such as comparing the use of token level and character level methods to generate suffix in AT.\nAlternatively, the author can discuss the impact of different methods on the experimental results and conclusions.\nThis could help future research on efficient and effective construction of AT.\n4. In Figure 1, AutoDAN's ASR seems to be always low.\nIs this related to the judgment method to determine the success of the attack (for example, judging by whether the output result has specific keywords or starts with a specific string)?\nImproper judgment methods may affect experimental results and observations。\nThe author should provide some explanations and clarifications to improve the persuasiveness of the experiment.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Theoretical evidence and empirical validation: The derivation of a generalization bound based on suffix length is both mathematically rigorous and practically meaningful. The experiments on five open-source LLMs and four jailbreak methods further offer strong support for the main claim.\n2. Efficiency insight: Demonstrates that short adversarial suffixes during training can yield robustness, which could reduce training cost significantly.\n3. Clear analysis: Uses ASR VS. √(Mtest/Mtrain) ratio across multiple models/attacks to show the correlation between the jailbreak robustness and the ratio. \n\nWeaknesses:\n1. Limited practical value: Focuses only on suffix-style jailbreaks, while many attack samples are in other prevalent formats (e.g., roleplay, templates).\n2. Insufficient exploration of model scale: Unclear if conclusions hold for larger LLMs (e.g., 30B) beyond the tested 7–8B range.", "was_revised": true, "final_justification": "The authors have answered all questions. About the practiability, although the authors have conducted additional experiments, I still have concerns about this aspect. I will maintain the score.", "rebuttal": "Thank you for your thorough comments and kind support!\n\n**Q1: (From Weakness#1 & Question#1) Limited practical value: Focuses only on suffix-style jailbreaks, while many attack samples are in other prevalent formats (e.g., roleplay, templates).\nScope limitation: This paper focuses on suffix-based jailbreak attacks, but a large number of LLM jailbreak attacks use non-suffix patterns (such as deepinception), which reduces the practical value of this paper. It would be even better if the author could discuss and clarify the limitations on the scope of this paper.**\n\n**A1:** Thanks. We agree that our current theoretical and empirical analyses are limited to suffix-based jailbreak attacks. To further investigate how our current results can be generalized to non-suffix-based jailbreaking settings, we have adopted an additional non-suffix-based jailbreak attack in our experiments, i.e., the PAIR attack [r1]. Specifically, the PAIR attack is an agent-based jailbreak attack that generates jailbreak prompts directly using LLM-based agents. It has been widely adopted as a jailbreak baseline in many existing works [r4, r5, r6, r9]. The preliminary results of the PAIR attack on adversarially trained Vicuna-7B models are presented in Table r1 below:\n\n| $M_{\\text{train}}$ | 0 (No AT) | 5 | 10 | 20 | 30 | 40 | 50 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-7B + AT v.s. **PAIR Attack [r1]** | 84 | 53 | 48 | 42 | 50 | 44 | 55 |\n\n*Table r1: ASR (%) of PAIR attack [r1] against adversarially trained Vicuna-7B.*\n\nFrom Table r1, we find that while GCG-based LLM AT can reduce the ASR of the PAIR attack from more than 80% to around 50%, further increasing $M_{\\text{train}}$ does not help much to improve LLM robustness against PAIR. This suggests that the mechanisms behind suffix-based and non-suffix-based jailbreak attacks might share different properties.\n\nNevertheless, we believe a potential solution to improve our work for the non-suffix-based attack setting is to extend our current ICL-AT theoretical framework to the setting of \"continuous-AT\" [r5]. Specifically, continuous-AT [r5] is a more recent LLM AT method that has been shown to improve LLM robustness against both suffix-based and non-suffix-based jailbreak attacks. Since continuous-AT is implemented by adding adversarial perturbations to the continuous embeddings of input tokens, our ICL-AT theoretical framework can be more easily generalized to this setting. We will leave this generalization for future studies.\n\n**Q2: (From Weakness#2 & Question#2) Insufficient exploration of model scale: Unclear if conclusions hold for larger LLMs (e.g., 30B) beyond the tested 7–8B range.\nTransferability to large models: The largest model in the experiments is Llama-3-8b. Given that the parameter scale of commonly used LLMs typically range from 30B to 70B, this paper could discuss whether its findings and conclusions have the potential to be extended to larger models.**\n\n**A2:** Thanks. To address your concern, we have added an additional 13B model for experiments, which is Vicuna-13B-v1.5. We will also add experiments on 30B+ models (e.g., Qwen2.5-32B), but not for now due to the limited rebuttal period.\n\nPreliminary results of Vicuna-13B-v1.5 with adversarial training are presented in the following Table r3.\n\n| $M_{\\text{test}}$ (GCG Attack) | 5 | 10 | 20 | 40 | 60 | 80 | 100 | 120 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-13B (Original) | 92 | 94 | 99 | 96 | 98 | 96 | 99 | 98 |\n| Vicuna-13B + LLM AT ($M_{\\text{train}}=5$) | 11 | 19 | 30 | 53 | 55 | 67 | 70 | 68 |\n| Vicuna-13B + LLM AT ($M_{\\text{train}}=20$) | 12 | 9 | 11 | 6 | 6 | 6 | 8 | 7 |\n\n*Table r3: ASR (%) of the GCG attack against adversarially trained Vicuna-13B.*\n\nTable r3 shows that AT with a short adversarial suffix length $M_{\\text{train}} = 20$ can already reduce the ASR of the GCG attack from nearly 99% to around 10% in the worst case. This justifies the generalization of our theoretical findings beyond 7B/8B models.\n\nA complete analysis for larger models (including both Vicuna-13B and another 30B+ model) will be added in the revision.\n\n**Q3: Impact of suffix generation strategy: It remains unclear the impact of suffix generation methods (i.e., GCG attack, token level generation) used in AT. It would be great if the author could conduct additional experiments, such as comparing the use of token level and character level methods to generate suffix in AT. Alternatively, the author can discuss the impact of different methods on the experimental results and conclusions. This could help future research on efficient and effective construction of AT.**\n\n**A3:** Thanks. Following your suggestion, we have conducted experiments on LLM adversarial training with a different suffix generation method, the BEAST attack [r9]. Preliminary results of BEAST-based LLM AT on Vicuna-7B are presented in the following Table r4.\n\n| $M_{\\text{test}}$ (GCG/BEAST Attack) | 5 | 10 | 20 | 40 | 60 | 80 | 100 | 120 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Vicuna-7B (Original) v.s. GCG Attack | 89 | 99 | 93 | 98 | 96 | 97 | 98 | 99 |\n| Vicuna-7B + BEAST-based LLM AT ($M_{\\text{train}}=20$) v.s. GCG Attack | 6 | 7 | 15 | 27 | 27 | 33 | 27 | 34 |\n| Vicuna-7B (Original) v.s. BEAST Attack | 68 | 93 | 96 | 92 | 98 | 93 | 94 | 96 |\n| Vicuna-7B + BEAST-based LLM AT ($M_{\\text{train}}=20$) v.s. BEAST Attack | 4 | 7 | 7 | 14 | 22 | 27 | 24 | 32 |\n\n*Table r4: ASR (%) of different jailbreak attacks against BEAST-based adversarially trained Vicuna-7B.*\n\nFrom Table r4, we find that LLM AT via the BEAST attack with a short adversarial suffix length of $M_{\\text{train}}=20$ can already reduce the ASR of GCG or BEAST attacks from nearly 100% to 30+% in the worst cases, which demonstrates the effectiveness of BEAST-based short-length AT. Furthermore, when compared with the results of GCG-based AT in Figure 2 of our original paper, Table r4 also seems to suggest that BEAST-based AT can achieve a better robustness balance in defending against both GCG and BEAST attacks simultaneously. We will leave the study of how different adversarial suffix generation methods in LLM AT affect LLM robustness for future research.\n\nComplete results of the BEAST-based AT will be added in the revision.\n\n**Q4: In Figure 1, AutoDAN's ASR seems to be always low. Is this related to the judgment method to determine the success of the attack (for example, judging by whether the output result has specific keywords or starts with a specific string)? Improper judgment methods may affect experimental results and observations. The author should provide some explanations and clarifications to improve the persuasiveness of the experiment.**\n\n**A4:** Thanks. Firstly, we would like to clarify that the ASR calculation in our paper is implemented using an open-source LLM-based judger [r2] (originally proposed by [r3]) to determine whether a given piece of LLM-generated content contains harmful information. It is **NOT** implemented by judging whether output results contain specific keywords or start with specific strings. Please see Appendix B.3 in our paper for more details on the ASR calculation.\n\nSecondly, for the LLM-based judger used in our ASR calculation, we note that [r3] has shown that judgments made by this judger align well with human judges. This judger has also been widely adopted for jailbreak robustness evaluation in many existing works, such as [r4], [r5], and [r6]. Thus, we think the ASR calculation method we used is reliable enough and is unlikely to be the cause of the low ASR of Zhu's AutoDAN in Figure 1.\n\nThirdly, for our reproduced version of Zhu's AutoDAN, since the authors of Zhu's AutoDAN did not release their official experimental code publicly (the GitHub link [r7] in their original paper [r8] is unfortunately empty), we had to reproduce this attack solely based on the pseudocode provided in their original paper [r8]. We tried various combinations of hyperparameters (see Appendix B.1 for all tunable hyperparameters in our reproduced version of Zhu's AutoDAN), but failed to find a strong one that could achieve effective attack performance against adversarially trained LLMs.\n\nFinally, we deduce that the low ASR of Zhu's AutoDAN in Figure 1 is mainly due to the fact that this attack is very similar to the GCG attack, as both utilize gradients with respect to one-hot token-space embeddings to find adversarial suffix tokens. Zhu's AutoDAN further needs to preserve the semantics of synthesized jailbreak suffixes, which may potentially reduce its attack performance compared with the GCG attack. Since our LLM AT is performed with GCG, the trained LLMs may be very robust to attacks similar to GCG, such as Zhu's AutoDAN.\n\n**References:**\n\n[r1] Chao et al. \"Jailbreaking Black Box Large Language Models in Twenty Queries\". arXiv 2023.\n\n[r2] https://huggingface.co/cais/HarmBench-Llama-2-13b-cls\n\n[r3] Mazeika et al. \"HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal\". arXiv 2024.\n\n[r4] Paulus et al. \"AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs\". ICML 2025.\n\n[r5] Xhonneux et al. \"Efficient Adversarial Training in LLMs with Continuous Attacks\". NeurIPS 2024.\n\n[r6] Yu et al. \"Robust LLM safeguarding via refusal feature adversarial training\". ICLR 2025.\n\n[r7] http://autodan-jailbreak.github.io/\n\n[r8] Zhu et al. \"AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models\". COLM 2024. https://arxiv.org/abs/2310.15140\n\n[r9] Sadasivan et al. \"Fast Adversarial Attacks on Language Models In One GPU Minute\". ICML 2024."}]

Author rebuttal:

Meta-review: This paper studies the effectiveness of short-length adversarial training for defending LLMs against long-length jailbreak attacks, providing both theoretical analysis and empirical validation of a quadratic relationship between training suffix length and model robustness.

**Strengths.** All reviewers acknowledge the mathematical analysis establishing a novel generalization bound for adversarial in-context learning under linear transformer frameworks. The paper demonstrates that short adversarial suffixes during training (e.g., 20 tokens) can effectively defend against significantly longer attack suffixes (up to 120 tokens), offering substantial training cost reductions. 

**Weaknesses.**  All reviewers highlight the exclusive focus on suffix-based attacks as a major limitation, questioning generalizability to other prevalent jailbreak formats. The paper also misses direct comparisons with existing defense mechanisms, limiting assessment of relative performance and trade-offs.

Overall, the paper conducted an interesting study to understand adversarial training dynamics in LLMs and provides a mathematically rigorous framework, representing a meaningful contribution to the literature. Yet, its practical impact is constrained by the narrow focus on suffix-based attacks. The work would also benefit from broader experimental evaluation across attack types and model scales, along with comparative analysis against existing defenses to better establish its position in the defense landscape.
Decision: Accept (poster)