Understanding and Improving Continuous LLM Adversarial Training via In-context Learning Theory
Authors: Shaopeng Fu
Di Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM adversarial training
Jailbreak attacks
In-context learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We theoretically explain why continuous adversarial training helps LLMs defend against jailbreak prompts from the token space via the in-context learning theory.

Abstract: Adversarial training (AT) is an effective defense for large language models (LLMs) against jailbreak attacks, but performing AT on LLMs is costly. To improve the efficiency of AT for LLMs, recent studies propose continuous AT (CAT) that searches for adversarial inputs within the continuous embedding space of LLMs during AT. While CAT has achieved empirical success, its underlying mechanism, i.e., why adversarial perturbations in the embedding space can help LLMs defend against jailbreak prompts synthesized in the input token space, remains unknown. This paper presents the first theoretical analysis of CAT on LLMs based on in-context learning (ICL) theory. For linear transformers trained with adversarial examples from the embedding space on in-context linear regression tasks, we prove a robust generalization bound that has a negative correlation with the perturbation radius in the embedding space. This clearly explains why CAT can defend against jailbreak prompts from the LLM's token space. Further, the robust bound shows that the robustness of an adversarially trained LLM is closely related to the singular values of its embedding matrix. Based on this, we propose to improve LLM CAT by introducing an additional regularization term, which depends on singular values of the LLM's embedding matrix, into the objective function of CAT. Experiments on real-world LLMs demonstrate that our method can help LLMs achieve better jailbreak robustness-utility tradeoff. The code is available at https://github.com/fshp971/continuous-adv-icl.

Published as a conference paper at ICLR 2026
UNDERSTANDING AND IMPROVING CONTINUOUS AD- VERSARIAL TRAINING FOR LLMS VIA IN-CONTEXT LEARNING THEORY
Shaopeng Di Wang
Provable Responsible AI and Data Analytics (PRADA) Lab King Abdullah University of Science and Technology, Saudi Arabia (shaopeng.. fu, di di.wang)Okaust, edu.sa
ABSTRACT
Adversarial training (AT) is effective defense for large language models (LLMs) against jailbreak attacks, but performing AT on LLMs is costly. To improve the efficiency of AT for LLMs, recent studies propose continuous AT (CAT) that searches for adversarial inputs within the continuous embedding space of LLMs during AT. While CAT has achieved empirical success, its underlying mechanism, i.e., why adversarial perturbations in the embedding space can help LLMs defend against jailbreak prompts synthesized in the input token space, re- mains unknown. This paper presents the first theoretical analysis of CAT on LLMs based on in-context learning (ICL) theory. For linear transformers trained with adversarial examples from the embedding space on in-context linear regression tasks, we prove robust generalization bound that has negative correlation with the perturbation radius in the embedding space. This clearly explains why CAT can defend against jailbreak prompts from the LLM's token space. Further, the robust bound shows that the robustness of an adversarially trained LLM is closely related to the singular values of its embedding matrix. Based on this, we propose improve LLM CAT introducing an additional regularization term, which depends on singular values of the LLM's embedding matrix, into the objective function of CAT. Experiments can help LLMs achieve a better jailbreak robustness-utlity tradeoff. The code is available https: : / /gittub....// con/ fshp971 cont inuous adv icl.
real-world LLMS demonstrate that our method
INTRODUCTION
While large language models (LLMs) are increasingly adopted in various real-world applications, their safety is found to be compromised by jailbreak attacks (Wei et al., 2023). By feeding jailbreak prompts, which are specially constructed harmful instructions, one "jailbreak" safety-aligned targeted L.LMs to induce harmful behaviors in them. To ensure the robustness of LLMS against jailbreak attacks, one of the most effective defenses adversarial training (AT) (Mazeika et al., 2024; Fu et al., 2025), which trains LLMs on synthesized jailbreak prompts to help them better recognize and refuse these harmful inputs. However, the synthesis of jailbreak prompts during AT usually requires solving discrete optimization problems and is thus computatiinnconsuuingg (Zou al., 2023; Chao al., 2023), which restricts the use of AT for LLMs in practice.
To improve the efficiency of AT for LLMs, recent studies introduce continuous AT (CAT) (Xhonneux al., 2024; Sheshadri et al., 2024; Arditi et al., 2024; Dékány et al., 2025), which performs AT on LLMss with adversarial inputs synthesized in the LLMs' continuous token embedding space. Compared with the vanilla AT for LLMs, CAT can use projected gradient descent (PGD; Madry et al. 2018) to search for adversarial examples in the embedding space, which is significantly faster than that in vanilla AT where one needs to perform heuristic search to find jailbreak prompts in the input token space. However, despite the empirical success of CAT, the reason behind it is still unknown. In fact, the training data CAT and vanilla LLM AT are very different from each other: data in CAT sequences of embedding vectors, while data in vanilla AT sequences of token

---

Published as a conference paper at ICLR 2026
indices. This raises the research question: Why can adversarial perturbations in the embedding space help LLMs learn to defend against jailbreak prompts from the original input token space?
In light of recent advances in understanding LLM jailbreak robustness via in-context learning (ICL) theory (Fu et al., 2025; Kumano et al., 2025; Anwar et al., 2025), this paper presents the first theoret- ical analysis of LLM CAT,, also based ICL theory. We rigorously study how AT helps improve the robustness of linear transformers trained on linear regression tasks against in-context suffix adversar- ial attacks. To simulate the embedding space adversarial perturbation process in CAT, we introduce an additional trainable embedding matrix into linear transformers and perform adversarial perturba- tions on ICL imput embeddings obtained from this embedding matrix, rather than on the vanilla ICL inputs. Under our new ICL embedding AT theoretical framework, we prove a robust generalization upper bound for linear transformers trained via ICL, embedding AT. This robust bound has a neg- ative correlation with the embedding space adversarial perturbation radius during ICL embedding AT, which clearly explains why adversarial perturbations in the embedding space can help LLMS defend against jailbreak prompts from the original input space.
Besides, our robust generalization upper bound is closely related to the singular values of the em- bedding matrix in linear transformers. Specifically, it suggests that an adversarially trained linear transformer with an embedding matrix that has "not too large nor too small singular values" would enjoy a small robust upper bound and thus strong adversarial robustness. Based on this finding, we further propose Embedding Regularized Continuous AT (ER-CAT), a new LLM AT approach improved from CAT by introducing the variance of the embedding matrix singular values as an additional regularization term into the objective function of the original CAT. The motivation is to simultaneously reduce large singular values and increase small singular values, thereby helping to improve the robustness of LLMs according to our proven theory. To verify ER-CAT, we conducted experiments on six real-world LLMs and six common jailbreak attacks. Results show that compared with original CAT, ER-CAT helps LLMs achieve a better jailbreak robustness-utin tradeoff.
2 RELATED WORKS
Jailbreak attacks. Jailbreak prompts (Wei et al., 2023) are adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015) that can induce targeted unsafe behaviors from LLMs. Existing jail- break attacks include token-level and prompt-level attacks. Token-level attacks synthesize jailbreak prompts via modifying/inseringg tokens in prompts with heuristic methods (Zou al., 2023; Sadasi- van et al., 2024; Hayase et al., 2024; Zhu et al., 2024; Paulus et al., 2025; Jin et al., 2024; Liao & Sun, 2024; Andriushchenko et al., 2025), while prompt-level attacks use prompts crafted by humans (Wei et al., 2023; Li et al., 2023; Shen et al., 2024; Xu et al., 2024) or LLM-based agents (Chao et al., 2023; Liu et al., 2024b;a; Sabbaghi et al., 2025; Zhang et al., 2025) to jailbreak LLMs. Though the synthesis of jailbreak prompts in token-level attacks and prompt-level attacks is different from each other, CAT has been shown to be able to defend against both types of attacks.
LLMs adversarial training (AT). To tackle jailbreak attacks, effective method is to align LLMs via AT (Madry et al., 2018) to help them better recognize and refuse harmful inputs. A standard LLM AT aims solve a minimax problem that minimizes the training loss most adversarial jailbreak prompts (Mazeika et al., 2024). However, searching for jailbreak prompts in the discrete token space of LLMS resource-intensive, limiting the broader application of LLM AT. More recent studies propose continuous AT (CAT) for LLMs, in which adversarial examples are embedding vectors obtained from adversarially perturbing token embeddings of the original prompts (Xhonneux al., 2024: Casper et al., 2024; Sheshadri et al., 2024: Yu et al., 2025). Such a perturbation process can be efficiently performed with gradient- based optimizations. which thus significantly reduces the computational burden of L.M AT. Dékány et al. (2025) adopt both jailbreak prompts and perturbed prompt embeddings as adversarial examples to further improve the performance of L.LM AT.
In-context learning (ICL) theory. ICL theory aims to understand how taaaformer-based LLMs can make decisions for different task-specific sequential context inputs (i.e.. "prompts") without adjusting model parameters. Existing ICL works have proven that one can construct explicit trans- formers layer by layer to mimic the process of learning a variety of function classes (Garg et al., 2022: Von Oswald et al., 2023; Ahn et al., 2023; Chen et al., 2024; Wang et al., 2024; Mahankali et al., 2024; Li et al., 2025). Efforts have also been made to analyze how one-layer transformers can learn ICL prediction abilities from massive task-specific contextual data (Lu et al., 2024; Magen
2

---

Published as a conference paper at ICLR 2026
et al., 2024; Frei & Vardi, 2025; Shi et al., 2024; Zhang et al., 2024; Yang et al., 2024b; Huang al., 2023; Wu et al., 2024; Lin et al., 2024; Lu et al., 2025). More recent studies have leveraged ICL theory to amalyze the adversarial robustness of LLMs. Anwar et al. (2025) find that by only per- turbing a single in-context sample in the entire real space, one can manipulate the ICL, prediction of transformers to arbitrary results. Fu et al. (2025) study AT of ICL models under more restricted and realistic ICL adversarial attacks, where each ICL sample can only be perturbed within a restricted space. They prove that AT on contextual data with a very small number of perturbed in-context sam- ples can already help trained transformers achieve strong robustness. Kumano et al. (2025) show that adversarially pretrained transformers focus more on robust features (Ilyas al., 2019) and thus can generalize to downstream tasks robustly without additional AT. Our analysis mainly stems from Fu et al. (2025), with the goal of explaining the robust generalization ability of continuous AT.
PRELIMINARIES
Large language models (LLMs). An LLM is function that maps sequential inputs to sequential outputs based on a parameterized distribution pg. Let x E Viol be an input prompt of length |æc| where := {1. .................... [V]} is the token space. The probability that the LLM generates response E Vesl of length |y] is: po (o((z) = I po (wvla x @ V1-(t-1)), where "œ" denotes concatenation.
Jailbreak attacks. Current jailbreak attacks be divided into token-level and prompt-level at- tacks. Given two token sequences z²(h) and ydh), where z((h) is harmful instruction and y(h) is targeted harmful response, a standard token-level attack concatenates a synthesized adversarial suf- fix (+()) to the prompt c((h) to form a jailbreak prompt that increases the probability of the LLM in generating (h)), The synthesis of the suffix (x()) can be formalized as solving the below problem, Po (2x @ 2.(").
min =
Meanwhile, a prompt-level attack uses an attack oracle A, which can be human experts or Al agents, to directly rewrite the original harmful prompt (f) to a jailbreak one a(h), as shown below, log po (h) (,()))).
min
(((()(()()))
It should be noted that performing both token-level and prompt-level attacks is resource-consumiigg as solving Eq. (I) requires sophisticated discrete optimizations, while solving Eq. (2) requires human annotation or additional computing resources for AI agents to perform inference.
Continuous AT for LLMs. Let (*)) be a safety dataset, where each sample (x, y, g) consists of a harmful prompt æ. targeted harmful response y, and safe reference response ÿ. Additionally, let D(a) be utility dataset, where each sample (I, y) consists of a pair of normal instruction and its answer. Then, Mazeika et al. (2024) formalize the first AT algorithm for LLMs as solving the following optimization problem:
min
E (E.9.D)
loglo(ūlă") +
E (a.
logpo(vlx))
(3)
Adversarial Loss
Utility Loss
where == arg max log (oyl) is the jailbreak prompt for the harmful input-output pair (I, y) from the safety set and B(x,)) y) denotes the search space of jailbreak prompts. In Eq. (3), the adversarial loss helps LLMs learn to respond harmlessly even when the most adversarial jailbreak prompts are present, while the utility loss helps retain the utility of pre-trained LLMs. As explained before, finding strong jailbreak prompts â E B(x, y) from the search space determined by the used attacks is costly, which limits the efficiency of LLM AT.
=
More recent studies suggest using continuous AT (CAT) to reduce the computational overhead of vanilla LLM AT. Let E(-) be the embedding function of the LLM fo. with an embedding matrix F E Idx Vl as its parameter, where W.E. E IRd is the embedding vector for the to- ken V. For any token sequence x E yiul, E(- maps it its embedding sequence as E(x) := (W.F, ;,2; W.F W.E...)) E Rdx|=l, CAT is then formalized as solving the below prob- lem (Xhonneux al., 2024).
log (o( + +8) ((())((x) 8*)
min
E (a. (()
E log bogpo(yl)) ())x)) (a.))D())
(4)
3

---

Published as a conference paper at ICLR 2026
where 8* |argmax, max 66 1a.l.....,, [11 laSe log (og(o()))()) + 5]] Rdx[= is the most adversarial per-
turbation 8t for the harmful input-output pair (x,y) from the safety set D(h, c 0 is the em- bedding space perturbation radius, a > 0 weight parameter, and (8())+88 8= := ((E(x)) + 81) ... (a(p)) + ()) E [Rdx|=l is the perturbed harmful prompt embeddings. The idea behind LLM CAT in Eq. (4) is to adopt adversarially perturbed embedding sequences as adversarial exam- ples rather than jailbreak prompts in the vanilla LLM AT in Eq. (3). Additionally, the embedding space adversarial perturbation 8 can usually be searched via the fast rojjectedgraddees descent (PGD) method (Madry et al., 2018), which makes CAT efficient in practice.
4 THEORETICAL ANALYSIS FOR CONTINUOUS AT
While LLM CAT has achieved empirical success (Xhonneux et al., 2024; Casper et al., 2024; She- shadri et al., 2024: Dékány et al., 2025), the mechanism behind it, i.e. why adversarial perturba- tions in the embedding space help LLMs defend against jailbreak prompts from the token space, remains unclear. This section tackles this research question by conducting a theoretical analysis for CAT based on in-context learning theory (Zhang et al., 2024; Fu et al., 2025; Kumano et al., 2025).
In-context learning (ICL) theory. In the ICL theory, a prompt input of length N, specified by lask indexed by т, is formalized
sequence 1r,,,, , Vr, N Nrrrg) where the first are task-specific in-context training samples, and the last item, Fr.g, is the query sample. Then, the goal of an ICL model is to make a prediction for the query sample Fr,y solely based on the N in-context training data.
N labeled samples {(I=,,, Vr.a) y+.,) |
ICL linear regression. Our analysis focuses on training ICL models on different linear regression tasks. Suppose T is task index and we E Rdo is the corresponding task weight drawn from w,, NO, Idn). We assume that each ICL training point Fy,s (1 S i NN) and the query point Ir,4 are drawn from ær,á Zr,q A0, where A E Rddoddo is the covariance matrix, and their labels are Yr.i = w Fr, and Vr.q = w, Tr.q. Then, the ICL input Z, specified by the task T is given by
Z, = Tr,1
Nr,N VT,N
Iz.q
E ER [R((o+1)x(NA))
(5)
Vr,1
Other notations. We denote [n]] := {I, ,.] for any n E N+ For any A E 12=xm, we denote ||A||2,,c = maxigism |A,.l12 ||A|| be the operator norm, and ||A|e be the Frobenius norm. Besides, (A), Amax(A) and Amin(A) denote its i-th largest, largest, and smallest eigenvalues, while (A), T (A), and amin (A) denote its i-th largest, largest, and smallest singular values. Finally, we denote Tr(A)= = Ei= Ar, for any A E Runn, We use standard Big O notation (-).
In the remainder, explain why it can approximate real-world L.LM CAT under the ICL theoretical framework. A robust generalization bound will then be proved for linear transformers trained from ICL embedding AT. Based on this bound, we will explain why CAT can work and how to further improve CAT.
will first establish an ICL embedding AT' problem for linear transformers and
4.1 ICL ADVERSARIAL TRAINING IN EMBEDDING SPACE
Linear self-attention with embedding module (LSA-E). Linear self-attention (LSA) models are linear transformers that have been widely used for theoretical ICL analysis (Zhang et al., 2024; Shi et al., 2024; Frei & Vardi, 2025). However, the LSA model studied in the previous work does not have input embedding module and thus cannot be naturally adopted for the analysis of LLM CAT, which requires performing adversarial perturbations in the input embedding space. To tackle this challenge, we design a novel I.SA- wibbd-Emeedding (LSA-E) model to approximate real-world CAT in our theoretical analysis. Specifically, let E()) be an embedding function that maps any ICI. input Z,& E R(d++1)x(N+1)) to its ICL embedding matrix E E(Z,) E (ddd-))((++) as follows:
E F
WE WE xy, Vr,N
E(Z,) :=
E R((+I)x(++))
(6)
Vr.1
where F. E dx is the trainable parameter of the embedding function and d is the dimension of the embedding space. Intuitively, the function E(-) in Eq. (6) aims map each in-context point from
4

---

Published as a conference paper at ICLR 2026
the original input space R to the embedding space F via a linear mapping. With the embedding function E() in Eq. (6), the LSA-E model fLSAE, 9 is then formalized as below,
+(2z)+VVEC2 O (+) = [(2z)) WV EV,, 8(Z,) ( E RR(+II(N+1)
N
where WKQ E [Rd++)x((d=1 is matrix fused from the key and query projection matrices, W E R((+1)x(d+1) is the value projection matrix, and 0 = (WE WkQ WV) contains all trainable parameters of the ISAA- model. The prediction (Z-) for the query point Izgy given by the right-bottom entry of the LSA-E model output, i.e., Fy.8 (Z,, == fLSAE, (Zr (d+1)) If
follow Zhang et al. (2024); Frei & Vardi (2025); Fu & Wang (2024) to write matrices Wko
W
and WV W =
O E (KQ, V). where WH E Raddd w IR, and
(w=2)T
W22
12 22 E Rdx1, then the LSAAE model prediction $2.0 be further simplified as follows,
((Z))
(
W.k (m22 (T)
0g,((z)) = ((wh))
W222
W ZT.4
N
ICL embedding AT for LSA-E models. To approximate the real-world setting of LLM CAT, the theoretical ICL embedding AT also adopts ICL adversarial examples found in the embedding space train LSA-E models, An ICL adversarial example in the embedding space is defined as below,
Eady (Z,,
W 24,1
7,1
W N + 5 W WErg
[R((++)((N+)), (8)
Vr,1
Vr,N
where Z, is the ICL input for the task (see Eq. (5)), W is the parameter of the embedding function (() (see Eq. (6)), and A == (8 added the embeddings of the N in-context training points. The prediction of the LSA-E model SISSAF, for the adversarial example gady (Zr, A is then given by the following Ng.0 (Zr, A/),
7,N) E d denotes all adversarial perturbations
gadv (Z,, A gadv (Zr, AT)
WKQ (W-K0) 21
gody (Zr, A = ((WY)
m222))
W FF..q-
(9)
With all these notations, the ICL embedding AT for LSA-E model eventually formalized the following minimax optimization problem.
1 9 L || ol Se 2 11ady (Z,, (Z, A.) V/r.g
Lady min LISAE (0) min
(10)
Where >> 0 is the embedding space adversarial perturbation radius and the expectation E, is calcu- lated over the randomness of Wr, , UT,N, Zr..g. The restriction in the inner maximization in Eq. (10) ensures that each adversarial perturbation _ is confined within the ball-sphere ||&5,,12 ll S c.
Robust generalization risk for LSA-E models. We use the ICL suffix adversarial attack (Fu et al., 2025) to assess the robustness of ICL models. Nevertheless, we note that our experiments Sec- tion 5 also consider attacks beyond suffix jailbreaking. Specifically, given an ICL input Z, with context length N, the ICL suffix adversarial attack adversarially perturbs the last M (M N) in-context training points of Z, as follows,
[TT11 Vr.1
IrNN-M
TT.N- + 50
ΣT.N + 5%
ZT,9 )00)
(1I)
M++1
Dr.N
where 50: E da is the adversarial perturbation added to the i-th ICL suffix point Tr.N- Al- though the ICL suffix adversarial example 7 in Eq. (11) looks similar to the adversarial example defined in Eq. (8), the mechanisms behind them are very different: in Eq. (II), adversarial pertur- bations are directly added in-context points, whereas in Eq. (8), perturbations are added to the embeddings of these in-context points. The robust generalization risk R.p. M (0) with perturbation and adversarial suffix length M for an LSA-E model fLSAE,0 then defined as below,
1
P.M M (0) =
max 7 1|A0T|22.05
10 (2...) Vt.gl
(12)
112,00
5

---

Published as a conference paper at ICLR 2026
where 0g.0 is the LSA-E prediction function in Eq. (7), Z is the ICL suffix adversarial example 011) E R contains all perturbations added to the suffix of Z and p > 0 is the adversarial perturbation radius for the suffix attack, which restricts each
in Eq. (11), 4 == (69
perturbation of to the ball-sphere ||_ S p. A lower robust risk p. (0) indicates stronger adversarial robustness of the model fLSAE, and vice versa.
4.2 BRIDGING ICL EMBEDDING AT AND LLM CONTINUOUS AT
Before introducing our main theoretical results, here we explain why the established ICL embedding AT in Eq. (I0) be good artifact for approximating real-world LLM CAT in Eq. (3) under the ICL theory by analyzing the similarities between them.
Firstly, LSA-E models are very similar to real-world LLMs. Ahn et al. (2024) empirically show that the linear self-attention module in LSA-E models share similar properties with those non-linear ones in LLMs and thus are useful for theoretically understanding LLMs. We further argue that the embedding processes of LSA-E models and real-world LLMs are also very similar. If we replace each token in an LLM prompt with its one-hot encoding vector defined over the token vocab- ulary space, the embedding process for each token in an LLM can be seen as matrix multiplication between the LLM's embedding matrix and the corresponding token's 's one-hot encoding. This matrix multiplication-basee LLM prompt embedding process is almost identical to the ICL input embed- ding process shown in Eq. (6), where input features are also lincarly transformed by the LSA-E model's embedding matrix. Therefore, we believe the embedding spaces of both LSA-E models and LLMs are similar, as they are both obtained via linear transformation.
Secondly, the training goals of ICL embedding AT and LLM CAT are very similar to each other. The two AT problems both aim to enhance models' robustness by training them on sequential data where their embeddings are adversarially perturbed. The only difference is that in ICL embed- ding AT, the goal of adversarial perturbations is to reduce the utility of linear regression prediction made by LSA-E models, while in LLM CAT such a goal is to induce harmful content from LLMs.
Finally, the adversarial robustness of LSA-E models is also very similar to the jailbreak ro- bustness of LLMs. Fu et al. (2025) has already illustrated why ICL suffix adversarial attacks are similar to real-world jailbreak attacks through theoretical and empirical justifications. Since we also leverage this ICL suffix adversarial attack to assess the robustness of LSA-E models, we believe that analyzing the robust generalization ability of LSA-E models will effectively help us understand how LLMs trained from CAT gain robustness against jailbreak attacks.
4.3 ROBUST GENERALIZATION BOUND OF ICL EMBEDDING AT
We now start to establish a robust generalization bound for the LSA-E model trained via ICL em- bedding AT as formalized in Eq. (10), The derivation consists of three steps: (1) derive an upper bound Cady (0) for the original loss function C in ICL embedding AT (see (10) and formalize surrogate AT problem that would minimize the upper bound LLSAE (0); (2) calculate the closed-form solution for the previously obtained surrogate ICL embedding AT problem; and (3) prove robust generalization bound for the LSA-E model trained with the surrogate problem.
Surrogate ICL embedding AT. The surrogate problem for Eq. (10) is formalized as
min
Eady (SA (0) = minn (1 + (2(0) + (0) + {x))).
(13)
(6) == _ ((( is the surrogate objective function, and where čady (T
E(Z.)E(Z.)
W W (wxke
Ur.o 12
(())) 2 E (ww))
w
W IT.9 W.Irr
N
2x
W ...
WER,NN Vr.N
T
W-KQ
(2) N
w 1
E
W
Vr.1
(w
2
(WExT.1
W.ETNN Vr.N
(0()) = N
E
((wn)T
113]
E
vr.1

---

Published as a conference paper at ICLR 2026
((0) = 2E* || 12 E
IlW flwk&w"
The new objective function Cndy (6) is a closed-form upper bound for the original ICL embedding AT loss function Cudby ( as shown in the following Lemma (see Appendix B.2 for the proof).
Lemma 1. For the objective function (T) in ICL embedding AT (Eq. (10)) and the objective (0) in surrogate ICI. embedding AT (Eq. (13)), we uniformly have L LSAE (0) s LSAE ()) for any 8 (WE, WWk,, WV).
function Cad LSAE
The reason for studying the surrogate ICL embedding AT problem in Eq. (13) instead of the original problem is because the objective function LSSAE (0) in the original AT problem difficult to tackle in closed-form manner. With the new surrogate Cady (f), which is in closed-form, one can easily analyze the training dynamics of the LSA-E model trained from the surrogate problem. Further, since the surrogate objective function is an upper bound of the original one, minimizing it can also help to reduce the original AT loss and thus improve the robustness of the trained LSA-E model.
Closed-form solution of the surrogate problem. To solve the new surrogate AT problem in Eq. (13), we first make the below Assumption on the initialization of the model parameter 0. Assumption 1. Let S > 0 be parameter and θ E Rdxd be any matrix satisfying |100 ||
0aad and OA * Oaxd- We assume that WV (0) (0)==0xxx 01xd
( and wadwwkoo) ( 0lxd
(COOT
0dx1
Remark 1. Assumption I is widely adopted in the ICL theoretical analysis (Zhang et al., 2024; Frei & Vardi, 2025; Wu al., 2024; Fu et al., 2025). The idea behind Assumption to (1) zero out terms w22 KQ KQ WY, and w12 that do not contribute to the ICI. prediction function $g.0 in Eq. (7) w12"
KQ and V and (2) zero out terms W22
21 to ensure symmetric initialization.
Under Assumption 1, the optimal solution for the LSA-E model trained from surrogate embedding ICL AT is calculated the following Theorem (see Appendix B.3 for the proof).
Theorem 1 (Optimal solution of surrogate ICL embedding AT). Suppose Assumption I holds and fiSAE, is trained from the surrogate embedding AT problem defined in Eq. (13) with continuous
gradient Then, the optimal model parameter 0. := (W.,, W.Q ,W) should satisfies w.,12 KQ
KQ
w _,21 = ".Y12 = w\21 0dx11 w. _ = 0, WW,,, = Odxd, and
(W. wKOw (T [W.TwA(WF5" + IT -1 "W.A.
where NN = ( TTA)&&)))) E Ro
Applying Theorem to the LSA-E model prediction function in Eq. (7), the optimal prediction fune- tion 0y (.) given by the model fLSAE, trained from surrogate ICL embedding AT is as follows,
1 00.0 = N (((x))(WF))) (WW.))(WTTAAOWO (W.TNA(WWE) + IT(A)e)A))
W Ax
(14)
Robust generalization ability. Finally, we leverage the robust generalization risk Rady to assess the robustness of the optimal model fLSAB(6.) trained from surrogate ICL embedding AT. For- mally, we prove a robust generalization upper bound for the robust risk Rady (fx), as shown in the following Theorem 2 (see Appendix B.4 for the proof).
Theorem 2 (Robust generalization upper bound). Suppose Assumption 1 holds, d S do, and 0_ is the solution of the surrogate ICL embedding AT in Eq. (13) obtained from Theorem We have
(1 Mp²/N2) Omin (WF)+
Rady (0.) & O0(
+0(1)
Remark 2. Theorem 2 additionally requires that the embedding space dimension d of the LSA-E model be no larger than the input in-context sample dimension dn- Such a requirement means that the L.SAE E models would implicitly compress the input data.
7

---

Published as a conference paper at ICLR 2026
4.4 IMPLICATIONS
So far, we have calculated the optimal ICL prediction function 9y.0, obtained from surrogate em- bedding ICL AT Eq. (14) and further prove a robust generalization bound for in Theorem 2. We now start to investigate how these results can help to understand and improve CAT for LLMss
-mpedding-space adversarial perturbations provably enhance input-space adversarial ro- bustness. The robust generalization bound in Theorem 2 clearly shows that the robust risk Rp. (0.) of the trained LSA-E model, which is calculated based on adversarial ICL examples from the in- put space, has a negative correlation with the embedding-spee space adversarial perturbation radius c. A large perturbation radius in the embedding-spee -ppcce can help reduce the robust upper bound and thus improve the robustness of the trained LSA-E model against adversarial ICL examples. This explains the main mechanism behind ICI. embedding AT and also that in CAT for real-world L.Mss.
The role of the embedding matrix in ICL robust generalization. An interesting observation from Eq. (14) is that the optimal ICL prediction function gg.0- 0- depends only on the embedding matrix W.F but not remaining LSA-E model parameters R2 and WV. Therefore, a "good" embedding ma- trix W. is expected to provide strong robustness for the ICL model. Besides, from Theorem 2, we have two insights:
(1) if those large singular values of W are not "loo large", then the numer- ator of the first term in the robust upper bound can be reduced, which helps to reduce the overall bound; and (2) if those small singular values of W.E. are not "too small", it helps increase the denominator of the first term in the robust upper bound, which also helps to reduce the overall bound. Thus, may expect an LSA-E model or even real- world LLM have an embedding matrix that has "not too large nor too small singular values" for strong model robustness.
Improve CAT with an optimized embedding matrix. Based on the previous analysis, we now propose Simbedding-Regularizee Regularized continuous AT (ER-CAT), a new AT method for LLMs designed by introducing an additional regularization term, defined as the variance of all singular values of the LLM embedding matrix, into the objective function of CAT in Eq. (4). Concretely, training an LLM fa via ER-CATIA is formalized as solving the following optimization problem:
E (i -(W
min CER-CAT(O, a, a)
:== == min 0
(C) CAT Ioss Fq. (4)
+3-
(15)
d
Embeediing Regularization Term
where B > 0 is the coefficient for the regularization term and oW the mean of all singular values of W The reason for using the variance of singular values regularization term is that minimizing it can help to reduce too large singular values and increase too small singular values of the embedding matrix simultaneously, which, as explained before, helps to reduce the overall robust upper bound in Theorem 2. In addition, while in theory the singular values of W in Eq. (15) not differentiable, in practice their gradient calculation can be automatically handled by native PyTorrc functions. This enables us to implement our ER-CAT method easily.
is
EMPIRICAL ANALYSIS OF ER-CONTINUOUS AT
In this section, we follow Eq. (15) to perform the theory-inspired ER-CAT on real-world LLMs, which can further help empirically justify our proved robust generalization bound in Theorem 2.
5.1 EXPERIMENTAL SETUP
Models. We adopt six common pre-trained L.Mss, which are: Vicuna-7B-vl.55 (Zheng et al., 2023), Mistral-7B-IIntruut--oo (Jiang et al., 2023), Llama-2-7B-Cha (Touvron et al., 2023), Llama-3- 8B-Instruct (Grattafiori et al., 2024), (0wen2.5-7B-Instruue (Yang et al., 2024a), and Gemma-2B. it (Team et al., 2024). All models were downloaded from the Hugging Face model repository.
Datasets. During LLM AT, we follow Xhonneux et al. (2024) to use the training set of Harm- bench (Mazeika et al., 2024) the safety data and UltraChat 200K (Ding al., 2023) as the utility data. During robustness evaluation, we follow Fu et al. (2025) to use a safety datasets that consists of the first 50 samples from the test set of Harmbench (Mazeika al., 2024) and the first 50 samples
8

---

Published as a conference paper at ICLR 2026
Table 1: ASR on different models and attacks, low ASR indicates strong model robustness.
Model
Type
Argus ASR (8) &
GUG
BEAST
GC3
AusaDAN
Deepinecption
PAIR
Original
Vicuna-BB
CAT
12.6
14.4
4.6
04
5.8
7.8
ER-CAT (Ours)
16.4
16.2
6.4
2.2
8.2
16.0
Original
74.6
6588
69.8
43.0
49.8
36.0
Mistral-8
CAT
7.4
5.
3.2
0.8
1.0
12.2
EX-CAT
7.6
666
3.2
16.2
Original
41.0
18.2
5.6
7.2
30,8
24.2
Llama
23.6
17.2
13.4
ER-CAT (Ours)
15.6
10.4
1.2
2.0
Original
30.8
6.0
41.8
Llnma-3.1-88
CAT
3.4
4,4
0.0
0.9
0.0
5.01
EX-CAI (Ours)
2.4
9.0
0.0
0.0
0.0
3.8
Otiginal
71.2
71.6
59.8
15.6
58.5
46.2
Qwen2.5-B 5-BB
CAT
20.6
21.2
17.8
0.0
0.2
15.2
(Ours)
13.6
Original
41.8
17.2
10.6
15.4
21.0
Gemmo-28
18.0
11.2
2.6
EX- (Ours)
16.0
10.4
6.2
1.6
0.0
3.6
Table 2: LC- WinRate on different models. A high LC- WinRate indicates strong model utility.
(Ublit) LC WinRate (%)
Type
Vicuns-78
Mistral-78
Llama-2-78
Llama-3,1-88
Qwen2.5-78
Gomma-28
Original
76.86
90.96
86.70
85.99
91.14
63.96
CAT
36.66
15 35.76
67.51
45.71
77.07
41.75
(Ours)
from AdvBench (Zou et al., 2023). For the utility analysis, we follow Dubois et al. (2024) to use the AlpacaEval dataset for calculating the LC-WinRate utility metrie.
Adversarial training, We use AdamW to train each model via CAT in Eq. (4) or our ER-CAT in Eq. (15), where the embedding space perturbation radius is fixed 0.05. To improve the efficiency of tuning LLMs, LoRA (Hu et al., 2022) is applied to the embedding layer and all query and key projection matrices in attention layers. For the hyperparameter a of CAT, we follow Xhonneux al. (2024) to set it 0.5. For the hyperparameters and / of our ER-CAT, we set them to 0.1 and 0.2, respectively. We also follow Xhonneux et al. (2024) to apply the loss cut-off technique to the objectives of both CAT and ER-CAT avoid over-optimizing, but with less strict thresholds to help the trained LLMs better preserve utility. Please refer to Appendix C.I for omitted details.
Jailbreak attacks. We use six different jailbreak attacks to assess the jailbreak robustness of LLMs. Among them, four attacks are token-level suffix attacks, which are: GCG (Zou et al., 2023), BEAST (Sadasivan et al., 2024), GCQ (Hayase et al., 2024), and Zhu's AutoDAN (Zhu al., 2024). The remaining two attacks are prompt-level attacks, which are: DeepInception (Iii et al., 2023) and PAIR (Chao et al., 2023). Please refer to Appendix C.2 for implementation details.
We evaluate the jailbreak robustness and the utility of trained LLMs. For the robust- ness evaluation, report the Avg@5 Attack Success Rate (Avg@5 ASR) of jailbreak attacks. Specifically, each jailbreak prompt needs to repeatedly attack the targeted model for 5 times. An LLM-based judger from Mazeika or not. The final Avg@5 ASR is averaged on all repeated attack results. For the utility evaluation, we report the AlpacaEval's 's Length-controlled WinRate (LC-WinRate) (Dubois et al., 2024) of targeted models against a reference Davinci003 model, evaluated under the Llama-3-70B- model. An LC- LinnRee of 50% means that the output qualities of the two models are equal, and higher LC-WinRate means that the targeted model is better than the reference model.
Evaluations.
(2024) is used determine whether an attack is succeed
5.2 RESULTS ANALYSIS
Robustness&utility Avg@5 ASR and LC-WinRate on different models are reported in Table and Table 2, respectively. We have two main observations. Firstly, when maintaining the same

---

Published as a conference paper at ICLR 2026
Table 3: Time cost on different AT methods on different models.
Mathod
Time Clost (5)
Vicuns-7B
Mistral-78
Llama-2-78
Liamo-3.1-98
Qwea2 5-7B
Gemman
933.00
904.59 1094.90
801.42 1004.30
335,99
ER CAT (Ouns)
1074.87
JO52. 12
1100.33
456.81
Table 4: LCt (%) and ASRs (%) L.LMs trained from ER-CAT under different embedding regularization coefficient B.
Embesking Regularization coefficien. in Eq, (15)
Model
Thility or ASR
0.2
0.8
I.C-WoRise
$7.19
70.30
68. 76.76
Vicuns-78
GCG BEAST
13.8
12.0
16.4
30,0
17.0
16.2
158
LC WinKuse
74.60
74.52
74,05
05.93
68.67
72.15
Qwen2,5-78
GCG
16.8
17.8
16.8
14.2
14.6
18.8
BEAST
12.0
level of jailbreak robustness, ER-CAT achieves better utility. From the ASR results on Vicuna and Mistral, we find that our ER-CAT beaten by CAT by more than 4% in most of the attack scenarios. However, Vicuna and Mistral trained from ER-CAT achieved a nearly two-times better LC-WinRate than those trained from CAT. Secondly, when maintaining the same level of utility, ER-CAT achieves stronger jailbreak robustness. For Llama-2 and Qwen2.5, ER-CAT reduces LC- A.--iinatte by no more than 3% when compared with that of CAT. However, ER-CAT helps Llama-2 reduce ASR on GCG and BEAST attacks by around 7%, and Qwen2.5 reduce ASR on GCQ by 11 11%. All these suggest that our ER-CAT can achieve a better cabustness-utilijy tradeoff.
Time cost. As explained in Section 4.4, calculating the ambbedding-regllariztioo term (see Eq. (15)) for ER-CAT can be efficiently implemented via native PyTorch functions. Here we empirically justify that ER-CAT does not add significant time overhead when compared with the original CAT. Specifically, we collect and present the time cost of performing CAT and our ER-CAT on different base models in Table 3, from which find that ER-CAT only increases the time cost by 100 200 seconds. This suggests that the relative time overhead of ER-CAT is low.
Ablation studies on the coefficient 9 in ER-CAT. In main experiments, the embedding regular- ization coefficient / in ER-CAT (see Eq. (15)) is fixed to 0.5. We now analyze how this coefficient affects the performance of models trained from ER-CAT. Specifically, within the range [0, 1]. train models via ER-CAT, and calculate their utility (i.e., LC-WinRate) and robustness (i.e., GCG and BEAST). Preliminary results are reported in Table from which we surprisingly find that varying the coefficient 3 does not change the utility or the robustness of the trained model too much. We deduce this is because we use the AdamW optimizer to perform the training. and the gradient normalization procedure in AdamW implicitly performs reweighting on the ER-CAT objective to mitigate the effect of tuning coefficients for different terms.
vary the coefficient
CONCLUSIONS
This paper aims to theoretically explain the mechanism behind CAT for LLMs, i.e., why embedding space adversarial perturbations help LLMs learn to defend against jailbreak prompts from the token space. We first establish new ICL embedding AT problem under ICL theory and show that this problem is good theoretical artifact for approximating real- wal-world LLM CAT. A robust generaliza- tion upper bound for this new ICL embedding AT is then proved, which shows a negative correlation with the embedding space perturbation radius. This clearly explains why CAT achieves empirical success. Our bound also suggests that the jailbreak robustness of an LLM closely related to sin- gular values of its embedding matrix. Thereby, design new ER-CAT approach for LLMs, with the goal of optimizing the LLM embedding matrix to be more robust. Experiments on real-world L.LMs and jailbreak attacks suggest that our ER-CAT enjoys better robustness-utile utility tradeoff.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Di Wang and Shaopeng Fu are supported in part by the funding BAS/I//699-) 01-01, FCC/1/5940-20-055 FCC/1/5940006002, and King Abdullah University of Science and Tech- nology (KAUST) Center of Excellence for Generative AI, under award number 5940 and gift from Google.
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning. Conference on Neural Information Processing Systems, 36:45614-45633 2023.
Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Lin- ear attention is (maybe) all you need (to understand transformer optimization). In International Conference Learning Representations, 2024.
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned LLMs with simple adaptive attacks. In International Conference on Learning Represen- futions, 2025.
Usman Anwar, Johannes von Oswald, Louis Kirsch, David Krueger, and Spencer Frei. Understand- ing in-context learning of linear models in transformers through adversarial lens. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. In Conference on Neural Information Processing Systems, 2024.
Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Badfield-Menell Defending against un- foreseen failure modes with latent adversarial training. arXiv preprint arXiv:2403. 05030, 2024.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, and Chuan Guo. Aligning LLMs to be robust against prompt injection. arXiv preprint arX1v:2410.05451, 2024.
Csaba Dékány, Stefan Balauca, Robin Staab, Dimitar Dimitrov, and Martin Vechev. MixAT: Com- bining continuous and discrete adversarial training for LLMs. arXiv preprint arXi::2505.16947,
2025.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 2 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-comtrolled Al- pacaEval: simple way to debias automatic evaluators. arXiv preprint arXiv:2404.044F 2024.
Spencer Frei and Gal Vardi. Trained transformer classifiers generalize and exhibit benign overfitting in-context. In International Conference on Learning Representations, 2025.
Shaopeng Fu and Di Wang. Theoretical analysis of robust overfitting for wide DNNs: An NTK approach. In International Conference on Learning Representations, 2024.
Shaopeng Fu, Liang Ding, Jingfeng Zhang, and Di Wang. Short-length adversarial training helps Iims defend long-length jailbreak attacks: Theoretical and empirical evidence. arXiv preprint 25Nii:250220422044: 2025.
Shivam Garg, Dimitris Tsipras. Percy S Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Conference on Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2026
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2015.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv preprini arXiv:2407.2178. 2024.
Jonathan Hayase, Ema Borevković, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query-based adversarial prompt generation. In Conference on Neural Information Processing Systems, 2024.
Roger A Horn and Charles R Johnson. Matrix analysis (2nd). Cambridge university press, 2012.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint arXiv:2310.05249, 2023.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 32, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
Haibo Jin, Andy Zhou, Joe D. Menke, and Haohan Wang. Jailbreaking large language models against moderation guardrails via cipher characters. In Conference on Neural Information Pro- cessing Systems, 2024.
Soichiro Kumano, Hiroshi Kera, and Toshihiko Yamasaki. Adversarially pretrained transformers may be universally robust in-context learners. arXiv preprint 2xXi:22555./4422 2025.
Tianle Li, Chenyang Zhang, Xingwu Chen, Yuan Cao, and Difan Zou. On the robustness of trans- formers against context hijacking for linear classification. arXiv preprint arXiv:2500. 2025.
Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. DeepInception: Hypnotize large language model be jailbreaker. arXiv preprint arXiv:2311.03199 2023.
Zeyi Liao and Huan Sun. AmpleGCG: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs. In Conference on Language Modeling, 2024.
Licong Lin, Yu Bai, and Song Mei. Transformers decision makers: Provable in-context reinforce- ment learning via supervised pretraining. In International Conference on Learning Representa- tions, 2024.
Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak LLMs. arXiv preprint arXiv:2410.05295. 2024a.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts aligned large language models. In International Conference on Learning Representa- tions, 2024b.
Yue M Lu, Mary I Letey, Jacob A Zavatone- Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:240s.1175. 2024.
Yue M Lu, Mary Letey, Jacob A Zavatone-Vert Anindita Maiti, and Cengiz Pehlevan. Asymp- totic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences, (22(28)e2500259912 2025.
12

---

Published as a conference paper at ICLR 2026
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Roey Magen, Shuning Shang. Zhiwei Xu, Spencer Frei, Wei Hu, and Gal Vardi. Benign overfitting in single-head attention. arXiv preprint arXiv:2410.07746, 2024.
Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent prov- ably the optimal in-context learner with layer of linear self-attention. In International Con- ference on Learning Representations, 2024.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: Sf efffiettr-efffiient fine-tuning methods. https: /ithhub. htmm/hhagginggaacppeettt 2022.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendryeks. HarmBeneh: A stan- dardized evaluation framework for automated red teaming and robust refusal. arXiv preprint 244Y:22402.04299, 2024.
Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Ad- vPrompter: Fast adaptive adversarial prompting for LLMs. In International Conference Ma- chine Learning, 2025.
Mahdi Sabbaghi, Paul Kassianik, George Pappas, Amin Karbasi, and Hamed Hassani. Adversarial reasoning at jailbreaking time. In International Conference on Machine Learning, 2025.
Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models one GPU minute. In International Conference on Machine Learning, 2024.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts large language models. In Pro- ceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 1671-1688, 2024.
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfeld-Menell, and Stephen Casper. La- tent adversarial training improves robustness to persistent harmful behaviors in LLMs. arXiv preprint arXiv:2407. 155499 2024.
Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? In International Conference on Machine Learning, 2024.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2014.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08299 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv.2307.09288, 2023.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers leam in-context by gradient descent. In International Conference on Machine Learning, pp. 35151-35174, PMLR, 2023.
Sheng- -D Wang, Te-Son Kuo, and Chen-Fa Hsu. Trace bounds the solution of the algebraic matrix riccati and Iyapunov equation. IEEE Transactions on Automatic Control, 31(():654-655 656, 1986.
13

---

Published as a conference paper at ICLR 2026
Yunjuan Wang, Kaibo Zhang, and Raman Arora. Benign overfitting in adversarial training of neural networks. In International Conference on Machine Learning. 2024.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Conference on Neural Information Processing Systems, 2023.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In Interna- tional Conference on Learning Representations, 2024.
Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in LLMs with continuous attacks. In Conference Neural Infor- mation Processing Systems, 2024.
Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Moban Kankanhalli. An LLM can fool itself: A prompt-based adversarial attack. In The Twelfth International Conference on Learning Representations, 2024.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint 2IX1::4122.11155. 2024a.
Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization of trained transformers. In Conference on Neural Information Process- ing Systems, 2024b.
Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust LLM safeguarding via refusal feature adversarial training. In International Conference on Learning Representations, 2025.
Jie Zhang, Meng Ding, Yang Liu, Jue Hong, and Florian Tramèr. Black-box optimization of I1m outputs asking for directions. arXiv preprint arXhv:2510.1679. 2025.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLMeee-a-judge with MT MT-bench and chatbot arena. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu. Joe Barrow, Zichao Wang. Furong Huang, Ani Nenkova, and Tong Sun. AutoDAN: Interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 15045 2023.
14

---

Published as a conference paper at ICLR 2026
A LLMS USAGE IN THIS PAPER
LLMs were used only occasionally to help polish the writing (propose new words, grammar and spelling correction). All technical ideas, experimental designs, analyses, conclusions, writing were developed and carried out entirely by the authors. Authors have full responsibility for the final text.
B PROOFS
This section collects all proofs omitted from the main text. Without loss of generality, we assume that the order of differentiation and integration (or say expectation) interchangeable.
B. TECHNICAL LEMMAS
This section collects technical lemmas that will be repeatedly used in our proofs.
Lemma B.1 (c.f. Lemma D.2 in Zhang et al. (2024)). If I E Id is Gaussian random vector of d dimension, mean zero and covariance matrix A, and A E Iddxd is a fixed matrix. Then
xx
Azx |
MAA A A)A Tr(AA)A
Lemma B.2 (c.f. Lemma A.2 in Fu et al. (2025)). If x E Rdx1 is Gaussian random vector of dimension, mean zero and covariance matrix A, and A E Rdxd is fixed matrix. Then
E|x [r]] Tr(AA).
Lemma B.3 (c.f. Lemma A.3 in Fu et al. (2025)). For any matrices A E Raxm and B mxx
Tr(AB) = Tr(BA).
Lemma B.4 (c.f. Lemma I in Wang al. (1986)). Let A, B.BC E gn be two symmetric matrices and further positive semidefinite. Then
Amin (B) Tr(A) < Tr(AB) (B) Tr(A);
where Amin (B) and Amax B) are the minimal and maximal eigenvalues of B respectively.
Lemma B.5 (Rayleigh Quotient Theorem; Also in part of Theorem 4.2,2 Horn & Johnson (2012)).
Let A paxn be a symmetric matrix. We have
x Ar
Amax (A)
max
max
Az,
ZER" ,r
*ER",]|<
||
x Ax
Amin (A) = min
min
x T
AA.
ZER" ao9h0,, x #ER"
||
B.2 PROOF OF LEMMA
Proof of Lemma 1. Denote that X- = (Ir,1 Fr.N) E Ro Y, (V+,1 Br,N) RIxN and AE = (
5F.N) RdxNN Then, by applying the inequality that la + b12 < 2. (a² + b)), the ICL embedding AT loss ( defined in Eq. (10), can be bounded as follows, cadv
LSAA (0) = |A Se
2 Isged (Zr AT) Vrual
WEX, Yr
WEITA
(WE,, Yr
WEITO
E max
2. 2.(((1))) wk)
WKQ (w (
W Zr.9
Fr.g
2
7 ||A 11 Se
2 |(wxi)
(A. A 0ax1) (WEX, WF 01xN
W
2
| 12. Se
max
N2
wk))
(w 11 KQRT TT)
W. ZT.q
E
||A
max
12. Se
A2
2
((w))
wxk)
(WEx,
Y,
WEBTS)
( A
01xN
0ax1
T
(w
WKQ )
KQ
IT. E.F.
W
15

---

Published as a conference paper at ICLR 2026
E
||A 12,00 Se N2
max
2
((wh))* T
w//2
01xN
AA
0ax1
0
( A
01xN
Odx1
0
(
(ww21
wkk
((Z))E(Z,)T
WKK
1
< 2- E ((w2))
w//2
WW IT..-
N
(w21 KQ
TT)
Vn
2
(WRX: X
wKQ
+ E
| Sr
m18
N2
(ww21) T. A
Y
(w (m)r))T)
-W IT.
:A((()
2
((wx)) T w/22
(WEx,)
(( wHO WB IT.
+
E
| S0 N2
max
Y,
A2(9)
+ E
| 22.0
max
Se
N2
2
(ww)))
(T W.K W IT
(B.1)
Aaa(()
For (0) in Eq. (B.1), we have
2
(wu)) A. (WEX,) (MEx..)²
w.k
(0) :- E
max
Se
N2
Y
(wel
)T
(T)
w E IT,1
<
E
| 12 Se
max
N2
2 [1(w21)
(((W)))AF1133 1 11
(WEX;)
Y
(w KQ
w.Q
TT)
W 2
2
NN 11 -11 (WFxx²) T
wKQ (wx21) T
E N2
W Z
Yr
2ea
T (WEX;)
WKQ (w )T
|
12
E ||
W a
(B.2)
N
Y
For A2()) in Eq. (B.1), we have
A2 = E
T
IA
maa
Se N2
2
(((ww))
w//)
(WEX)) (A,.) WKO.
Y
W IT.
< E
||ا."
max
112.00
N²
2
(
w/2)
[WEx,
Y,
11:))(
TWCKWWE F 12
WFX,)
$ R N2
|| ((wW))
T w/22
11 Neee NNeellw k R x 12
Y,
2xa
[WEx,)
E
||
((w))
w
112]
E BIllw
ke W B xr, 113
(B.3)
N
Y=
For Ao in Eq. (B.1), we have
2
As(O)= := EE max
(mb21
V
(A W W WFX
Se N
2
< E
|| 02 Se Nº
max
5 T W W x
2
max
(1
max 5
W KQ 11
W R
x 11
N2
Sc
2
N V 11
Ne
KQ
W xr.q 11
N2
2c
E
M13 E [I 111
(B.4)
16

---

Published as a conference paper at ICLR 2026
Inserting Eqs.(B.2), (B.3) and (B.4) into Eq.(B.I, eventually have that
(iSAE(() (0)
E(Z,)
w.k
)
S I
(ww)))
w
(w (
W.
W
IT.
yrr
N
2
(
(w221 WKO
1 12
B
W
N
Y
2ea
1 ((w))T
w/22)
W X
E |
@ 20 xr
N
YG
2. E B1(
E
(LSAE(0))
Eadv
which completes the proof.
B.3 PROOF OF THEOREM 1
The proof idea of Theorem 1 similar to that Zhang et al. (2024) and Fu al. (2025). Specifically, we first show that when training the LSA-E model fīS (0) via continuous gradient flow, w and w/ stay zero during the surrogate ICL embedding AT (Lemma B.6), which can help us further simplify the surrogate objective function Eady (0) (Lemma B.7). The global minimizer for the [[
surrogate ICL embedding AT problem is then derived based on the simplified surrogate objective function (Lemma B.8).
We start by stating and proving Lemma B.6.
Lemma B.6. Suppose Assumption 1 holds and the LSAE model fisxp(() is trained via minimizing the surrogate objective function LLSAE continuous training time 2 we uniformly have that W21 (1) w (1) 0dx1-
Cady (0) in Eq. (13) with continuous gradient flow. Then, for any KQ
Proof. Under Assumption 1, we already have that w time / O. As result, prove Lemma B.6, we only need further show that gradients for parameters w21 (t) and w (t). which are given by continuous gradient flows, stay zero during the overall surrogate ICL embedding AT. Formally, we need to prove that for any 2 0,
KQ (0) = w (0) = 0ax1 at the initial training
KQ (t) = - xx w 21
Eady (0) = 01xd.
d v (t) == -dny ())) = 01xd-
To this end, we adopt the decomposition of Zady F (0) in Eq. (13) here as follows,
čadv (f) == (0) + (0) + (0) + La(0),
where
((Z,)
WKQ Re ) (w221
WE xT,9 Vr 2
(9) = 2:
((w))
2e
WEX;) Y,
WKk TT
(2( N
V
112
E
W S 1
KQ)T (w WEX, Y-
2.2
(s(0) = N
E
||
((wX)))
wxr)
[12]
E
|I
xr. xr. 11
(a(0) = 2 E (w))
B BiIw W 2r.911.
and X, == (x+,1
T,.N) E RdxN Y, = (y+.1
Vr.N) E RIXN Then, we will
KQ show that when u121
w O one always has O 1n
( = 0 (,( = 01xd for every
i E [4], which thus automatically demonstrates that 0 x, Lady (0) = o,y Ladv ( = 01xa for
any continuous training time / under Assumption 1.
17

---

Published as a conference paper at ICLR 2026
KQ why Oax1 indicates 0 ( Step 1: Show that W21
Duy ( 01xd- For the i-th
22
entry 221, of W21
KQ
KQ, have that
kg (0))
21,
w =
Oax1
((Z))
w (w22 (wx)))
E
(((w))))
w//2)
W IT. - yT,9) WBTTg
N
((Z))
(Oaxd 6q W
((w2))
w/
W Lr. wis2
N
-
mwyy -0x1
w ((Z)) N
4.
(0ixd
W. ITT Vr.a)
O Olxd
(01xd w//2) ((2))
(Oaxd E
W WT.0
N
4 E
W22 V
(Y+ 0) (WEX, WF
-W-K
w
Wh 2r,4 yr
N
122 (Y, 0) (Y, eeT w -WETTa))
N
4
E
-. V
w. X, xw) w.K W 2T, wy ITa
N
Waz w X, wy ee W ):
(B.5)
where E dx denotes elementary vector that its i-th entry is 1 and all remaining entries are 0. We then have two observations for Eq. (B.5): (1) for the first multiplication term in Eq. (B.5), each of its summarization term contains exactly one element from the task parameter wy and (2) for the second multiplication term in Eq. (B.5), each of its summarization term contains exactly two elements from w.. Based these observations and the independency of w, with respect to X, and Fr.g, Eq. (B.5) can further be re-organized as follows,
0 ( 21
E Bx., 0]] E W Why wrr] (B.6)
where Er (X,, 0)] the coefficient for the term Er [wr,A - Wri wra] and depends only on X,, Zn.g and O. Recall that wi ~ NO, ld),, which means E, [, W wr,] = 0 holds for any k. E [d]. Combine this result with Eq. (B.6), we thus have
i (
= =0, Vi [d].
22.11
=
which indicates 0 (()
01xd
xg = 0ax1 1/21
Besides, for the i-th element whi of w. similarly have that
0ny (()
KR K0-=0 =wy =0xx
w/k ((Z)) N
wKke
= =4.F E (((w))))
(n: (
W
xiiigggg Vrig))
0) E(Zr)E(Z,)
WKQ (w ((e))
(eT
W. ITMR) (WBITT))
N
= 44 R
w T X, (x w.k. WW IT. - w. -ZT.9)
N
1
T. (WEX, WBxro) (WEX, WFXrg) T. T.Kk wBxTg)) (B.7)
N
18

---

Published as a conference paper at ICLR 2026
From Eq. (B.7) we notice that: (1) each summarization term in the first multiplication term of Eq. (B.7) contains exactly one element from w,, and (2) the second multiplication term of Eq. (B.7) does not contain any element from w, Thus, Eq. (B.7) can be re-organized as below.
8 (1()) Wh1,
Σ (x 2..
03 Elur,.)
(B.8)
KQ M21
=
Dax1
where Er (X; -T 0)] is the coefficient for the term E- (w and depends only on X,, Zryg and 0. As a result, by again applying w/ ~ NO0, Ido), we have E,, [wraal 0 for any l [d]. which means
Ouyy (0)
0 Vi E [d],
KO may, =0x1
2121
and thus indicates Owy (0)
01xd
RR =
=0ax
Step 2: Show that
KQ 22
=
wa1
Oax1 indicates 0 (2) = 0uy (2( = 01xd= For
2221
kg (0), have 21
0 Kg (0) x0 =uy;=
1221
u21
=0ax
2c N
12 a
221
Kg
El
(WEx)
Yr
WKQ (wi )T)
WF TT (1)l
x0 =wy =0ax
2ca
W
)
T
W-Q (wx
110dxi112
2
xe
7
Ell
Yr
W xr.
w/1 =0axx
01xd-
For 2n (2(),,
we have
26
(Exx)) Y
w.QQ (w (1ntt)) )
Ouy (2)
2 (Wx))
Ell
W xTTel1
w21
K2 =u:==wy--0 =0ax1
N
w
Kg = w
=lax
22
(WEx))
WKQ (w KQ)T
2 (0ax
T. Ell
W 2T
Yr
KQ 0sx1
01xd.
Step 3: Show that W21
KQ w 0ax1 indicates a (0) day, Es(
=01xd- For KQ
1221
21
since it does not exist in (0), thus we always have θ kg (0) = 0.
Besides, for the i-th element whi of whi we have that
0 21,
(0)
KQ =u =0x11
26a
E
2 ((wX) T
w//2
WWXx) Yr
(WEX,)
((G)T o) Y,
E [lI KQ KOKEEEE E
N
will - =0a1
2e
(WEX))
(WEX:) T ((e)T l) E | Q x 2
B 2. ((0ax)) T
wx2
N
V-
Y,
2ea
E
V Y, (7-WEX,) X,)
es E ||w KW
W |
x. [2]
N
2
E
Wb2
w. WEE, (WEX,) c.] E w 11
w W 11
2
v
Elw=1
E xxx ( xx," T. [c]
k
12
11
N
4e
w
0xxd
L8 W -W"x,) .e] Ww
=0
N
19

---

Published as a conference paper at ICLR 2026
where e, E R is elementary vector that its i-th entry is 1 and all other remaining entries 0. As a result, we thus have
(a( 12 = 0ax1
01xd-
KQ w/11 0ax1 indicates O k (1)) Dwy, La(0) -1xxd For W221 Step 4: Show that W21
KQ
since it does not exists in (0), thus we always have i Kg (z( = O.
12
Besides, for ouy, (0)), we have
Duy, (0) K2 = =0x1
2e [2 (w21) " Twy =0x11 TR
[[[IWKow' W x
-2e 12. (0ax1) 1- |W
|Wk. W 21.1
01xd
The proof is completed.
Based on Lemma B.6, we then simplify the objective function LSAE (0) in the surrogate ICL em- bedding AT in Eq. (13), as shown in the following Lemma B.7.
Lemma B.7. Under Assumption 1, the surrogate ICL embedding AT objective function Eadv
LSAE (0)
defined in Eq. (13) can be simplified as
LSAE (0) = 2(wy2) 22 Tr ( + Tr(A)e
Cady
-4 Tr (W (WWE) + +TT((A),
where IN = ( \ (r(A)a&,)) E do
KQ and why become zero vectors Proof. When Assumption holds, by applying Lemma B.6, w21 during the surrogate AT. Then, (2( and (0) in the surrogate AT loss (S) will stay zero
Endv
Besides, for ei (0) in Cady LSSA
(0) in Eq. (13), it becomes
(0) 2- E ((0ixd w// E(Z,) (ZZ,)
(W.K))
1 Vra 12
N
Oixd
1
w/y (Yr 0) (WEX, W T. WKO W. xT. Vtgg1
= -2 7
N
2(wW22²
E
(WWB)
WKWW X,Y.T Y,X, ( T ( (W
N²
4w
E.
Y,X.T
TW Pa Fr. +2.E
N
2(w )2
((WB)TTKKW X,X, Ei X, (WWE) w NykoWBBa E)
E
x 7.
N²
4w
KQ w (I
)I ()) T. wr
+2 +2 Elw, Ira wr]
N
22w 2.
Tr
E N2 x
X.N, (WWB) T WKOWWB A)
Lemma
1 E[X=X,]| ((WE) (ARR)T(Tie W E( T +2Tr(A)
-wwbe N
Tr
Lemnna B.2
((wB))2T11 (T:(((WE))))) (WWTTTKKWWETTT N2
Σ
B Bla B ((WE)
WWKWWE)A
20

---

Published as a conference paper at ICLR 2026
-4wy Tr A ([WF)TTR
A + 2Tr(A)
=2(w22) 2 ( N² N (2A² (T(A))) +(N2 -(N2-N)A)) ((WE)TWHH w .
1
Lemana
-4w'2 Tr A ( A| + 2Tr((A
=2(wl)) Tr ( TW .NA TROTTTRRWWW A
- -wwy2 Tr A (IWWETTRKRN A 2Tr(A),
(B.9)
where IN = ( (N N 2) E Rdoxd
For (z( in Cadv LSAE (0) in Eq. (13), we have
(0) = 26a
E (0xxa
WEXx) wxx2
12 LE W x
N
Y
2c 2x2w22)) )
Y l2 E Trg (W
W. W Ir.q
E
N
= 222
B l
W w Fr,il
Tr (w w WEK
N
Lemma B.2
222 Tr(A) Tr (W (WKKWD)
(B.10)
Finally, by inserting Eqs. (B.9) and (B.10) back into the surrogate AT loss Zady LSAB( (0) Eq. (13) and repeatedly applying Lemma B.3 and the commutativity between A and I'N, we thus have
Cady (SA 2(w 2. Ir (WWW)TWKKWWE -'A (OWE)TTK&W'S A)
4w1 Ty A (WW)TW**WBB A| + 2Tr(A)
+ 22 Tr(A): Ti (W T W
= -2 Tr TTWW (W wKee WF Ab). (W AAb)
4w0% Tr A ( + +TTr(A)
+ (22 ( (F - (WKOWEEA)]
2(wx)22 Tr (W IT(A)e -w (w,k F -
4wh/2 (W - A + +2T((A).
(B.11)
The proof is completed.
Based on the simplified surrogate AT loss, we now calculate the global solution for the surrogate embedding ICL AT problem in the following Lemma B.8.
Lemma B.S. Suppose Assumption holds. Then, 0, = (WE, E W.KQ WV) is global minimizer for the surrogate embedding ICL AT problem defined in Eq. (13) if and only if
(W..) (T (W.T)((W + T1
21

---

Published as a conference paper at ICLR 2026
Proof. Under Assumption by applying Lemma B.3 and Lemma B.7, can re-organize the surrogate AT loss LESAE (0) in Eq. (13) as follows,
CLSAE(
=2(w22)² 2 Tr (WETNA(WE) + Ir(A)c (WHRW -
4w0/2 TH(WKKOWEN A (A(WE) +2Tr(A)
2- Ir
(WET'NAWAE) + (T(A))
+2TH(A)
-4. + + (w - (. (W"T)((""TAAW" + TT(A))2T) 7
2 Ir (WETNA(WE) +TM(AEE
WWWW F (W"TNA(WT) +T(AA) W=*a)
WWWKKWWEN (WETNA(WET) +T(A)) -1 ( ) Tr - + TT(A) TWE (WE)] +TT((A).
(B.12)
Note that the second and third summation terms in Eq. (B.12) are constants. Besides, the first term Eq. (B.12) is non-negative and can achieve zero via setting
w. V W ByKQQO 2,1 W. A (WA.T TT =0,
which is
22 T -V -
The proof is completed.
B.4 PROOF THEOREM 2
We first prove useful Lemma B.9 that will be frequently used in this section.
Lemma B.9, For the inverse matrix FWFTx + Ia e paxd in the optimal
surrogate ICL embedding AT solution in Theorem 1, we have
(1 (W.TTNA(W55] + TT(A))
|1
J (T' Jmin (WE ( Tr(A)
Proof. According to the definition of matrix operator norm (i.e., || (12)), we have
[W.TTA(WW3) + TT(A))2Ta)
"11
Tmax (W.5T Tr(A))2Ta) -1
1
(B.13)
Owin W.T Tr(A)e21 Omin W.TNAAWFE)T + Tr(A)e2
Notice that W.FTNA(WE T is positive semidefinite matrix, we thus have
min W x(W:5) = Amia W.F W.TTx~(WW:)).
(B.14)
By further applying Lemma B.5.
Amin WW.TwA(W:5)
min || v T...W
(B.15)
22

---

Published as a conference paper at ICLR 2026
Denote that (W.))oo o R", then Eq. (B.15) can be re- re-written
Amin [
min "ER", 1
J.W.INA(WBB)
min
u (TNA)u
a (TwA)u u u
min
VER vella11
uTu w/ (T'
min w'ER",Iw']
min
u Tu
uT
NER 1....110 1|101211
min Amin (Tx
u² W. "W."
VER-_110|1==1
Amin (A)
Amin (W. )
min (I A)
min (W. (W.
miin (I's A) min (??
(B.16)
where both (+1) and (*2) obtained via again applying Lemma B.5, and (*3) due the fact that both I' A and W RW are positive semidefinite matrices. Combining Eqs. (B.13), (B.14), and (B.16), we finally have that
Th (W.TTxA(WW)) I= "
mmiu (A' T ( (r
which completes the proof.
With the help of Lemma B.9, we can now start to prove Theorem 2.
Proof of Theorem 2. For the converged LSA-E model fLSAE,0~() trained from the surrogate ICL embedding AT, by inserting its prediction function Uq.0- given in Eq. (14) into the robust risk P.M (-) defined in Eq. (12) and using the inequality |a b/2 2(a² b2), we have that
R. Mr (6.)
YX,, + (Odox(N-M)) 1G1) (W.5) (W.Txx((W5) +T(A)el))) " "WFAA ar Vr.n
E
7 |A
112.00 Sp
2 N
(+(X)) (W.5) (W.TNA(WW?)T + TT(A)(1))
".F.A Ir.g
N2
E [(xx(X)) (T.B) (W.T.WA(WE)] + Tr(A))²1s)
W.E.A Fr.g Vr.q
Ellu²
N
=C2
:=
A Sp N Y Y ( ( T (W.TWA(WE) + +T(AA)(1s)
"W. A Tr.g
E max
(B.17)
where Y=(N-M+1) (Vr.N-M11
Br.N) E RRIM
23

---

Published as a conference paper at ICLR 2026
For 1j in Eq. (B.17), we have
C = N E (X,)) (W:)) (W.T)(WWT + TT(A))²Ta) W.E. A IT 2
1
1
E 2 (W.A) (W.TNA(WE)) (T((A))(aa) W. B|Xx, X2 wo - X, (X,)
N2
(W.") (W.TXA(W:5) + TTA) W. 27g
E TT (WPA) (W.TTA(W.5) +TMA)e2la) W.
N2
(W +
"W.". 2T.9
".. (W.A)) (WEA)T(W.TTAAA (W.TTA(WW5) +T(A)))Ta) ".e N22
1
N(2A2 Tr(AAA)+(( +(N2 +(N0)+++NN+))0 -NJA3)
Lemma
(W:)) (W.THA(W.") +TM()) W.F.A 2T.9
E TT. (W. (W.TTA(W.F) + (T(A))2
W. INA
-WW (W.TNA(W.") + LT "W.AA Zn.9
= TT (W
[WTTWA(W5) + (T(A))))) "W. TNA (WF)T (W.TTA(W:5) + +Tr(A)e1a) + -WFA-A]
Lemma
Then, by repeatedly applying Lemma B.3 and Lemma B.4, we further have
o Tr (W (W.TTA(W55) Tr(A)=+aa w. -NA (W.)T (W.))(W (W.TTA(W55) "V WW,A3]
( (T
W.F. -NA. (W.5) (W.TNA(W?5) + +T(A)ETA) W.F. 13)
(2)
< Tr I[((W5) - W.TTA(WW5) (W.TNA(WE)) + (TAA)LL)) WWE] Amax (A3)
Tr TT( (W.TNA(WW.) + (W (W.TWA(W.") + "W.TNA]
Amax (A)
(W.TTA(WW?) + TTAA))TT))
"W.(W::5 (W:TT<A(WWW) - -w!e]
+THA))-Ta)
Amax (TNA). (A)3
Tr Tr(W:& B +
TT(A))?1a) -1 " Amax (W.TTxAWWE) + TT(A) "1
Amaax (TN) AA)x
(2)) Tr ( ( + Tr(A)(21a) -1
Amaxe (T (N): -max AA)
(W.FN
Omin (T (TxA) Omin (WF)² + Tr Tr(A)²
(37) W. (W TWE
Amax (T') Amax (A))
T (Tx amin (W.))² + Tr(A)e?
mmxx (T Df (WBB)
(B.18)
Jomin (omin(TTX) miin (WF)2 + Tr(A)e2]2
where (+1) and (*a) is due to Lemma B.3, (*2). (*u) is due to Lemma B.4, (#5) is by Lemma B.3 and Lemma B.4, (*6) is by Lemma B.3 and Lemma B.9, and (+7) is by Lemma B.4 and Lemma B.9.
24

---

Published as a conference paper at ICLR 2026
For C2 in Eq. (B.17), have
C2 = N E (x+)) (W:5) (V.T)T...TAWW.. + IT(A)e
W. A 2r.9 Vr.a
N B : (x,))" (W.3) (W.TT)(WWWWWWW + TT(A))21a)
W.F.A A Itrig Fig WT
2 Tr E|X, 11 (WW.)) (W.TTA(WW)))) + +T(A)E
"W.A A El
N Tr NA. (W.)² (W.TNA(W99) + Tr(A))TAa) W.A.-A]
2
2 Ir A -wW (W.TRA(W!5) + TTA))=
W. A 2
(B.19)
where (*) is by Lemma B.2 and (**) is by Lemma B.3.
For C3 in Eq. (B.17), we have
C3 = BB = B|w 27 " = Tr(B/or.07..) = Tr(A).
(B.20)
For C4 in Eq. (B.17), we have
1 Y Y=IM-NHHINN(PP) (W.5) (W.TTA(W!5) TT(A) W.F.A. A IT 2
C4
=
max |
la Sp N
1
(A:) -WW.) ( IMA))e W.F. Ir.
2
max El - "110 11.0005
N
M
Tr(A) p² : (W.TTNAWWW)) +T(A))=a)) TWFA and 1
N
(M)TN)))/
TT(A(WE) T
(W5TWA(W55) + TTAA)21))
TT(A))Ta)
-1 "WFA. A. A).
where (+) follows Lemma B.2. By using a similar derivation as that for Eq. (B.18), we thus have
M Mr) C4
(A)3 Ei (WWP)
(B.21)
N2
[omin [amin (A mmin (W.5)3 2T
Finally, inserting Eqs. (B.18), (B. (B.20), and (B.21) into Eq. (B.17), we thus have
Raddy (0+)
Tmax (TNA2). E (:WB) Mnin (A Omin Imin (WEY² Tr(A)e2J2 0+ Tr(A) + Mp²Tr(A)
m (A)3 Edd o2 (W
N2
[Tmin x mmiu (W.52 Tr(A)c?
Tmaax (T NA) + MP²TH(A) N=
mmxx (A)3, Ed (W.5)
+ Tr(A)
(amin (T AA) mmin (WB)2 + 22
0 (1 + Mp² N2 Ed ),(W))))) O())
mmin (WB)s 4el
The proof is completed.
C ADDITIONAL EXPERIMENTAL DETAILS
This section collects experimental details omitted from Section 5.
25

---

Published as a conference paper at ICLR 2026
Table 5: LC-WinRate on models trained via CAT with different toward/away cut-off thresholds.
Type (Toward/ Away Cut-offi)
(Uulity) LC-WinRate (
Vicuna-7B
Mistral-RR
1amma-2-78
1lama-3.1-88
Qwen2.5.7B
Gemma-2BB
Original CAT (1.0/-3) -3.0) CAT 10.5/-500
76.86
90.96
86.70
85,99
91.14
63.96
36,66
15.76
67.51
45.71
77.07
41.75
23.60
12.12
67.65
52.46
71.11
38.91
C.I ADVERSARIAL TRAINING
Searching embedding space adversarial perturbation. We leverage the projected gradient de- scent (PGD) (Madry et al., 2018) to solve the following embedding space adversarial perturbation searching problem for any harmful input-output pair (z, v),
8
arg max log po (y(EE()) + +6]
1| lla. 11
Specifically, PGD will first initialize a starting perturbation as 5!0) :- Odx|z and then iteratively update it for times. In the k-th iteration, the update as follows,
0s, loggo(y)E(()) + 1) 1(
((() = II
kkk-1 +1 18s, log (o (y)E(() + 8 1))12
Vi (1, (k,),
11
where (A)) is the intermediate adversarial perturbation found in the k-th iteration, 0% is the pertur- bation for the i-th token embedding. 7) > 0 is the step size, and II | means that the perturbation for the i-th input token embedding each token embedding projected in ball sphere centered Oax1- The eventual adversarial perturbation is 8* : =((K)
In all experiments, the perturbation radius c is set as 0.05, the update number K set as 10, and the learning rate is set as 1 x 10-2
Loss cut-off technique in LLM AT. The adversarial loss in CAT and ER-CAT can be decomposed into a toward loss and away loss follows,
log po + +*) + log log + +5)
Toward
Away Loss
Then, the original CAT paper (Xhonneux al., 2024) suggest "cut-off" each loss function L' as
C = [[C'] 0.999c [I[C' > c] 0.001 + c
To prevent over-optimizing both the toward and away losses, we use cut-off parameter e and the indicator function I[-]. Xhonneux et al. (2024) originally set the cut-off value 0.5 for the toward loss and -5.0 for the away loss. However, we have empirically found that such hyperparameter choice significantly reduces the utility of trained LLMs. Specifically, we calculate the LC-WinRate of model trained from CAT with toward cut-off value of 0.5 and an away cut-off value of -5.0 in Table From the table, we find that, for example, the original cut-off setting from Xhonneux al. (2024) can result in the LC-WinRate of Vicuna being 23.600 23.60% which is significantly lower when compared with its original LC- WinRate of 76.86% before finetuning.
As result, in our experiments, we relax the cut-off values to 1.0 for the toward loss and -3.0 for the away loss in both CAT and ER-CAT to help trained LLMs better preserve their utility.
LoRA setting. We use the PEFT library (Mangrulkar et al., 2022) to apply LoRA (Hu et al., 2022) to the embedding layer and all query and key projection matrices attention layers of LLMs. For the embedding layer, 1ora.dropout=0.. 1. Besides, for the remaining layers, we set their PEFT hyperparameters as r=64, lora_ooraaaa wow..a.paaaa...2.. alpha=32, and lora parra_droouuttta
set its PEFT hyperparmeters as r=1024, Iora-alpha=32, and
Adversarial training. We use AdamW to train each model via CAT in Eq. (4) or our ER-CAT in Eq. (15), where the embedding space perturbation radius is fixed 0.05. To improve the efficiency
26

---

Published as a conference paper at ICLR 2026
of tuning LLMs, LoRA (Hu al., 2022) is applied to the embedding layer and all query and key projection matrices in attention layers.
Other AT pyprraarametess. In every AT experiment, we perform LLM AT with AdamW for 60 iterations, where the learning rate is fixed to 2 x 10 The batch size is set as 64, where samples "Sorry,
adversarial inputs and the remaining 56 samples can't t do that." " as the safe response for harmful inputs. For the hyperparameter a of CAT, follow Xhonneux al. (2024) to set 0.5. For the hyperparameters a and / of our ER-CAT, we set them to 0.1 and 0.5, respectively.
utility inputs. We always
C.2 JAILBREAK ATTACKS
We adopt six jailbreak attacks to assess the jailbreak robustness of LLMs. Four of them are token-level suffix attacks, which are: GCG (Zou et al., 2023), BEAST (Sadasivan et al., 2024), GCQ (Hayase et al., 2024), and Zhu's AutoDAN (Zhu et al., 2024). The remaining two are prompt- level attacks, which are: DeepInception (Li et al., 2023) and PAIR (Chao al., 2023). We implemented all six attacks by ourselves to enable efficient and fair jailbreak evaluations. Addi- tionally, for every suffix attack, the length of adversairal suffix token length is set 20. Other hyperparameters of jailbreak attacks are set as follows:
GCG: According to Algorithm I in Zou et al. (2023), hyperparameters that we need to set for GCG include the iteration number T, the top-k parameter k and the "batch-size" B. We set as 500, as 256, and T as 64.
BEAST: According to Algorithm I in Sadasivan et al. (2024), hyperparameters that we need to set for BEAST are two beam-search parameters ki and k2. We set ką as 64 and k2 as 16.
GCQ: According to Algorithm I in Hayase et al. (2024), hyperparameters that we need to set for GCQ include the iteration number T, the proxy batch size bp. the query batch size by. and the buffer size B. We set T = 200 and by = by = B = 128.
Zhu's AutoDAN: According to Algorithm and Algorithm in Zhu et al. (2024), hyper- parameters that we need to set for Zhu's AutoDAN are the iteration number T in each step, objective weights wi and W2, the top- parameter B, and the temperature т. We set as 3, W1 as 10, W2 as 100, B 256, and as 2.
DeepInception: According to Li et al. (2023), DeepInception attack leverages human- crafted jailbreak prompts to perform attacks. Therefore, no hyperparameter need to he set for DeepInception. Additionally, we use the role play-based prompt from Li et al. (2023) to conduct the attck.
PAIR: According to Chao et al. (2023), PAIR leverages an LLM based attacker and an LLM-based judger to iteratively synthesize, evaluate, and refine jailbreak prompts. We use Miistrl-8877B-nnstrutt-a I as the base model for attacker and Llama-3-70B-Inss base model for judger. Besides, the number of iteratively refining is fix to 10.
as the
C.3 ADDITIONAL RESULTS
This section collects experimental results that omitted from Section 5.
Evolutions of singular values of W As illustrated in Section 4.4, our proposed ER-CAT aims to regularize the embedding matrix of trained LLMs so that its singular values be neither too large nor too small. Here we empirically show that the introduced regularization term in ER-CAT can indeed help to achieve such a training goal. Specifically, we plot the maximum singular value, the minimum singular value, the standard deviation of the singular values, and the mean of the singular values of the embedding matrix over the training progress of CAT and ER-CAT in Figure 1. From the figure, we find that when compared with the original CAT method, our ER-CAT can optimize the LLM embedding matrix to: (I) reduce its maximum singular value, (2) increase its minimum singular value, (3) reduce the standard deviation of all its singular values, and (4) do not change the mean of singular values too much (the change of mean is less than 2% every base model). In other words, the newly introduced regularization term in ER-CAT can make these singular values more concentrated. However, we also notice that the reduction in singular values" standard deviation
27

---

Published as a conference paper at ICLR 2026
CNT
GR-CAl 10.10
(a) Maximum singular value of the emhedding matrix W E of LLM along AT.
CAT
1400
(b) Minimum singular value of the embedding matrix LLM along AT.
Viera-70
Marsa79
Liara279
Liana-3 340
Cerra:20
Rex
Sowe
(c) Standard derivation singular values of the embedding matrix of LLM along AT.
Vicano79
Mirra-70 7D
Usmre
Ua-aa-1-00
Qwint
Gemma-20
Thaiing Drep
Triming Siyp
Thanig Diis
Tranmng thrp
(d) Mean of singular values of the embedding matrix of LLM along AT.
Figure I: Evolutions of singular values of the embedding matrix of LLMs along AT.
Table 6: 1 @@5 ASR on different models and attacks. A low ASR indicates a model robustness.
GCG
BEAST
GCQ
7hi's AutoDAN
DeepInception
PAIR
Original
26.0
77.0
Vicuna-7B
CAT
31.0
31.0
14.0
2.0
22.0
14.0
ER-CAT (Ours)
34.0
5.0
25.0
30.0
Original
94.0
89.0
92.0
64.0
84.0
73.0
Mistral-7B
CAT
25.0
17.0
14.0
4.0
4.0
33.9
ER-CAT (Ours)
20.0
18.0
8.0
2.0
3.0
40.0
Original
61.0
30.0
9.0
11.0
70.0
44.0
Llama 2-7B
CAT
45.0
30.0
15.0
8.0
24.0
25.0
ER-CAT (Ours)
32.0
19.0
5.0
1.0
8.0
12.9
Original
75.0
1aama-3.1-5
CAT
9.0
10.0
0.0
0.9
0.9
12.0
ER-CAT (Ours)
17.0
0.0
0.0
9.0
Original
86.0
86.0
82.0
26.0
90.0
63.0
Qwen2.5-78
CAT
41.0
43.0
40.0
0.0
1.0
26.0
ER-CAT (Ours)
30,0
31.0
14.0
0.0
4.0
21.0
Original
79.0
68.0
21.0
8.0
45,0
34,0
Gemma. 2B
CAT
40.0
27.0
8.0
1.0
1.0
10.0
ER-CAT (Ours)
40.0
27.0
15.0
3.0
0.0
10.0
might not be large enough, which means that there is still room for improving the effectiveness of concentrating embedding matrix singular values.
"Worst-case" (1@5 ASR) robustness analysis. In Section 5, we leverage the metric Avg@5 ASR to evaluate the jailbreak robustness of LLMs. Here we focus more challenging setting assess LLMs robustness via the I @@5 ASR metric. Specifically, under the @@5 ASR metric. each jailbreak prompt needs to be repeatedly fed into the targeted model 5 times. If any of the 5 responses is judged
28

---

Published as a conference paper at ICLR 2026
CAT
ER CAT (Ours)
CAT (ARI55ASR) ER-CAT (Avg ASR)
GAT (1@5AAR)
LC-WinRate
ER-CAT (1685 ASR)
GOG
Atlack Suncess Rate (4SE) (9Q
BEAST
LC LC-WiRaain (99
Suncess Rate (ASR)
80
80
ka
20
20
0.000
Embedding Perturbation 0.100
0,060
9 0.025
3,000
a.050
0.075
0.100
9,000
0.005
0.050
0,075 0.100
Embedding Perturbation Radius
Embedding Per Perturbation Radius
LC- IC-WinRate on Vicuna-7B.
(b) ASR Vicuna-7B.
Figure 2: Utiliyy (measured by LC-WinRate) and jailbreak robustness (measured by ASR) on Vicunna-7B trained with different embedding perturbation radius c within the range [0. 0.1.. A high LC-WinRate indicates a better utility, while low ASR indicates strong jailbreak robustness.
CAT
ER CAT (Ours)
CAT (ARI@5ASR)
CAT (S@5AAR)
LC-WinRate
ER-CAT (Avg ASR)
ER-CAT (165 ASR)
GCG
BEAST
LC-WinRate (96)
Amtck Suncess Rate (ASR)
Amaak Suncess Rate (ASR)
18g
80
60
80
k0
40
20
0.000 0.025 0.000 0.0 org
0.100
0.000
0,025
0.060
0,075 0.100
Embedding Perturbation
Embedding Perturbation
Embodding Perturbation Rodius
(a) LC- WinRate Qwen2.5-7B.
(b) ASR Qwen2.5-7B.
Figure 3: Utility (measured by LC-WinRate) and jailbreak robustness (measured by ASR) on Qwen2.5-7B trained with different embedding perturbation radius within the range [0, 0.1]. A high LC- indicates a better utility, while low ASR indicates strong jailbreak robustness.
as a harmful response, then this jailbreak prompt is considered to have successfully jailbroken the targeted model. Results of @5 ASR collected and presented in Table 6. From the table, we have similar observations as those for Avg ASR (see Figure in Section 5). i.e., ER-CAT achieves significantly better jailbreak robustness Llama-2 and Qwen2.5 models, while maintaining similar robustness on Vicuna, Mistral, Llama-3. and Gemma models. is also worth noting that while ER- CAT and CAT achieve similar robustness on Vicuna and Mistral, according Table in Section 5, the utility achieved by ER-CAT is significantly better than that by CAT.
Ablation studies the embedding space perturbation radius in AT. In main exper- iments, the embedding space perturbation is fixed to 0.05 for both CAT and our ER-CAT. We now analyze how different radii affect the performance of adversarially trained mod- els. Specifically, we train LLMs with different embedding perturbation radii from the set (0, 0.02, 0.04, 0.05, 0. 0.06, 0.08, 0.1) and then calculate their utility (i.e., LC-WinRate) and robust- ness (i.e,, GCG and BEAST). Preliminary results are shown in Figure 2 for Vicuna-7B and Figure for Qwen2.5-7B. From Figure we observe that as the radius increases, ER-CAT maintains sim- ilar jailbreak robustness that of CAT, but the utility of CAT models degenerates rapidly. Besides, from Figure 3 we find that CAT is very sensitive to the change of radius e while ER-CAT less sensitive. We also observe that when the radius is large (i.e., 0.02), the jailbreak robustness of ER-CAT is better than that of CAT in most cases.
29


Reviews:
[{"review_id": "hFvYafRBWI", "reviewer": "Reviewer_8RQd", "summary": "This paper takes a step toward understanding why adversarial training (AT) in the embedding space provides robustness to attacks in the token space. To enable analysis, they focus on linear self-attention with embedding module (LSA-E) trained on linear regression ICL tasks and prove that embedding-space AT on LSA-E yields an upper bound on the robust generalization risk of LSA-E models under ICL suffix adversarial attacks, an attack in the token space. \n\nThey show that this upper bound (1) has a negative correlation with the adversarial perturbation during AT; therefore, a larger perturbation radius provides better input-space robustness, and (2) links robustness to singular values of the embedding matrix. Motivated by this, they propose Embedding-Regularized CAT (ER-CAT), which adds a penalty on the variance of the singular values of the embedding matrix to push them away from being too large or too small. They compare ER-CAT with CAT on several open-source LLMs and jailbreak attacks to show that ER-CAT achieves a better robustness-utility tradeoff.", "questions": "1. In the ICL linear-regression surrogate, what is the intended correspondence between ($x_{\\tau, i}$, $y_{\\tau,i}$) and standard LLM training data? Specifically, should $x_{\\tau, i}$ be thought of as a single-token feature vector or a compressed/pooled representation of a multi-token prompt example, and if the latter, how is it modeled, and at which stage relative to $W_E$? Also, in this case, how realistic is the modeling choice that $x_{\\tau, i}, x_{\\tau,q} \\sim N(0, \\Lambda)$? A clarification on this helps in understanding the modeling better.\n\n2. What is the reason behind choosing ICL suffix embedding attack for the robust generalization risk analysis? What other types of token-space adversarial attacks do you expect your analysis to extend to?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The theoretical analysis on LSA-E models and the robust generalization upper bound is an interesting contribution that sheds light on how robustness of embedding-space adversarial training is connected to the singular values of the embedding matrix and the adversarial perturbation radius.\n\n2. Motivated by the theoretical findings, the paper proposes ER-CAT, a singular-value-based regularization for CAT, and presents experiments where ER-CAT provides some improvements in model robustness (lower ASR) while maintaining approximately similar utility. Although the evaluation setup could be improved (see below), this shows some promise in using embedding-based regularization within CAT.", "weaknesses": "1. The connection between ICL embedding AT for LSA-E models and continuous AT in LLMs is underexplained. The theoretical analysis relies on a linear self-attention surrogate and does not capture non-linearities in modern LLMs. Although the paper claims ICL embedding AT is a good approximation for real-world LLM CAT, this is insufficiently supported, and while the empirical results are encouraging, it's unclear when and why robustness gains proven for LSA-E transfer to CAT on large non-linear multilayer transformers.\n\n2. The empirical results present robustness (Table 1) and utility (Table 2) separately, making it difficult to evaluate whether ER-CAT truly provides a better robustness-utility tradeoff than CAT. When ER-CAT achieves lower ASR (better robustness), it often comes at the cost of reduced utility. The paper would benefit from visualizations such as ROC-style curves that plot utility against robustness, allowing direct comparison of the methods at equivalent operating points. \n\n3. Writing/notation issues:\n- The loss defined for continuous AT in equation (3) has a mistake. Given that in the text, $y$ is the target harmful resposne and $\\tilde{y}$ is the safe response, the current version of the loss in equation (3) is encouraging the harmful response and decreasing the likelihood of the safe response. $y$ and $\\tilde{y}$ should be swapped in the Adversarial Loss. This holds similarly for equation (4).\n\n- (minor point) In several of the equations (such as equations (7) and (9), dot (\".\") is used for matrix multiplication, which to me is a non-standard and confusing notation. I suggest using dots only for dot products and not matrix multiplication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mOnaCN34KO", "reviewer": "Reviewer_hwXR", "summary": "This paper performs a theoretical analysis on the Continuous Adversarial Training (CAT) method for defending LLMs against jailbreak attacks. Building upon ICL theory, the authors leverage a systematic derivation using a linear Transformer framework and propose a deference approach, termed Embedding-Regularized Continuous Adversarial Training (ER-CAT).", "questions": "See the Weaknesses part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Improving Adversarial Training (AT) efficiency through embedding-space perturbations offers a promising and theoretically grounded direction for enhancing model robustness.\n2. The authors provide a comprehensive theoretical proof under the in-context learning (ICL) framework, establishing clear connections between embedding-space adversarial perturbations and robustness in input space. Based on this analysis, they ultimately propose a new CAT method.\n3. ER-CAT extends CAT by introducing an additional embedding regularization term, which constrains the variance of singular values in the embedding matrix. It can be seamlessly implemented within existing CAT training pipelines.", "weaknesses": "1. The theoretical analysis is derived using Linear Self-Attention (LSA) models. While this is a common in ICL, the gap between the linearized setting and real large-scale LLMs is neither fully explored nor quantitatively characterized. A discussion or empirical validation of how the derived theory translates to practical LLM is preferred.\n2. The paper does not report ablation studies on key hyperparameters. For instance, it remains unclear how sensitive the results are to the regularization strength (β) and perturbation radius (ε).\n3. All evaluations are conducted using known jailbreak attack benchmarks. It is unclear for other types of attacks.\n4. One of CAT’s main motivations is to improve training efficiency over standard adversarial training. However, ER-CAT’s computational overhead relative to CAT is not analyzed.\n5. Since the proposed regularization explicitly constrains the singular value distribution of the embedding matrix, it would be insightful to show how these singular values evolve during training. A visualization could empirically support the theoretical claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E5IlqWzZ1U", "reviewer": "Reviewer_iaCj", "summary": "This paper addresses the problem of Continuous Adversarial Training (CAT) for LLMs, an efficient defense against jailbreak attacks that operates in the continuous embedding space. While CAT is empirically effective, the theoretical reason why perturbations in the embedding space can confer robustness against attacks in the discrete token space has remained unknown.\n\nThe authors provide the first theoretical analysis to bridge this gap. They leverage In-Context Learning (ICL) theory, modeling the problem using a Linear Self-Attention with Embedding (LSA-E) model trained on linear regression tasks. Their key theoretical contribution is a robust generalization bound for this model. This bound demonstrates that the model's robust risk (against input-space perturbations) is negatively correlated with the perturbation radius (epsilon) used during embedding-space AT. This result provides a formal explanation for CAT's effectiveness.\n\nFurthermore, the bound reveals that model robustness is closely tied to the singular values of the embedding matrix. Based on this insight, the authors propose a novel method, Embedding-Regularized Continuous AT (ER-CAT), which adds a regularization term to the CAT objective to minimize the variance of the embedding matrix's singular values. Empirical results on six real-world LLMs and six jailbreak attacks show that ER-CAT generally achieves a better robustness-utility tradeoff than standard CAT.", "questions": "1. How was the regularization coefficient beta = 0.2 for ER-CAT selected? Could you provide a sensitivity analysis or ablation study on at least one model to show how performance (both ASR and LC-WinRate) changes with different values of beta?\n\n2. What is the practical computational overhead (e.g., increase in training time per step/epoch) of calculating and backpropagating through the singular value variance regularization in ER-CAT compared to the standard CAT baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper's primary contribution is highly original. It tackles a novel and important theoretical question (i.e., \"Why does CAT work?\") that sits at the intersection of LLM robustness and efficiency. The choice of using ICL as the analytical framework is creative and non-trivial, establishing a new and insightful link between these fields. The development of a new defense (ER-CAT) directly from the theoretical findings is a good example of theory-driven research.\n\n**Significance**: The work is of high significance to the ICLR community. As LLMs become more pervasive, efficient and principled defense mechanisms are critical. CAT is a pragmatically important method, and this paper provides the first solid theoretical grounding for it. This understanding can lead to more principled improvements beyond simple heuristics, strengthening the foundation for future work on robust LLMs.\n\n**Clarity**: The paper is well-written. The abstract and introduction clearly articulate the problem, the gap in existing knowledge, and the paper's contributions. The logical flow from the core research question to the theoretical setup, the key findings (Theorem 2), the implications (Sec 4.3), and the proposed method (ER-CAT, Eq. 15) is clear and easy to follow.", "weaknesses": "**The Theory-Practice Gap**: The primary weakness of this paper is the gap between the theoretical model and the practical application. The analysis relies on a Linear Self-Attention (LSA-E) model trained on a linear regression task. This is a massive simplification of a multi-billion parameter, highly non-linear Transformer LLM performing the complex task of \"refusal.\" While the authors provide some justification (Sec 4.1), the leap of faith required is substantial. The paper would be strengthened by a more explicit discussion of these limitations and the potential risks of generalizing these linear insights to the non-linear deep learning regime.\n\n\n**Missing Ablation for New Hyperparameter**: The proposed ER-CAT method introduces a new hyperparameter, beta (the coefficient for the singular value regularization), which is set to 0.2. The paper provides no ablation study or sensitivity analysis for this hyperparameter. How was this value chosen? Is the method's performance robust to changes in beta? This is a crucial missing piece of the empirical validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhTuEnbSTx", "reviewer": "Reviewer_3RMS", "summary": "The authors investigate the ability of continuous adversarial training (CAT) to improve the robustness of LLMs using in-context learning (ICL) theory. The authors formalize a simplified surrogate problem based on linear transformers trained on in-context linear regression tasks. They prove that the robust generalization bound of models trained with CAT negatively correlates with the perturbation radius in embedding space and depends on the singular values of the embedding matrix. Based on their theoretical observations, they argue that the singular values should be constrained in the embedding matrix and provide a new version of CAT that achieves better accuracy robustness tradeoffs.", "questions": "- The theory provided in the paper provides some connections between the operator norm of the embedding function and the robustness of LLMs. Do the authors think that there is a general connection concerning the Lipschitzness of the function (LLM) and its robustness in the context of adversarial robustness in LLMs? (e.g., similar to [1])\n- Limited empirical analysis regarding model properties. How robust are models to continuous attacks? Do the regularized embedding matrices successfully reduce the effect of perturbations? Are the singular values of the final matrices considerably lower? Is there a connection between the magnitude of the singular values and the robustness? Do more robust existing models generally have lower singular values?\n- The surrogate setting appears somewhat simplified and not able to capture the complexity of the actual problem. Could the authors motivate why this might not be the case?\n\n[1] Roth, Kevin, Yannic Kilcher, and Thomas Hofmann. \"Adversarial training is a form of data-dependent operator norm regularization.\" Advances in Neural Information Processing Systems 33 (2020): 14973-14985.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The beginning of the paper is easy to follow and motivates the proposed analysis intuitively with prior work. The relevant context is clearly explained and provides a better formal description than the original works in some cases\n- Provides the first theoretical analysis of CAT, explaining why embedding-space perturbations can improve robustness to input-space jailbreaks, a contribution missing from the large number of adversarial training works in the LLM space.\n- Based on their theory, the authors are able to provide a practical algorithm (even if empirical results are somewhat inconclusive in my opinion)", "weaknesses": "- I found the description of the ICL theory hard to follow and would have appreciated additional text motivating / explaining the provided notation. Section 4.1 is hard to read.\n- The results provided regarding the performance of ER-CAT are somewhat inconclusive. For LLAMA-2-7B there is no clear advantage visible for ER-CAR. Moreover, it remains unclear if the benefits stem from the regularization approach or just from insufficient hyperparameter tuning of both methods. The large number of hyperparameters in LLM adversarial training appear to make it difficult to compare two appraoches (specifically if results are somewhat close)\n- Instead of averaging the ASR for the @5 approach, I would argue it's more sensible to compute the \"max\" harmfulness over all trials, as we are generally interested in lower bounds on the robustness.\n- Could the authors provide an \"ensemble ASR\" value that describes the lower bound robustness against all attacks (e.g., if one attack out of all succeeds for a prompt it can be considered broken). This would enable easier comparisons between CAT and ER-CAT regarding utility robustness trade-offs \n- Assumptions made in the theoretical part could be more clearly highlighted / summarized\n- Relevant hyperparameters that are important in the theoretical part are not analyzed (e.g., the perturbation magnitude constraint)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)