Explaining the role of Intrinsic Dimensionality in Adversarial Training
Authors: Enes Altinisik
Safa Messaoud
Husrev Taha Sencar
Hassan Sajjad
Sanjay Chawla
Venue: ICML 2025 poster (2025)
Keywords: adversarial training
robustness
data manifold
Primary area: deep_learning->robustness
TL;DR: SMAAT provides an explanation for the discrepancy in robustness and generalization trends between vision and language models. Using it, SMAAT offers better robustness for encoder-based LLMs by using 25-33% less GPU time compared to standard AT.

Abstract: Adversarial Training (AT) impacts different architectures in distinct ways: vision models gain robustness but face reduced generalization, encoder-based models exhibit limited robustness improvements with minimal generalization loss, and recent work in latent-space adversarial training demonstrates that decoder-based models achieve improved robustness by applying AT across multiple layers.
We provide the first explanation for these trends by leveraging the manifold conjecture: off-manifold adversarial examples (AEs) enhance robustness, while on-manifold AEs improve generalization.
We show that vision and decoder-based models exhibit low intrinsic dimensionality in earlier layers (favoring off-manifold AEs), whereas encoder-based models do so in later layers (favoring on-manifold AEs). 
Exploiting this property, we introduce SMAAT, which improves the scalability of AT for encoder-based models by perturbing the layer with the lowest intrinsic dimensionality. This reduces the projected gradient descent (PGD) chain length required for AE generation, cutting GPU time by 25–33% while significantly boosting robustness. We validate SMAAT across multiple tasks, including text generation, sentiment classification, safety filtering, and retrieval augmented generation setups, demonstrating superior robustness with comparable generalization to standard training.

Explaining the role of Intrinsic Dimensionality in Adversarial Training
Enes Altinisik
Safa Messaoud'
Husrev Taha Sencar'
Hassan Sajjad 2 Sanjay Chawla'
Abstract
Intrinsic D emeeroorality Id
Adversarial Training (AT) impacts different archi- tectures in distinct ways: vision models gain ro- bustness but face reduced generalization, encoder- based models exhibit limited robustness improve- ments with minimal generalization loss, and re- cent work in latent-space adversarial training (LAT) demonstrates that decoder-based models achieve improved robustness by applying AT across multiple layers. We provide the first ex- planation for these trends by leveraging the man- ifold conjecture: off- mififoldf adversarial exam- ples (AEs) enhance robustness, while on-manifold AEs improve generalization. We show that vi- sion and decoder-based models exhibit low intrin- sic dimensionality in earlier layers (favoring off- manifold AEs), whereas encoder-based models do in later layers (favoring on-manifold AEs) Exploiting this property, introduce SMAAT. which improves the scalability of AT for encoder- based models by perturbing the layer with the lowest intrinsic dimensionality. This reduces the projected gradient descent (PGD) chain length required for AE generation, cutting GPU time by 25-33% while significantly boosting robust- ness. We validate SMAAT across multiple tasks, including text generation, sentiment classification, safety filtering, and retrieval augmented gener- ation setups, demonstrating superior robustness with comparable generalization to standard train- ing.
Generalization Robustrees
Reco Recnnsruct coruuciutoonn Error 1.05)
Gene peneralztin aaties Score Rozustr (421)
Cn Mureiete Rati
26
Layer Depth
Figure I. Impact of applying LAT at different layers of the LLaMA- model, illustrating the relationship between Intrinsic Dimen- sionality (background color), Generalization (bluc), Robustness (orange), and Off- Manifoaiiold Ratio (green, based on reconstruc- tion emror). Markers show the average of measured values across multiple training configurations; lines depict overall trends. The off-manifold ratio measures the percentage of adversarial examples that fall outside the data manifold using reconstruction error. As deeper layers, the Intrinsic Dimensionality increases, resulting in decrcase in the off- manifold ratio. According to the manifold conjecture, this leads to increase generalization (more on- on-manifold samples) and a decrease in robustness.
2017). It is formulated as a min-max optimization problem, where the outer minimization optimizes model parameters, and the inner maximization seeks worst-case input perturba- tions. In deep networks, the inner maximization is typically solved approximately using multiple iterations of projected gradient descent (PGD, Madry et al. (2018)). However, the trade-off between robustness and generalization remains poorly understood. For instance, AT reduces generaliza- tion in vision models, leading to an 8% drop on CIFAR- (Shafahi et al., 2019), whereas encoder-based language models often retain or even improve generalization, achiev- ing 1% gain on AGNEWS (Zhu al., 2020). Moreover, ro- bustness gains significantly higher in vision models (e.g.. 40% on CIFAR-10 compared to encoder- models (e.g.. 10% on AGNEWS). Recent work on Latent Adversarial
1. Introduction
Adversarial Training (AT) is the most effective approach for improving the robustness of deep neural networks against small input perturbations (Bai et al., 2021; Kurakin et al.,
Qatar Computing Research Institute, HIKKU Doha, Qatar FFaculty of Computer Science,Dalhousia University. Hali- fax, Canada. Correspondence to: Enes Altinisik <ealtin- i hbbku.ed..qa..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Training (LAT) suggests that applying AT across multiple layers yields better robustness and generalization than fo- cusing on single layer (Sheshadri al., 2024). While prior

---

SMAAT
Standard
Frusts
Serto
ISMI
SHART IDund
the difference in the robustness magnitudes across vision models and enc-LLMs. Our findings are also consistent with YOPO (Zhang et al., 2019a) and TMD (Minh & Luu, 2022). Specifically, YOPO highlights the critical role of the first layer in vision models for AT, while TMD uses the last layer to detect AEs in enc-LLMs.
536647 (duno)
SMAAT 1Ouno
94-
Rebust Rdp300 (S)
We further leverage our insights on the impact of the mani- fold ID on robustness and generalization and hypothesize that perturbing the intermediate layer with the highest off- manifold AE ratio (equivalently lowest ID) should achieve high robustness at low computational cost. Intuitively, a lower ID corresponds to a higher proportion of OFM-AEs generation and greater improvement in robustness by the manifold conjecture. Additionally, layers closer to the out- put result in shorter PGD chains, leading to efficient AT. To this end, we propose SMAAT', Scalable Manifold Aware AT approach that applies AT at the layer with the highest proportion of OFM-AEs. We found that this critical layer is consistently the last layer across enc-LLMs on several applications. Hence, for enc-LLMs, SMAAT leads to a sig- nificant speed-up of AT by avoiding a full backward pass as it calculates the gradients only until the last layer rather than the entire network. Moreover, we empirically show that this results in higher robustness. Yet, this ID trend is different for vision or dec-LLMs, where we find that the first layer is always the one with the lowest ID. In these cases, SMAAT effectively reduces to standard AT.
-23
Training
SMAAT
(Oun)
92
Robust Acc
Robus Arc (%)
Figure 2. Comparison of SMAAT robustness (x-axis), generaliza- tion (y-axis), and run time (marker size) against basclines for robustifying (a) topic classifiers, (b) retriever models in the Re- trieval Augmented Generation (RAG) setup and (c) safety filters for decoder-based LLMss SMAAT significantly enhances model ro- bustness compared to seven different baselines, while maintaining nearly the same clean accuracy. Besides, it is significantly more scalable than AT (marker size).
research has investigated generalization loss in vision mod- els (Madry al., 2018; Wang et al., 2019b; Altinisik al., 2023a; Zhang et al.. 2019b; Cheng al., 2022), no study has systematically explored the robustness- generraiingnnn dynamics across vision models (CNNs and Vision Trans- formers), decoder-based (dec-LLMs), and encoder-based (Lnc-LLMs) language models. Additionally, the high com- putational cost of AT limits its practical deployment. Recent efforts to reduce the number of PGD steps- by reusing or accumulating gradients during updates (Shafahi al., 2019; Zhang et al., 2019a; Zhu et al., 2020)- have improved ef- ficiency but still require full network passes, making AT computationally expensive.
Given enc-LLMs continue to play a crucial role in machine learning pipelines, there is significant value in effectively en- hancing their robustness. To this end, evaluated SMAAT in improving the robustness of (1) classifiers, (2) safety filters, and (3) retrievers within Aatrivval-Aummeneer Gen- eration (RAG). SMAAT achieved state-of-the-art (SOTA) runtime and robustness results on all tasks, while maintain- ing clean accuracy (generalization) comparable to standard training. Specifically, in sentiment and content classification tasks, SMAAT improved the robustness of enc-LLMs AG- NEWS (Zhang et al., 2015), IMDB (Maas et al., 2011), and YELP (Zhang et al., 2015) datasets by 8.6%, 15.7%, and 28.8% for BERT (Devlin et al., 2019) and by 6.0%. 5.8%, and 19.0% for ROBERTa (Liu et al., 2019), respectively. For safety filtering, SMAAT accurately identified harmful prompts generated by GCG (Zou et al., 2023), achieving 97-100% accuracy. In RAG experiments, SMAAT signif- icantly enhanced the robustness of the Contrevier model (Izacard et al., 2022) on the RAG setup, achieving over 80% robustness against poisoning attacks (Zhong et al., 2023). Besides, SMAAT required only about 25-33% of the GPU time compared to the standard AT. A summary of all results is presented in Fig. 2.
In this work, we investigate how differences in robustness and generalization trends across foundational models relate to the intrinsic dimensionality (ID) of the data manifold. The data manifold is a potentially non-linear subspace spanned by the dataset, and its dimensionality influences whether ad- versarial examples (AEs) lie on the manifold mannfoiifld AEs) or fall outside it (off- ooff-manifold AEs) during training. We show that the ID of the data manifold in the first layer is much higher in enc-LLMs compared to vision models and dec-LLMs which results in AEs being more on- mn-manifold (ONM-ABs) in enc-LLMs and more off manifod (OFM- AEs) in vision models and dec-LLMs. In accordance with the manifold conjecture (Ethayarajh, 2019; Shamir et al., 2021; Gilmer et al., 2018), we find that OFM-AEs lead to better robustness, while more ONM-AEs lead to better gen- eralization (Fig. 1, with details provided in Section 4). To the best of our knowledge, this is the first explanation for
'The code is publicly available at: https:/ hggggggtthh...ommm KnesAl t iniiiiikkissi SMAAT- 20M-AAA---///A httttee///aaa//////ee/eeeee////t
2

---

SMAAT
Studying the Difference in the Generalization & Robustness Trends in AT
SMAAT: Scalable Manifold Aware Adversarial Training
Encoder-Based LLMs
(x)
Decoder-Bass
(x(
Find (Eq. 6)
AT (Eq. 5)
LLMss & Vision
Standard AT AE; x + 80
Models
Iernisic Dimanaaa
1x
(x)
Layer depth
Layer depth
SMAAT
AES (On-manifold)
E
AES (Off-manifold) Generalizution 4
AE: f aaln-- (x) + 8g-1
Generalization
Scalability
ttf
Robustness 1
Robustness tif
Layer depth
Generalization
4 4-maniffold
4-maniffold
Robustness
Figure 3. Left: In classical AT, adversarial examples (AEs) are created in the data layer. For encoder LLMs, the intrinsic dimensionality tends to be high the initial layers and therefore the AEs created tend to be on-manifold which results in better generalization. In Vision and decoder LLMs, we observe the opposite behavior and AEs tend to be aff-mmnifold resulting in better robustness. Right: The key idea of SMAAT is to ereate AEs in intermediatc layers where the intrinsic dimensionality is low and AEs will tend be off-manifold. This results in better robustness while (surprisingly) maintaining generalization. The speed-up in SMAAT is due to the fact we need shorter backprop chains to create AEs intermediate layers.
In summary, the major contributions of our work are:
maps these latents to predictions i = her (f.). ). The standard AT objective under an Ly norm constraint of e is:
1. An explanation for the discrepancy in the robustness and generalization trends in foundation models.
min Σ max C(ha,(go, (x+ + 6c)), 0
2. SMAAT, novel AT algorithm that leverages the intrin- sic dimensionality across layers of foundational models to control robustness, generalization and scalability.
Similar input-space AT, the inner maximization finds the worst-case perturbation 6, while the outer minimization updates 9 both solved via gradient-based methods. How- ever, AT with P-step PGD is significantly more computa- tionally expensive than standard training, as it requires P forward-backwer passes per update, compared to just one in standard SGD.
3. Comprehensive experiments demonstrating enhance- ments in robustness and scalability while keeping curacy in classification and retrieval tasks compared to standard AT.
Adversarial Attacks on LLMs. Adversarial attacks on LLMs are more challenging than on vision models due to the discrete nature of text and tokenization. Early attacks on Lss used word substitutions guided by embedding similarity (Jin et al., 2020b), synonymity (Zang et al., 2020), or masked language models (Li al., 2020), primarily to flip classifier outputs. In contrast, recent attacks dec- LLMs focus alignment breaking objectives by appending adversarial prefixes or suffixes. For example, AutoDAN (Liu et al., 2024) and PAIR (Wei et al., 2023) craft prompts that encourage harmful completions, while Greedy Coor- dinate Gradient (GCG) attack (Zou et al., 2023) appends a gradient based generated adversarial suffix to user inputs. GCG combines affirmative prompting (Wei et al., 2023: Car- lini al., 2023) with greedy and gradient- based discrete optimization (Shin et al., 2020), and is notable for its strong transferability across prompts and models.
2. Related Work
Adversarial Training (AT) aims robustifying a model against AEs which are imperceptibly perturbed inputs that can lead to incorrect predictions. Formally, AT seeks opti- mal parameters 0* for a classifier (a) that remains robust to perturbations ò within norm ball:
min ((r))~
[
max E(fe(r + o), o),,),, [o
(1)
where is the loss function and D = (ax 1l represents the training data. The outer minimization is typically solved using stochastic gradient descent (SGD), while the inner maximization addressed via projected gradient descent (PGD) (Madry et al., 2018). LAT extends AT by applying adversarial perturbations to the model's 's latent representa- tions instead of its inputs (Casper et al., 2024; Sheshadri et al., 2024). Given a model with parameters = (0,, 02), computes heg g9,, where ge, is a feature extractor that maps inputs to latent representations l, = go, (mx), and ho,
Manifold-Basee Defenses. The manifold conjecture stands as one of the most compelling explanations for the suscep- tibility of deep neural networks to AEs (Tanay & Griffin,
3

---

SMAAT
2016; Gilmer al., 2018; Shamir al., 2021). The conjec- ture posits that data resides low-dimensional manifold within a high-dimensional representation space and that the deepnet learns to approximate this manifold. Consequently, an off-manifold sample, deviating from this foundational manifold, leads to undefined behavior of the model. This conjecture has inspired a novel line of defenses against ad- versarial attacks on images (Samangouei et al., 2018; Meng & Chen, 2017: Song et al., 2018; Schott et al., 2019) and text (Minh& & Luu, 2022). These methods approximate the data manifold and, during testing, project samples onto this manifold to either detect or correctly classify AEs. Differ- ently, we leverage the manifold conjecture during training to improve both robustness and scalability of AT.
3. Notation
Consider a deep neural network classifier fo with n layers and parameters 0 = ) We define the transforma- tion spanning layers to as far..1 = fain fate, with fait denoting its standardized version across the dataset D. The input matrix X' E ddd stacks all samples {a, 1 while fan.u (X) E R4 stacks their transformed rep- resentations. The data manifold is defined by the high- density region of the sample distribution, distinguishing on- manifold (high-density) from off- off-manifold (low-density) points. Similarly, the 1th layer manifold is characterized by the density of standardized representations (f (1:)) ) approximated via the eigenspace of the covariance matrix Sou..n Sooa (X)². We denote its eigenvectors and eigen- values as (U,, Mi), where Ui E [d xdi and Dn E R4x4,
Robustness and Generalization Trends in AT. AT is rec- ognized for enhancing model robustmess in both vision and enc-LLMs (Zhang et al., 2019b; Altinisik et al., 2023b). While this improvement in robustness comes at the cost of increased generalization error in vision models (Zhang et al., 2019b), AT enhances generalization in enc-LLMs (Altinisik et al., 2023b; Gan et al., 2020). There has been recently efforts relating robustness to OFM-AEs and generalization to ONM-AEs (Stutz al., 2019;; Xiao et al., 2025). We extend this line of research by connecting these trends to the intrinsic dimensionality of intermediate layers of a network.
The projection error on the top & eigenvectors is:
cx = fan.a()) UU, (x), E. (2)
The digenbbased manifold dimension ky is the minimum required for a reconstruction error yy:
{ k [1, {
|| (x) sn). l
h1 min k
Σ wED
(3)
Scalable AT. Different optimizations have been proposed to mitigate the cost of the PGD additional forward-backward backward passes, including (i) replacing multi-step PGD with single- step FGSM (Shafahi et al., 2019); (ii) omitting redundant computations during PGD (Zhang et al., 2019a); (iii) com- bining FGSM with random initialization (Wong et al., 2020). While these approaches alleviate the PGD overhead, they also come with limitations. As a solution, Freel.B (Zhu et al., 2020) proposes accumulating model parameter gradi- ents over multiple batches. We achieve scalability through an alternative method, i.e., by leveraging the manifold con- jecture perturb intermediate layers and thus reduce the length of backward-forwant PGD chains.
We classify adversarial examples (AEs) based projection error:
A (7, offf-mmit- AE (OFM-AE) has et > 7
(7, -onmmmiifll AE (ONM-AH) has et < 7
Finally, we denote the Intrinsic Dimension (ID) of layer 1, estimated via TwoNN, as It, and its normalized ID as I ===11 l/d1.
4. Exploring Layerwise ID and Its Effects
ID Estimation. It is well known that most real world data lives in a low-dimensional space relative to the ambient space where is defined. The ID of data is the minimum number of variables necessary to characterize important properties of the data. Singular Value Decomposition (SVD) is a well known method to estimate the ID assuming the data lives on a linear manifold (Stewart, 1993). While there have been many proposals to estimate the ID in non-linear set- tings, we will use the relatively recent twoNN ID-estimator based on the observation that for every data point x, the ratio y of distance of & to it's second and first nearest neighbor follows a Pareto distribution f [x([1) 1 (I+1), where I is the ID. It can be estimated as J = (og())(())) where log(a)
Our study is motivated by the not-well understood difference in the generalization and robustness trends between adver- sarially trained vision models and LLMs. Specifically, (1) robustness improvements are consistently more pronounced in vision models (Madry et al., 2018; Altinisik et al., 2023b), (2) generalization either remains stable or improves in enc- LLMs (Tsipras et al., 2019; Zhu et al., 2020), whereas it deteriorates in vision models, and (3) dec-LLMs achieve the optimal trade-off when AT is applied across multiple network layers (Casper et al., 2024; Sheshadri et al., 2024).
We demonstrate that the varying trends observed across dif- ferent network architectures are closely linked to the layer- wise distribution of OFM-AEs. Notably, we find that in the first layer, where conventional AT applied, adversarial
F(R) is the empirical CDF (Facco et 2017).
4

---

SMAAT
(a)
(b)
[c) Vision
(d)
Reconstructin Enror (e)) Normalzzed (22)
0.06
0.005
0.05
0.03
0.05
VIT
0.004
0.04
AGNEWS
0.04
AGNEWS
0.02
0.003
LLaMA-2
0.03
IMDB
IMDB
VGG-13
Mistral
0.002
0.02
YELP
0.02
YELP
VICUNA
0.00
20
0.30
AGNEWS
0.30
AGNEWS
0.60
VIT
0.40
IMDB
IMDB
DeiT
0.20
YELP
0.20
YELP
0.40
VGG-13
0.20
LLaMA-2
0.10
0.10
0.20
Mistral
0.00
0.00
0.00
0.00
20
Layer Depth 0)
Layer Depth 03)
Layer Depth
Layer Depth
Figure 4. The ID (row 1) trend follows the inverse OFM-/ONM-AEs ratio (row 2) trend. The average projection error (ef) is used proxy for estimating the OFM-/ONM- AEs ratio. The ID is computed using the twoNN approach. Enc-I.LMs (BERT, RoBERTa) have decreasing ID and OFM-AEs proportions trends unlike vision models and Msc-LLMs that have increasing ID and ONM-AEs trends.
examples for enc-LLMs tend to more on-manifold, while vision models and dec-LLMs show a higher proportion of off-manifold examples (see Fig. left). This explains the generalization and robustness trends as, by the manifold conjecture, higher proportions of ONM and OFM AEs lead, respectively. to better generalization and robustness (Ma et al., 2018; Stutz et al., 201 20; Alemany & Pissinou, 2020; Li et al., 2021a).
Mistrra--7-instrueet (Jiang et al., 2023)) on several datasets (refer to Appendix for details). Results are reported in Figure The ID (11²) in row computed using twoNN estimation (Facco al., 2017) and follows the OFM-AEs trend (row 2). To assess the proportion of OFM/ONM AEs across different layers, we use the reconstruction error ef at layer 1 (row 2) as a metric, i.e., higher projection error corresponds to higher proportion of OFM-AES.
ID & ONM/OFM Relationship We hypothesize that lower intrinsic dimensionality (4,5) leads to a higher proportion of OFM-AEs, whereas higher If" yields more ONM-AEs. For- mally, let .M C IR. be smooth, compact, low-dimensional manifold with intrinsic dimension 1/¹ << n. embedded in IR.. IR* be a classifier trained on data sampled
ID & Kobustness/Generalzaaiio Relationship
We extend our evaluation of the relationship between in- trinsic dimensionality (ID), robustness, and generalization by applying adversarial training (AT) to different layers of dec-LLMs (LLaMA-2-7B-chaa) enc-LLMs (BERT), and vision models (VGG). For the dec-LLM, we train LLaMA- 2-7B-chat on the LAT dataset (Sheshadri et al., 2024) using various hyperparameter configurations (see Appendix C for details), applying LAT at every even-numbered layer. To evaluate generalization, 2024), while robustness is assessed via failure rates against the GCG attack, which generates adversarial suffixes that bypass language model safety alignment. Additional results using the PAIR (Chao et al., 2023) attack are provided in Appendix C. As shown in Fig. 5(a), applying LAT to lower layers (light blue markers) enhances robustness but reduces generalization (bottom-right), while applying it to upper layers (dark blue markers) improves generalization at the cost of robustness (top-left). This trend aligns with the in- trinsic dimension (ID), which decreases in higher layers, as well as with the distribution of ONM/OFM adversarial examples. These results also explain the superiority of the LAT approach of Sheshadri al., 2024 which applies AT to four evenly spaced layers across the network.
Let / R" + from M. Since the classifier learns p(y x) without ac- cess to the true generative distribution p(z). lacks explicit knowledge of the underlying data manifold M. Conse- quently, the loss gradient V,L often contains components that are orthogonal to the tangent space of M. As result, the perturbation 5 is unlikely to remain entirely within the manifold and typically includes a non-zero component in the orthogonal direction. Moreover, when I is small relative to n, the proportion
MT-Bench (Zheng et al.,
that lies off the manifold increases. Thus, adversarial examples generated via gradient-based methods more likely to fall outside the data manifold, i... to be off-manifoll adversarial examples.
We base this conclusion on an empirical investigation into the relationship between the ID and the proportion of ONM/OFM AES across different layers of various deep neu- ral network models, including two enc-LLMs (BERT and RoBERTa), three vision models (ViT (Dosovitskiy et al., 2021), DeiT (Touvron al., 2021), VGG-13 (Simonyan & Zisserman, 2014)), and three dec-LLMs (LLaMA-2-7b- chat (Touvron et al., 2023), Vicuna (Zheng et al., 2024),
We conducted similar experiment on the BERT model us- ing the YELP dataset (see Sec. 6.1 for details). Our results
5

---

SMAAT
40-
Geneer Denerraltcation (MT--bench)
Generalizatoon (CA)
91-
47
Layer
R2
Layer
Generalizatio
96-
Layer
22
90-
24
25
89-
10
30
12
0.6
1.0
15
Robustness under
Robustness (AUA)
Robustress (AUA)
(a) LLaMA-2
(b) BERT
(c) VGG
Figure 5. Layer-wise effects of adversarial training on robustness and generalization across model types. Each subplot shows the impact of applying AT at different layers of (a) ILaMA-2 (dec-ILLM), (h) BERT cnc-ILLM), and (c) VGG (vision model). Marker colors transition from light blue (lower layers) to dark blue (higher layers). Observed trends align with changes intrinsic dimensionality and the distribution of on- vs. off manifold adversarial examples, as shown in Figure 4.
empirically show that, following the decreasing ID trend, applying AT at higher layers leads to increased robustness as the layer index i increases. Figure 5(b) presents the re- sults of AT on BERT for the YELP dataset. As expected, robustness (measured under the TextFooler attack) improves with higher layer indices while generalization remains unaf- fected.
to the input one is significantly improving scalability by a factor O P(n-1*) -
max (di,14 14 E [1,7* - 1]3 - di-
as it results in shorter P-step PGD chains.
5. SMAAT: Scalable Manifold Aware Adversarial Training
We also extend this analysis to vision models. We selected VGG-13 and trained it on CIFAR-10 10 (Krizhevsky, 2009) using AT applied to every ReLU layer over 20 epochs. We varied the attack strength (e 0.031-0.2) and learning rate (Ir 0.01-0.001] evaluating robustness via Robust- Bench O. As shown in Fig. 5(c), adversarial training lower layers (light blue) improves robustness but reduces generalization, whereas training on higher layers (dark blue) yields the opposite pattern. These results suggest that vision models resemble dec-LLMs in terms of their generaliza- tion-robustness tradeoff. Finally, our findings are consistent with those shown in Fig. 4, where higher ratio of OFM- AES in deeper layers correlates with improved robustness in both encLLLMs and vision models.
We propose SMAAT an AT algorithm that aims improving the robustness and scalability of standard AT by generating AEs in the layer leading to the highest proportion of OFM instead of from the input layer as classically. Specifically, given a pretrained model fe. SMAAT intentionally generates a higher proportion of OFM-AFs to enhance robustness. Formally, SMAAT solves the augmented AT objective:
min E(aN)D max E(fo(r (()((=+)))))) [1oliSe
(4)
s.t. (x (x is 1),
While the manifold conjecture refers to input space AEs, we relax it to encompass intermediate layers as well, i.e., we chose the perturbation that results in an OFM-AE at any layer across the deepnet. Intuitively, AEs that are off- or on- the transformed data manifold in any layer also affect robustness. The relaxed objective becomes:
Beyond providing an explanation for the difference in robust- ness and generalization trends in vision models, enc- and dec-LLMMs, results of Fig. 4 shed light on an interesting trend for the ID across layers. While the ID increasing in vision and dec-LLMs, it is monotonically decreasing in enc-LLMs (Bert and RoBERTa). This motivated the design of new al- gorithm that controls the gains in robustness and generaliza- tion via controlling the proportions of ONM- and OFM-AEs. This can be achieved by generating AEs in the intermediate layers of the deepnet based on their IDs. To this end, we pro- pose SMAAT (See Fig. 3, right), an AT algorithm that aims at achieving high robustness by perturbing the layer I" with the lowest ID to generate high proportion of OFM-AEs. A side effect of perturbing an intermediate layer opposed
min 6
E E( max {(fe(z + 6),
|
(5)
s.t. L]: + 8) is (7, I)-MM.
Note that solving the above objective using the method of Lagrangian Multipliers is possible, but it would require an approximation of the manifold to characterize OFM-AFs. Such approximations are either computationally expensive (e.g., GAN (Xiao et al., 2018)) or overly simplistic (e.g., eigenbasis (Xiao et al., 2025)). SMAAT uses an alternative approach find OFM-AEs by perturbing the layer with the
6

---

SMAAT
Table Robustifying sentiment classifiers AGNEWS, IMDB, and YELP datasets. We report the clean accuracy (CA) and the robust accuracy under attack (AUA) with PWWS (PW). TextFooler (TF), and Bert-Attack (BA), along with the average robust accuracy (AR) across these three attacks. The best performance bolded. Across data sets, SMAAT achieves high robustness (AR) while maintaining high generalization (CA).
AGNEWS
IMDB
YELP
Model
Defense
AUA
AUA
AUA
CA
CA
CA
PW
TF
BA
AR
PW
TF
BA
AR
PW
TF
BA
AR
Standard
94.5
36.9
28.1
37.5
34.2
92.2
15.0
5.8
5.4
8.7
97.0
12,2
6.5
5.3
8.0
ASCC
91.6
32.8
31.4
32.1
32.1
88.5
15.1
12.4
11.2
12.9
91.5
19.4
15.7
12.2
15.8
FreeLB++
95.1
47.9
51.5
41.8
47.1
93.2
12.5
45.3
39.9
32.6
95.6
19.3
8.8
3.7
10.6
BERT
SAFER
94.4
39.3
35.5
42.3
39.0
92.3
41.4
39.1
30.7
37.1
95.4
29.8
25.8
23.7
26.4
TMD
94.3
70.0
50.0
55.2
58.4
92.2
38.7
44.2
33.7
38.9
95.2
36.8
40.9
28.6
35.4
RSMI
92.7
76.1
63.2
NA
NA
92.2
58.7
56.4
NA
NA
95.4
45.3
52.3
NA
NA
SMAAT (Ours)
94.6
73.5
72.2
74.7
73.5
92.2
63.6
77.9
60.8
67.4
97.0
77.1
77.9
72.8
75.9
Standard
94.7
30.6
23.9
37.1
30.5
94.0
8.7
2.1
0.6
3.8
97.9
23.1
14.9
9.0
15.7
ASCC
92.6
48.1
41.0
49.1
46.1
92.6
23.1
13.5
11.8
16.1
95.4
15.0
8.6
4.5
9.4
ROBERTa
FreeLB++
95.6
61.0
49.8
56.6
55.8
94.3
33.6
14.6
6.1
18.1
97.0
38.6
46.0
35.2
39.9
SAFER
94.6
68.9
49.3
46.1
54.8
93.9
52.8
47.1
40.6
46.8
96.6
65.6
67.9
48.3
60.6
TMD
95.0
68.3
54.0
56.7
59.7
93.3
60.5
66.8
51.6
59.6
96.6
68.9
70.9
51.0
63.6
RSMI
94.3
81.9
74.1
NA
NA
93.0
76.2
73.4
NA
NA
96.3
68.9
65.9
NA
NA
SMAAT (Ours)
94.6
75.6
75.1
79.9
76.9
93.5
77.1
78.5
63.2
72.9
98.0
85.4
86.4
76.0
82.6
lowest ID without the need for manifold approximations. Particularly, SMAAT applies AT 1*-th layer where the layer with more OFM-AEs composition (lowest ID):
of classes small enough to be negligible. Hence, SMART enhances the efficiency of the AEs generation process by factor of l ((max(d;)). This improvement practically eliminates the cost of the AE generation process.
(Tgur..)) (JoL..)(x) + 61-)
min E (2
max |1
6. Experiments
(6) We additionally choose ( to correspond to the layer with the highest index as this would lead better scalability (shorter PGD chains). well to potentially the highest proportion of off off-manifold AEs at any intermediate layer, i.e. 1" max (1 [1, I s L <1
We assess robustbness, generalization and scalability of three types of models trained with SMAAT: (i) classifiers; (ii) retrievers in RAG settings and (iii) safeguarding models employed for moderating content produced by generative models. We consider four different attacks: (i) word substi- tution (Ren et al., 2019; 9; Jin et al., 2020a; Li et al., 2021b), (ii) adaptive (Tramèr et al., 2018), (iii) Greedy Coordinate Gradient (GCG) (Zou et al., 2023), and (iv) corpus poison- ing (Zhong et al., 2023) attacks. We include run-time analysis demonstrate scalability of SMAAT.
Complexity, Computing the layers IDs and searching for the optimal layer I" perturb done once per model and per task. Thus, they incur marginal overhead. For the AT part, when a P-step PGD attack is used, this results in P baakurdd passes with length (n (n- 1" +*++1 1) instead of n. The run-tine run-time of every forward/backwarrd pass depends on the layer dimensionality, i.e., (dd-1 x di) for the 1th layer. Overall, the complexity of one SMAAT forward-backwant O(n - 1' + 1) maxiell .n] ))) ). As a result, SMAAT is more efficient than standard AT by a factor of O(P x I "T-ax,e (d max max |e - m] )))) ).) In the case Enc- LLMs (e.g., BERT and RoBERTa), [" is equal to n. The total run-time can be simplified to O(P- (d))²) where d, is the number of classes. Typical enc-LLMs tasks consist of less than five classes (Wang et al., 2019a) which makes the factor
6.1. Sentiment and Topic Classifiers.
We adversarially trained two base models, BERT-base- cased and RoBERTa-base-case on the tasks of sentiment classification (IMDB (Maas et al., 2011) and YELP (Zhang et al., 2015)) and topic classification (AGNEWS (Zhang et al., 2015)) under three input space attacks including PWWS (Ren et al., 2019) (synonym based), TextFooler (Jin et al., 2020a) (neighbor based), and BERT-Autack (Li et al., 2021b) (masked-LM based). Attacks are conducted

---

SMAAT
Table 2. Run-time results IMDB dataset. Mean and standard deviation are computed over ten runs. Compared to FreeI.B++, training time of SMAAT is lower by a factor of nearly three.
Standard
ASCC
Fre+LB++
SAFER
RSMI
SMAAT
Training (min/epoch)
5.1 #0.1
25.7 #0.3
15.6 +0.5
8.2 20.6
15.4 +0.3
5.2 #0.2
Inference (msec/sample)
2.4 +0.1
41.4+0.7
2.4 #0.0
2.4 #0.0
5.6 #0.4
2.4 +0.1
0.5 drop in performance in the case of RoBERTa with the IMDB dataset. Besides, SMAAT consistently outperforms TMD, even though it is manifold-basee method differently from SMAAT, TMD estimates the manifold and projects the imput samples onto it before classification witch could lead to inaccuracies in the manifold estimation. SMAAT on the other hand is more robust to such inaccuracies as it only leverages the ID for selecting the layer to perturb. RSMI shows strong robustness against PWWS, as its masked in- ference mechanism inherently mimics synonym- bbased de- fenses without needing an explicit synonym set (Minh & Luu, 2022).
Table 3. Robustifying Lec-LLMs with encoder-based (BERT etc.)
safety filters on AdvBench and HH RI.FF datasets. We report clean accuracy (CA), and robust accuracy under attack (AUA) which indicates the percentage of harmful prompts augmented with adversarial suffices accurately classified as harmful. Notice that SMAAT achieves very high robustness while other models sometimes completely fail.
BERT
RoBERTa
DistilBERT CA
Dataset
Model
CA
AUA
CA
AUA|
AUA
Standard
100
0
100
0
99.6
0
AdvBench
FreeLB++
99.6
0
100
0
99.6
0
SMAAT
100
100
100
99.2
99.2
97.5
Standard
95.4
26.4
94.8
0.3
98.7
6.5
Runtime efficiency: In Table 2, we provide details on the training time per epoch and inference time per instance for the IMDB dataset using BERT (RoBERTa has the same architecture). SMAAT has comparable efficiency to standard training and is on average times faster than standard AT during training. This is attributed to shorter backpropagation chains as AT in SMAAT is performed in the last layer (l" n). Note that FreeLB++, which injects noise in the first layer, remains inefficient even when PGD is replaced with FGSM (Zhu et al., 2020; Li et al., 2021b), as a full-depth backpropagation chain is still required. Certified defense baselines (SAFER, RSMI) and input space attack (ASCC) are also time-consuming as they either require mapping samples into the manifold performing an extensive search over word substitutes. Additional evaluation of SMAAT on language understanding benchmarks, including GLUE and advGLUE are reported in Appendix E.
HIH-RLHF
FreeLB++
98.6
48.5
98.6
34.0
98.5
34.1
SMAAT
95.4
51.6
94.8
96.8
98.5
39.5
using the TextAttack framework (Morris et al., 2020) and following the settings introduced by (Li et al., 2021b). We compare SMAAT to standard (non-adversarial) training, and six baselines from three families of defenses: (1) Input space AT (ASCC (Dong et al., 2021)), (2) Embedding space AT (FreeLB++ (Li et al., 2021b)), and (3) Certified defenses (SAFER (Ye et al., 2020), TMD (Minh & Luu, 2022), RSMI (Minh & Luu, 2022)). FreeLB++ focuses on scalability and robustness by minimizing the number of PGD steps and applying AT in the initial layer. TMD leverages manifold features by projecting samples back to the manifold in the last layer. Implementation details: In both base models, = n, i.e., we generate AES in the last layer. Specifically, we perturb the [CLS] embeddings before the classifier layer by freezing all layers before I*. Further details are presented in Appendix D.
6.2. Safety Filters
The broad attack surface of LLMs has compelled model owners to implement solutions that extend beyond the safety alignment of model. Notably, this includes content moder- ation filters that verify the harmlessness of model's inputs and outputs (Kumar al., 2023; Cao et al., 2024; Fatchkia et al., 2025). These filters are essentially text classifiers, often based on lightweight enc-LLMs to minimize overhead and preserve the high responsiveness of Ms (Kumar et al., 2023). The robustness of safety filters is also a con- cern, as they too can be vulnerable to attacks. Base models: We evaluate SMAAT'S effectiveness on BERT, RoBERTa,
Robustness and generalization: In Table 1, we report re- sults on robustness and generalization. On average, SMAAT demonstrates superior robustness over all datasets, with an improvement of 8.6%, 15.7%, and 28.8% over the best score for the BERT model, and 6.0%. 5.8%, and 19.0% for the RoBERTa model on the AGNEWS, IMDB, and YELP datasets, respectively. Note that FreeLB++, which perturbs the first layer, showed the best generalization in four out of six cases, as it produces mostly on-manifold examples (the first layer has a high dimension, as illustrated in Fig. 4) which is in line with the manifold conjecture. SMAAT main- tains generalization in five out of six cases and only shows
"RSMI takes about 2k times longer than TextFooler to generate a AE with BERT-Attack, making it unfeasible to test.
8

---

SMAAT
adversarial passages compared the others. In terms of gen- eralization, FreeI.B++ yields the best results it applies AT' in the first layer (resulting in more ONM-AEs), while both SMAAT and standard training exhibit similar performance.
Table Robustifying retriever models of RAG on the Natural Questions dataset. We report @ 10 (R@) 10) and R 100 @R@ 100) on the clean corpus for generalization. Robust recall (RR) is measured by how many samples are selected without ad- versarial passages in the top-k passages, with R 10 and R@ 100 corresponding the top- 10 and top-100 100100 passages, respectively, During attacks, 10 and 50 adversarial passages are created, de- noted as (N=10) and (N=50), respectively. SMAAT makes RAG substantially more robust against selecting adversarial passages.
7. Conclusion
In this paper, we explain the varying effects of AT across architectures using the manifold conjecture, showing that ID influences robustness and generalization. Vision and decoder-based models favor off-manifold AFs in early lay- ers, enhancing robustness but harming generalization, while encoder-based LLMs favor on-manifold AEs, preserving generalization with limited robustness gains. Leveraging this insight, we introduce SMAAT, which improves AT seal- ability by perturbing the layer with the lowest ID. This reduces training overhead by 25-33% while boosting ro- bustness across sentiment classification, safety filtering, and retrieval tasks, maintaining generalization. We plan to ex- plore joint optimization of generalization and robustness by AE ratio through perturba-
Generalization
RR (N=10)
RR (N=50)
Model
R@ 10 R@ 100
@@ 10 R@ 100
R 10
R@ 100
Standard
36.4
79.7
46.0
22.2
26.1
7.2
+r++
39.6
81.8
75.2
60.7
51.7
32.2
SMAAT
34.9
79.5
99.5
97.1
85.7
73.5
and DistilBERT when used as safety filters. Datasets: The AdvBench dataset (Zou et al., 2023) and the Helpfulness- Harmfulness dataset (HH)RLFF) (Bai et al., 2022). Adver- sarial attack:, we use GCG attack. Metrics: we assess gen- eralization accuracy (ACC) for safe and harmful prompts, and robust accuracy under attack (AUA) by measuring the detection rate of harmful prompts augmented with adver- sarial suffixes. Further details are presented in Appendix F. Robustness and generalization: Results in Table 3 show that SMAAT significantly enhances the robustness of safety filters against the GCG attack across all models, while main- taining generalization compared to standard training. Also, FreeLB++ improves generalization as it applies AT in the first layer.
controlling the tions in intermediate layers.
Acknowledgement
This work is partial supported by the Qatar National Re- search Fund (QNRF) grant NPRP11C-1229-17007.
Impact Statement
This research aims to deepen the understanding of how AT methods influence generalization and robustness, leading to more effective and practical adversarial training techniques. As machine learning models become increasingly integrated into real-world applications, it is crucial to develop frame- works that mitigate misuse while ensuring ethical compli- ance. Building on prior work in adversarial training, our approach enhances the robustness of encoder-based models while significantly improving computational efficiency.
6.3. Retriever Models of RAG
RAG combines retriever model, which identifies relevant passages from large corpus, with generator model that constructs answers based on the retrieved information. Base model: We use the Contriever model (Izacard et al., 2022), fine-tuned on the Natural Questions (NQ) (Kwiatkowski et al., 2019) dataset, as our retriever. Baselines: We eval- uate the robustness of retriever models within the RAG framework under standard, Freel.B++ and SMAAT. Adver- sarial attacks:, we use poisoning attacks (Zhong et al., 2023), which manipulate the retrieval corpus by generat- ing adversarial passages. As metrics, we use robust recall (RR) measured by how many samples are selected with- out adversarial passages in the top-k passages, with R 10 and R @ 100 corresponding to the top-10 10 and top-100 pas- sages, respectively. Robustness and generalization: Table 4 shows that the standard model offers limited robustness against the attack, with a significant number of adversar- ial passages being retrieved. FreeLB++ shows some im- provement, reducing the number of adversarial passages retrieved. However, SMAAT significantly enhances robust- ness, demonstrating dramatic reduction in the retrieval of
References
Alemany, S. and Pissinou, N. The dilemma between data transformations and adversarial robustness for time series application systems. In AAAI, 2020.
Altinisik, E., Messaoud, S., Sencar, H. T., and Chawla, S. A3t: accuracy aware adversarial training. Machine Learning, pp. 1-20, 2023a.
Altinisik, E., Sajjad, H., Sencar, H., Messaoud, S., and Chawla, S. Impact of adversarial training on robustness and generalizability of language models. In Findings of the ACI,, 2023b.

---

SMAAT
Bai, T., Luo, J., Zhao, J., Wen, B., and Wang, Q. Recent advances in adversarial training for adversarial robustness. In UJCAI-21,000
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLRR 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint NrXii:2204.05862. 2022.
Ethayarajh. K. How contextual are contextualized word rep- resentations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In IJCNLP, 2019.
Cao, B., Cao, Y., Lin, L., and Chen, Defending against Hiignment-breaking attacks via robustly aligned LLM. In ACL, 2024.
Facco, E., d' Errico, M., Rodriguez, A., and Laio, A. Es- timating the intrinsic dimension of datasets by a mini- mal neighborhood information. Nature Scientific reports, 2017.
Carlini, N.. Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Koh, P. W., Ippolito, D., Tramèr, F., and Schmidt, L. Are aligned neural networks udversarially aligned? In NeurlPS, 2023.
Fatehkia, M., Altinisik, E., and Sencar, H. T. Sgm: A framework for building apecification-guidee moderation filters. arXiv preprint, 2025.
Casper, S., Schulze, I.., Patel, O., and Hadfield-Menell D. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030 2024.
Gan, Z., Chen, Y., Li, L., Zhu, C., Cheng, Y., and Liu, J. Large-scale adversarial training for iision-and-languaa representation learning. In NeurlPS, 2020.
Gilmer, J., Metz, L., Faghri, F. Schoenholz, S. S., Raghu, M., Wattenberg, M., and Goodfellow, Adversarial spheres. In ICLR, 2018.
Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Suajardo-Cessedess M., Yuan, S., Tar, C., Strope, B., and Kurzweil, R. Universal sen- tence encoder for English. In EMNLP, 2018.
Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense infor- mation retrieval with contrastive learning. TMLR, 2022.
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, J., and Wong, E. Jailbreaking black box large language mod- els in twenty queries. arXiv preprint arXiv:2310.0841 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S., Casas, d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7b. arXiv preprint arXiv:2310.06822 2023.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I. and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversar- ial nets. In NeurlPS, 2016.
Jin, D., Jin, Z., Zhou, J. T.,, and Szolovits, P. Is bert really robust? strong baseline for natural language attack on text classification and entailment. AAAI, 2020a.
Cheng, M., Lei, Q. Chen, P., Dhillon, I. S., and Hsich, C. CAT: customized adversarial training for improved robustness. IJCAL, pp. 673-679, 2022.
Jin, D., Jin, Z., Zhou, J. T., and Szolovits, P. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In AAAI, 2020b.
Dalvi, F., Sajjad, H., and Durrani, N. NeuroX library for neuron analysis of deep NLP models. ACI, 2023.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, 2009.
Devlin, J.. Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In ACL, 2019.
Kumar, A., Agarwal, C., Srinivas, S., Li, A. Feizi, S., and Lakkaraju, H. Certifying Ilm safety against adversarial prompting. ArXiv, 2023.
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by seal- ing high-quality instructional conversations. In EMNLP, 2023.
Kurakin, A., Goodfellow, J., and Bengio, S. Adversarial examples in the physical world. In ICLR, 2017.
Kwiatkowski, T., Palomaki, J., Redfield, o., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research. In ACI., 2019.
Dong, X., Luu, A. T., Ji, R., and Liu, H. Towards robustness against natural language word substitutions. In ICLR, 2021.
10

---

SMAAT
Li, L., Ma, R., Guo, Q. Xue, X., and Qiu, X. BERT- ATTACK: Adversarial attack against BERT using BERT. In EMNLP, 2020.
Ren, S., Deng, Y., He, K., and Che, W. Generating nat- ural language adversarial examples through probability weighted word saliency. In ACL, 2019.
Li, Y., Min, M. R., Lee. T., Yu, W., Kruus, E., Wang, W., and Hsieh, C.-J. Towards robustness of deep neural networks via regularization. ICCV, 2021a.
Samangouei, P., Kabkab, M., and Chellappa, R. Defense- GAN: Protecting classifiers against adversarial attacks using generative models. In ICLR, 2018.
Li, Z., Xu, J., Zeng, J., Li, L., Zheng, X., Zhang. Q. Chang, K.-W., and Hsieh, C.-J. Searching for an effective de- fender: Benchmarking defense against adversarial word substitution. In EMNLP, 2021b.
Schott, L.,, Rauber, J., Bethge, M., and Brendel, W. Towards the first adversarially robust neural network model on MNIST. In ICLR, 2019.
Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! NeurlPS, 32, 2019.
Liu, X., Xu, N., Chen, M., and Xiao, C. AutoDAN: Generat- ing stealthy jailbreak prompts on aligned large language models. In ICLR, 2024.
Shamir, A., Melamed, O., and BenShmuel, O. The dim- pled manifold model of adversarial examples in machine learning. arXiv preprint arXiv:2106.00152 2021.
Liu, Y., Ott, M., Goyal, N., Du. J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Heb- bar, V., Sleight, H., Stickland, A. C., Perez, E., Hadfield- Menell, D., al. Latent adversarial training improves robustness persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15554 2024.
Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S., Schoenebeck, G., Houle, M. E., Song, D., and Bailey, J. Characterizing adversarial subspaces using local intrinsic dimensionality. In JCLR, 2018.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting Knowledge from Lan- guage Models with Automatically Generated Prompts. In EMNLP, 2020.
Maas, I.., Daly, R. F., Pham, P. T., Huang, D., Ng, Y., and Potts, Learning word vectors for sentiment analy- sis. In ACL, 2011.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale scaee image recognition. arXiv preprint arXiv: 140l:::440..1550 1556, 2014.
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhace, E., Li, N., Basart, S., Li, B., Forsyth, D. A., and Hendrycks, D. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML, 2024.
Song. Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In ICLR, 2018.
Stewart, G. W. On the early history of the singular value decomposition. SIAM review, 1993.
Meng, D. and Chen, H. Magnet: two-pronged defense against adversarial examples. In ACM CCS, 2017.
Stutz, D., Hein, M., and Schiele, B. Disentangling adversar- ial robustness and generalization. CVPR, 2019a.
Minh, D. N. and Luu, A. T. Textual manifold-based de- fense against natural language adversarial examples. In EMNLP, 2022.
Stutz, D., Hein, M., and Schiele, B. Disentangling ad- versarial robustness and generalization. In ICCV, pp. 6976-6987, 2019b.
Morris, Lifland, E., Yoo, Y., Grigsby, Jin, D., and Qi, Y. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In 2020.
Tanay, T. and Griffin, L. A boundary tilting persepective the phenomenon of adversarial examples. arXiv preprint arXiv: 1608. 076900 2016.
Mrkšić, N., Séaghdha, D., Thomson, B., Gašić, M., Rojas- Barahona, L. M., Su, P.-H., Vandyke, D., Wen, T.-H., and Young, Counter Counter-fitting word vectors to linguistic constraints. In ACI. 2016.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention. In ICML, pp. 10347-10357, 2021.

---

SMAAT
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Bahaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019b.
Zhang, X., Zhao, J., and LeCun, Y. Character-level con- volutional networks for text classification. In NeurIPS, volume 28, 2015.
Tramèr, F., Kurakin, A., Papernot. N., Goodfellow, J., et al. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, 7., Li, 7.,, Li, D., Xing, E., et al. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, volume 36, 2024.
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. In ICLR, 2019.
Zhong, Z., Huang, Z., Wettig, A., and Chen, D. Poisoning retrieval corpora by injecting adversarial passages. In EMNLP, 2023.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analy- sis platform for natural language understanding. In ICLR, 2019a.
Zhu, C., Cheng, Y.. Gan, Z., Sun, S., Goldstein, T.. and Liu, J. Freelb: Enhanced adversarial training for natural language understanding. In ICLR, 2020.
Wang, B., Xu, C., Wang, S., Gan, Z.,, Cheng, Y., Gao, J., Awadallah, A. H., and Li, B. Adversarial GLUE: A multi- task benchmark for robustness evaluation of language models. In NeurIPS, 2021.
Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Uni- versal and transferable adversarial attacks on aligned lan- guage models. arXiv preprint arXiv:2307. 15043, 2023.
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu, Q. Improving adversarial robustness requires revisiting mis- classified examples. In ICLR, 2019b.
Wang, Z., Dong, Y., Zeng, J., Adams,
V., Sreedhar, M. N., Egert, D., Delalleau, O., Scowcroft, J. P., Kant, N., Swope, A., et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528 2023.
Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? In NeurIPS, 2023.
Wong, E., Rice, L., and Kolter, Z. Fast better than free: Revisiting adversarial training. In ICLR, 2020.
Xiao, C., Li, B., Zhu, J.-Y., He, W., Liu, M., and Song, D. Generating adversarial examples with adversarial networks. In IJCAI, 2018.
Xiao, J., Yang, L.,, Fan, Y., Wang, J., and Luo, Z.-Q. Un- derstanding adversarial robustness against on-manifold adversarial examples. Pallern Recognition, 2025.
Ye, M., Gong, C., and Liu, Q SAFER: A structure-free approach for certified robustness to adversarial word sub- stitutions. In ACL, 2020.
Zang. Y., Qi, F., Yang, C., Liu, Z., Zhang, M., Liu, Q., and Sun, M. Word-level textual adversarial attacking as combinatorial optimization. In ACL, 2020.
Zhang, D., Zhang. T., Lu, Y., Zhu, 2., and Dong, B. You only propagate once: Accelerating adversarial training via maximal principle. In NeurlPS, 2019a.
12

---

SMAAT
A. Supplementary Material
In this paper, we propose SMAAT, an AT algorithm that not only optimizes for better robustness and generalization also enhances scalability. Specifically, SMAAT leverages the manifold conjecture, which posits that OFM-AEs lead to better robustness while ONM-AEs enhance generalization. To achieve this, SMAAT perturbs the layer with the lowest intrinsic dimension (ID). Intuitively, perturbing this layer would yield the highest proportion of OFM-AEs across layers. Formally, SMAAT solves the following program:
(Jor (Juuec + 61 b
min (E.))~D max
We show that 1* corresponds to the last layer in enc-LLMs and the first layer in dec-I.LMs. This explains the difference in robustness and generalization trends between vision models and enc-LLMs (Stutz et al., 2019a; Shafahi et al., 2019; Zhu et al., 2020). Specifically, in vision models, improvements robustness are often accompanied by a decrease in generalization. contrast, in enc-LLMs, improvements in generalization are achieved with lesser gains in robustness. The algorithm for SMAAT is provided in Alg. 1.
Algorithm SMAAT
1: Input: D = {X, J: input data, fe: a deepnet model, e: attack strength, E: the number of epochs, a: PGD learning rate, and II: the projection operator into the e-ball.
2: Output fo: deepnet model
% Determine the ID behaviour of the model with twoNN
4: % Identify the optimal layer l' to perturb (Eq. (7))
5: for t==1 1,,,, E do
6: for (I;, do
7: 8t- ~ (0(0,001) % sample an initial perturbation
8: store forward pass for scalability
9: mid_rep + (x.)
for s 1,.., do
11:
loss l (Jaz ni (mid_rep + 8z- y:])) 8,- +- 11,-- (dp. + sign (Vá,, v,- (loss))
12:
13:
end for
14:
% update the model parameters
15: loss <- f [Jor mi (mid rep + 51- 5,-), W:))
16: Oll m1 -1 ml Ir ou' .x) (n)sssllsss)
end for
18:
end for
SMAAT has the advantage being significantly more runtime-efficient than classical adversarial training. The gain is the order of O P(n rn) max (dr, 12, [1,7" 1 -1)) -d+)). for n-layered deepnet under P-step PGD attack. dy is the dimensionality of the layer
Empirical results demonstrate that SMAAT leads to better robustness while maintaining comparable accuracy to standard training. We achieve compelling results on robustifying (1) sentiment classifiers, (2) safety filters decoder-bacee based models, and (3) retriever models in the Retrieval Augmented Generation (RAG) setup.
The remaining of the supplementary material is organized as follows:
Appendix B: Intrinsic Dimension Estimation
Appendix C: Additional Results of ID & Robustess/GGeneralzzaton Relationship
Appendix D: Additional Results of Robustifying Sentiment Classifiers
13

---

SMAAT
Appendix E: Additional Results of Language Understanding Benchmarks
Appendix F: Additional Results of Robustifying Safety Filters In Decoder Based LLMs
B. Intrinsic Dimension Estimation
In the following, we provide additional details on the experimental setup on OFM-/ONM- AEs ratio calculation and the ID estimation in Fig. 4.
Experimental setup. We conduct experiments on two enc-LLMs (BERT and RoBERTa), across three different datasets (AGNEWS, IMDB, YELP). Additionally, we evaluated three vision models (ViT (Dosovitskiy et al., 2021), DeiT (Touvron al., 2021), VGG-13 (Simonyan & Zisserman, 2014)) on the CIFAR-10 (Krizhevsky, 2009) dataset and three LLMs (LLaMA-2-7b (Touvron al., 2023), Vicuna (Zheng al., 2024), Mistral-7b (Jiang et al., 2023)) on the HelpSteer dataset (Wang et al., 2023). We utilize the train split of the datasets to estimate the ID and eigenspace of the layers. We extract token representations from various layers of the models similar to the implementation in NeuroX (Dalvi et al., 2023). AEs are generated using TextFooler for LMs and PGD for vision models on the respective test splits of the datasets. For L.MMs, AEs are generated using the Greedy Coordinate Gradient (GCG) attack (Zou et al., 2023) on the harmful prompt datasets released in the same paper. In our calculation, we use CLS embedding for BERT, RoBERTa, ViT, and DeiT models; last token embedding for LLMs; and conventional layer output for VGG-13.
Discussion of results. We present the average projection error, eř, of each layer on the first row of Fig. 4. Results in Fig. 4(a.b) indicate that for LMs, the average ef monotonically increases, suggesting that examples become more off- manifold at the latest layers, consistent with our hypothesis. Conversely, for vision models and dec-LLMs in Fig. 4(c,d), we observe the opposite characteristic with a lower average e/ at the latest layers as expected. The only unexpected behavior observed in vision models and dec-LLMs, except for VGG-13 and Mistral, is that they exhibit the highest off-manifold ratio (ef) in the initial layers rather than the first layers. For transformer models, the CLS/last token embedding starts with a specific value and gradually evolves represent the sentence. This means it takes time for the effect of AEs to manifest in the CLS/last token embedding. In contrast, since the VGG model's representation is directly calculated from the input, the highest off-manifol ratio is obtained at the first layer.
We hypothesize that this phenomenon can be explained by the ID of the layers. Specifically, if I < dr, AEs tend to be off-manifold. To validate this hypothesis for the same models and dataset, we measure I using the twoNN method and normalize with dj. The normalized It can be seen in the second row of Fig. 4. In line with our hypothesis, while it decreases for the LMs (Fig. 4(a,b)) it increases for vision models and dec-LLMs(Fig. 4(c,d)).
C. Additional Results of ID & Robustaess/GGnneraliatiio Relationship
We further examine the relationship between intrinsic dimensionality (ID), robustness, and generalization by applying Latent Adversarial Training (LAT) to different layers of the ILLMM-2-7B model. Unlike untargeted LAT, which disrupts model behavior, we employ targeted LAT to induce specific adversarial responses, following Casper et al. (2024). This is achieved by perturbing the residual stream with 22-nomm-bundde noise using PGD.
To stabilize training and mitigate unintended effects, interleave LAT with supervised fine-tuning on the UltraChat dataset (Ding et al., 2023). We evaluate generalization using MT-Bench (Zheng al., 2024) and robustness using GCG-based and PAIR attacks from HarmBench (Mazeika et al., 2024). The model is trained with a learning rate of 2e 4, applying LAT at every even-numbered layer with norm bounds ranging from 1 to 5.
The results are presented in Figure 6. Attacking lower layers (light blue markers) improves robustness but reduces generalization, as indicated by points concentrated in the bottom-right region. In contrast, attacking upper layers (dark blue markers) enhances generalization at the cost of robustness, shifting results toward the top-left. Additionally, while model robustness drops to as low as 30% under the GCG attack, it remains consistently above 75% across all configurations under the PAIR attack.
D. Additional Results of Robustifying Sentiment Classifiers
In the following, we provide more details about the experiments on robustifying sentiment classifiers. Datasets. We evaluate SMAATon three datasets: AG-News Corpus (AGNEWS) (Zhang et al., 2015), Internet Movie
14

---

SMAAT
General Ization (MT-Bench) 6
Generalization (MT-Bench)
Layer
Layer
16
10
20
22
24
10
26
28
14
30
0.3
0.4
0.5
0.6
0.7
0.80
0.85
0.90
0.95
1.00
Robustness under GCC
Robustness
(a) GCG
(b) PAIR
Figure 6. Layer-wise effects of adversarial training on generalization and robustness against GCG and PAIR attacks on the LLaMA-2 model. Marker colors represent layer depth, transitioning from light blue (lower layers) to dark blue (higher layers). The results show that robustness drops to as low as 30% under the GCG attack, while remaining above 75% under the PAIR attack across configurations.
Database (IMDB) (Maas et al., 2011), and Yelp Review Polarity (YELP) (Zhang et al., 2015). The AGNEWS dataset contains over 120000 samples, each belonging to one of the four labels: World, Sports, Business, Sci/Tech. The IMDB dataset contains 50000 data samples of movie reviews with binary labels for negative and positive sentiments. The YELP dataset contains nearly 600000 samples of highly polar Yelp reviews with binary labels. However, due to limitations in computing resources, we only use a subset of 63000 samples of the YELP dataset. In addition, we randomly sample 10% of the training set for validation in all datasets. For testing, we use subset of 1000 test samples from each dataset, following previous work practices. The AGNEWS dataset contains over 120k samples, categorized into four classes: World, Sports, Business, and Sci/Tech. The IMDB dataset consists of 50k movie reviews, each labeled with binary sentiments (positive or negative).
Base model. We employed the BERT base-cased (Devlin et al., 2019) and RoBERT RoBERTamaseco cased (Liu et al., 2019) models in our experiments. To conduct the evaluations, we utilize the fine-tuned models provided by TextArtack from HuggingFace for all datasets, except for the RoBERTa base model fine-tuned on YELP dataset. For the YELP dataset, we created a fine-tuned RoBERTa model for epochs with a learning rate of le 05 and batch size of 32.
Adversarial Attacks. which include the following constraints: (1) The maximum percentage of modified words is set to 0.3 for AGNEWS, 0.1 for IMDB and YELP datasets, respectively. (2) For word replacement, a maximum of 50 candidates are considered for each word. (3) The semantic similarity, measured using the Universal Sentence Encoder (Cer et al., 2018), between the original input and the generated adversarial example must exceed 0.84. PWWS uses word synonyms, TextFooler applies nearest neighbor search in counter-fittie fitting embeddings (Mrkšić et al., 2016), and BERT-Attack utilizes BERT masked language model generate candidate words.
Baselines. For input space adversarial training, consider Adversarial Sparse Convex Combination (ASCC) (Dong et al., 2021) which model the perturbation space the convex hull of word synonyms. ASCC incorporates entropy- bassd sparsity regularizer to capture word substitution geometry more effectively. In our investigation of embedding space adversarial training which recognized as the most impactful technique for enhancing generalization (Li et al., 202 we conduct thorough analysis of FreeI.B++ (L.i et al., 2021b) (employs gradient-guided perturbations centered around the most susceptible data points). For certified defenses, we evaluate SAFER (Ye et al., 2020), TMD (Minh & Luu, 2022), and RSMI (Minh& Luu, 2022). SAFER constructs set of randomized inputs by performing random synonym substitutions and using the statistical properties predicted labels to certify robustness. TMD employs infoGAN (Chen et al., 2016) to project adversarial examples the data manifold in the last layer to address the manifold issue. RMSI combines these ideas by applying importance-based masking to tokens and leveraging randomized smoothing in each layer.
Implementation details. To train the last layer of fa with adversarial samples, we create adversarial samples using 5-step PGD attacks. During training, we use epsilon values of 0.1, 0.1, and 0.8 for the YELP, AGNEWS, and IMDB datasets, respectively, for the BERT models. For the RoBERTa models, we employ epsilon values of 0.1, 0.6, and 0.03. All models are trained 10 epochs with learning rate of 0.1. our evaluation, we use a V100 GPU with 32 GB memory and 64 CPUs.
15

---

SMAAT
Table Average accuracy from the standard training, FreeI.B++ and SMAAT on GLUE and AdvGL.UE datasets. Results elearly
demonstrate that SMART enhances model generalization (GLUE results) and robustness (AdvGLUE results).
BERT
RoBERTa
Dataset
Standard
FreeLB+++
SMAAT (Ours)
Standard
FreeLB+++
SMAAT (Ours)
GLUE
85.9
86.3
86.3
89.3
89,6
89.7
AdvGLUE
39.5
42.5
45.1
27.5
37.1
39.6
Table 6. Hyperparameters in experiments robustifying the safety filters on decoding LLMs experiments. Learning rate (Ir) and e values for SMAAT and FreeLBB+++ methods.
FreeLB++ RoBERTa
SMAAT RoBERTa
Dataset
BERT
DisT Ir
BERT
DistilBERT
Ir
c
br
e
Ir
c
Ir
c
Ir
AdvBench
5e-3
0.05
5e-3
0.05
le-2
0.01
le-4
le-3
Ic-4
le-3
le-4
le-4
HH-RLHF
5e-3
0.5
53-3
0.05
5e-4
0.05
le-4
5e-4
le-4
le-3
le-4
5e-4
E. Additional Results of Language Understanding Benchmarks
To comprchensively evaluate SMAAT's performance against a broader spectrum of textual adversarial attacks, we employ the GLUE and AdvGLUE benchmarks. The GLUE benchmark (Wang al., 2019a) is a comprehensive evaluation suite featuring seven diverse NLP tasks to assess model performance. The AdvGLUE benchmark (Wang et al., 2021) is an extension of GLUE, incorporating 17 distinct textual adversarial attacks, covering werd-level transformations, sentence-level manipulations, and human- AEs. This extension ensures a thorough evaluation encompassing various adversarial linguistic phenomena. For our assessment, we employ the evaluation sets of four datasets across three different tasks: Sentiment Analysis (SST-2), Duplicate Question Detection (QQP), and Natural Language Inference (QNLI, RTE).
In our evaluation, we compare SMAAT against standard BERT and RoBERTa models", as well as their FreeLB++ incorporated versions. In the case of SMAAT, we conducted grid search for the learning rate, ranging from 0.1 to 0.001, and the value, ranging from 0.8 to 0.01, using 3-PGD steps As shown in Table 5, SMAAT demonstrates robustness improvement of 5.6% and 2.6% for BERT, and 12.1% and 2.5% for RoBERTa, compared to the standard and FreeLB++ models, respectively, while maintaining similar generalization performance.
F. Additional Results of Robustifying Safety Filters In Decoder Based LLMs
The GCG attack deliberately designed to bypass the safety alignment of LLMs by generating a response to a potentially harmful prompt through appending an adversarial suffix to the user's input. This strategy leverages previous methodologies by using (1) affirmative response tactic (Wei et al., 2023; Carlini et al., 2023) to direct the model's output towards the attacker's 's intended outcome (used for loss calculation) and (2) mix of greedy and gradient-based discrete optimization (Shin et al., 2020) to pinpoint the most susceptible tokens. A key attribute of the GCG attack is its transferability. demonstrating that adversarial suffix designed for specific prompt on one model can successfully affect a broad range of other models.
A suggested defense against the GCG attack involves incorporating a lightweight binary classifier model designed to identify harmful prompts (Kumar et al., 2023). It important to note, however, that these classifiers can still be vulnerable to such attacks. To assess the effectiveness of SMAAT against the GCG attack, we conduct the attack by setting the suffix length to 20 tokens. The attack is generated over 50 iterations, with 100 trials per iteration. Rather than leveraging the attack's transfcrability, we tailor an adversarial suffix for every individual prompt and model enhance the attack's 's impact. Our assessment is conducted using two datasets, AdvBench (Zou et al., 2023) and HH-RLHF (Bai et al., 2022), and involves three models: BERT, RoBERTa, and DistilBERT. Adv Bench features 640 training prompts (320 harmful, 320 safe) and 240 test prompts (120 harmful, 120 safe), whereas HH-RLHE includes approximately 44K harmful and 44K helpful training prompts, with 2.3K test samples for each prompt category. In all cases, standard models are trained over 5 epochs with a leamning rate of 1e-5, 5 Table 6 details the training hyperparameters for SMAAT.
'e use the fine-tuned models available from http:///tuggnnffeee.ooeeeeeeall
16


Reviews:
[{"review_id": "dsYK3MOf3k", "reviewer": "Reviewer_fFMr", "summary": "The authors investigate how the relationship between perturbations and the data manifold influences whether adversarial training leads to improved generalization or robustness. Based on this insight, they propose SMAAT, a method that generates perturbations at specific layers to target different manifolds—leveraging the fact that intrinsic dimensionality changes with layer depth—to achieve more precise trade-offs between generalization and robustness. The paper provides extensive quantitative results on LLMs supporting these claims, demonstrating that SMAAT is more efficient and achieves a better generalization-robustness tradeoff compared to existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and suggestions. In response, we have conducted additional experiments to address the critiques and provide a summary of further revisions that will be made to the paper.\n\n**Details on Fig. 1 and Fig. 2:**\n> Figure 1 and Figure 2 lack sufficient experimental details ...\n\nThese two figures summarize key findings presented in Sec. 4 and Sec.  5. \n\n\nFigure 1 presents results on the LLaMA-2 model by combining the ID characteristics from Fig. 4(d) (1st row), reconstruction error from Fig. 4(d) (2nd row), and the generalization and robustness trends from Fig. 5. For Fig. 1, we report the best results that maximize the sum of robustness and generalization scores for each layer. Additional details are provided in the supplementary material.\n\n\nFigure 2 compares  SMAAT against other  AT approaches for  encoder-based models, evaluating generalization, robustness, and run-time cost across different tasks.\n\n\nWe will revise the Introduction to provide further details and to reference the Supplementary for full details on the generation of Figs. 1 and 2.\n\n\n**Additional attacks:**  \n> Evaluating accuracy under attack against more recent and stronger adversarial attacks (e.g., [2,3]) would better demonstrate the method’s robustness improvements.\n\nAlthough we acknowledge that the suffix attack (the method used in Table 3) is not a recent technique, it remains one of the most powerful attack strategies. For the sake of clarity, we conducted new experiments using the PAIR attack, in addition to the suffix attack. Since Table 3 reports results for encoder-based models—which are not compatible with attacks that rely on generative capabilities like PAIR—we instead applied this attack to the LLaMA-2 model. We performed the robustness versus generalization experiments shown in Fig. 4 under this attack scenario. The corresponding results, for comparison with Fig. 4, can be viewed at [link](https://anonymous.4open.science/r/SMAAT-25-DD9F/ICML%20Rebuttal/llama2_gen_robust_pair.pdf). As seen in the results, we observe the same trend as with the GCG (i.e., the suffix attack) attack. However, while model robustness drops to **30%** under the GCG attack, it remains above **75%** across all setups for the PAIR attack. This further supports our decision to use the GCG attack over PAIR in our experiments.\n\n\n**Experiments on vision models:** \n\n> Q1. While I understand that evaluating vision model robustness requires significantly more experimental setup than adding another text-based experiment, is there a specific reason why no experiments were conducted on vision models using the proposed method?\n\n> While Figure 4 explores intrinsic dimensionality and reconstruction error for vision models, the paper does not provide robustness-generalization results for vision models using the proposed method.\n\nWe extend the robustness and generalization analyses of the encoder and decoder models to vision models. Following the approach in Figs. 5 and 6—where one model from each architecture was analyzed—we conducted similar experiments on VGGNet, chosen for its relative computational efficiency. It is important to note that performing a full grid search with adversarial training (AT) across all layers and measuring robustness for each model is highly resource-intensive. \nSpecifically, we performed  **AT** on VGGNet using the CIFAR-10 dataset by attacking every ReLU layer over 20 epochs, with **ε = 0.031–0.2** and **lr = 0.01–0.001**, evaluating robustness via **RobustBench**. [Results](https://anonymous.4open.science/r/SMAAT-25-DD9F/ICML%20Rebuttal/vgg_curve.pdf) show that attacking lower layers (light blue) improves robustness but reduces generalization (**bottom-right**), while upper layers (dark blue) enhance generalization at the cost of robustness (**top-left**). This shows that vision models more closely resemble decoder models in terms of their generalization versus robustness characteristics.\n\n\n**Abbreviations:** \n>  define terms when first used and in important locations (e.g. RAG in abstract, LAT in figure 1)\n\nWe will revise the text to define terms where they appear first.\n\n\n**Improving clarity of Fig. 1:** \n\n> figure 1: could improve clarity by just plotting points and adding best-fit lines to show trends more clearly\n\nThanks. We have regenerated Fig. 1 to display best-fit lines using a locally weighted regression technique. The updated figure is available at the [link](https://anonymous.4open.science/r/SMAAT-25-DD9F/ICML%20Rebuttal/llama_all_in_one_trend.pdf).\n\n\n**Code:** \n\n> Q2. Do the authors plan on releasing the code for their experiments?\n\nThe code is publicly available and can be accessed at the [link](https://anonymous.4open.science/r/SMAAT-25-DD9F/README.md)", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Figure 1 and Figure 2 lack sufficient experimental details. It would be helpful to explicitly clarify these details in the experiments section or provide additional explanations in the supplementary material.", "relation_to_broader_scientific_literature": "This work builds on previous research on adversarial training, particularly the accuracy-robustness tradeoff in image models [1], and extends these insights to generalization properties and different architectures. The empirical observation of a strong relationship between the data manifold and the robustness-generalization tradeoff is a valuable contribution, as it provides a deeper understanding that can inform more refined adversarial training strategies.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n- Clear writing with tightly integrated experiments, making the claims and findings easy to follow.\n- Strong and wide-ranging experiments, demonstrating both efficiency and a well-balanced generalization-robustness tradeoff.\n- Relatively simple yet effective method that directly leverages the observed manifold-robustness-generalization relationship, reinforcing its validity and practical significance.\n\nWeaknesses:\n- Weak/easy adversarial attack used in Table 3: Evaluating accuracy under attack against more recent and stronger adversarial attacks (e.g., [2,3]) would better demonstrate the method’s robustness improvements.\n- Limited experiments on vision models: While Figure 4 explores intrinsic dimensionality and reconstruction error for vision models, the paper does not provide robustness-generalization results for vision models using the proposed method. Since Figure 3 suggests the method can enhance vision model robustness, a direct comparison on vision benchmarks would strengthen the findings.\n\nreferences:\n\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International conference on machine learning. PMLR, 2019.\n\n[2] Liu, Xiaogeng, et al. \"Autodan: Generating stealthy jailbreak prompts on aligned large language models.\" arXiv preprint arXiv:2310.04451 (2023).\n\n[3] Chao, Patrick, et al. \"Jailbreaking black box large language models in twenty queries.\" arXiv preprint arXiv:2310.08419 (2023).", "other_comments_or_suggestions": "- define terms when first used and in important locations (e.g. RAG in abstract, LAT in figure 1)\n- figure 1: could improve clarity by just plotting points and adding best-fit lines to show trends more clearly", "questions_for_authors": "Q1. While I understand that evaluating vision model robustness requires significantly more experimental setup than adding another text-based experiment, is there a specific reason why no experiments were conducted on vision models using the proposed method?\n\nQ2. Do the authors plan on releasing the code for their experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMsZIMkHSs", "reviewer": "Reviewer_aVnu", "summary": "This paper reveals the fundamental reasons behind the varying effectiveness of adversarial training across different types of neural networks and proposes a novel and efficient training method, SMAAT. The study finds that early layers of vision models (e.g., CNNs) and generative language models (e.g., LLaMA) exhibit low intrinsic dimensionality, making them prone to generating adversarial examples that deviate from the true data distribution (off-manifold samples). This results in adversarial training significantly improving robustness at the cost of generalization. Conversely, in encoder-based language models (e.g., BERT), later layers exhibit low intrinsic dimensionality, causing traditional adversarial training to generate samples closer to the real data distribution (on-manifold samples), preserving generalization but limiting robustness improvements.\nBased on this discovery, the authors propose the SMAAT framework, which dynamically selects the network layer with the lowest intrinsic dimensionality for perturbation—applying it to the final layer for encoder models and to the input layer for vision/generative models. \nExperiments demonstrate that SMAAT outperforms existing techniques in various applications, including sentiment analysis, content safety filtering, and retrieval-augmented systems.  This study not only provides the first explanation of adversarial training discrepancies from a data distribution perspective but also introduces a new training paradigm that balances efficiency and security. The proposed method is particularly valuable for the rapid deployment of adversarially robust large language models in real-world applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and suggestions aimed at improving the clarity of the paper. Below are our responses to the reviewer’s points. \n\n\n**Theoretical proofs:** \n\n> While the theoretical claims are strongly supported by systematic experimental design—spanning different models, tasks, and attack scenarios—the paper lacks rigorous mathematical proof.\n\nThe paper relies on two statements:  (1) AEs generated from a low-dimensional manifold are likely to be off-manifold (similarly, AEs generated from a high-dimensional manifold are likely to be on-manifold) and (2) the manifold conjecture stating that off/on-manifold AEs lead to better robustness/generalization. While the manifold conjecture is well established in the literature (Ethayarajh, 2019; Shamir et al., 2021; Gilmer et al., 2018), statement (2) is straightforward:  Let $\\mathcal{M} \\subset \\mathbb{R}^n$ be a smooth, compact, low-dimensional manifold with intrinsic dimension $d \\ll n$, embedded in $\\mathbb{R}^n$. Let $f: \\mathbb{R}^n \\to \\mathbb{R}^k$ be a classifier trained on data sampled from $\\mathcal{M}$. Let $\\delta$ denote an adversarial perturbation obtained by maximizing the loss $\\mathcal{L}(f(x + \\delta), y)$ under a norm constraint $\\|\\delta\\| \\leq \\epsilon$. Since the classifier models $p(y \\mid x)$ without access to the generative distribution $p(x)$, it lacks explicit knowledge of the manifold $\\mathcal{M}$. Therefore, the loss gradient $\\nabla_{\\delta} \\mathcal{L}$ generally has components orthogonal to the tangent space. As a result, the perturbation $\\delta$ is unlikely to lie entirely within $\\mathcal{M}$, and typically has a non-zero orthogonal component. Moreover, the smaller the dimension $d$ of the manifold relative to $n$, the larger the proportion of $\\delta$ that lies off-manifold. Hence, adversarial examples generated in this way are very likely to lie off the data manifold. We will add this explanation to the paper. We hope this clarifies.\n     \n**Clarity of the introduction:** \n\n> The introduction of the paper is not simple and straightforward enough, and some basic concepts are explained in an overly complicated manner.\n\nThank you for your comment. Below, we clarify the information presented in the introduction. Moreover, we are happy to accommodate any specific suggestions you may have to improve it further.\nCurrently, our Introduction is structured as follows: (A) We begin by introducing adversarial training (AT) and highlighting two major limitations that our work addresses: (1) the poorly understood tradeoff between robustness and generalization (L.51–52, Col.1), and (2) the high computational cost of AT, which limits its practical deployment (L.51–52, Col.2). (B) To address the first limitation, we investigate how encoder LLMs, vision models, and decoder LLMs differ in intrinsic dimensionality, leading to distinct compositions of on-/off-manifold adversarial examples. This, under the manifold conjecture, explains their varying impacts on robustness and generalization. (C) To address the second limitation, we propose a scalable Manifold-Aware Adversarial Training approach that selectively applies AT at the layer with the highest proportion of off-manifold AEs, significantly reducing cost without sacrificing performance.\n (D) Finally, we summarize our experimental results supporting both contributions. We hope this clarifies the structure and motivation of the introduction. \n\n**Usage of the term manifold**\n\n> I am unsure about the specific meaning of “data manifold.” If “on-manifold” refers to high-density and “off-manifold” refers to low-density, it may not be necessary to use the term “manifold,” as it could confuse the readers.\n\nWe formally define the ‘data manifold’ and ‘on-/off-manifold samples’ in Section 3. To improve clarity, we have revised the introduction to introduce these definitions earlier. Specifically, we use the term data manifold to refer to a potential non-linear subspace spanned by the dataset as it propagates through the network layers, with its dimension quantified via a projection-based method.  In this context, on-/off-manifold samples refer to data points that are either captured by or fall outside the learned manifold during training.\n\n\n**Adversarial attacks in the text domain**\n\n> Adversarial attacks in the visual domain are well known, and it is worth going into more detail about adversarial attacks in the text domain.\n\nWe will revise the introduction and related work to better describe adversarial attacks on text, with an emphasis on most effective ones such as GCG attack, i.e., the suffix attack, and PAIR attack.", "claims_and_evidence": "I think that the claims presented in this paper are reasonable to some extent and are supported by experimental evidence.", "methods_and_evaluation_criteria": "The paper proposes the SMAAT method, which selects adversarial training perturbation layers by analyzing the intrinsic dimensionality (ID) of different model layers to enhance robustness and training efficiency. The study covers vision models, encoder-based language models (such as BERT), and decoder-based language models (such as LLaMA) and evaluates the approach across multiple tasks, including text classification, safety filtering, and RAG retrieval.\nOverall, the proposed method directly addresses the core challenges of adversarial training in encoder-based models with a well-structured and rigorous evaluation framework. While there are limitations in its applicability to specific scenarios, the approach offers a valuable paradigm for efficient and robust model training.", "theoretical_claims": "The paper primarily presents two theoretical insights: (1) the relationship between intrinsic dimensionality (ID) and the manifold properties of adversarial samples (ONM/OFM); and (2) the effectiveness of the SMAAT method in enhancing robustness and efficiency by perturbing low-ID layers. While the theoretical claims are strongly supported by systematic experimental design—spanning different models, tasks, and attack scenarios—the paper lacks rigorous mathematical proof.", "experimental_designs_or_analyses": "The paper provides strong evidence for the effectiveness of SMAAT through cross-validation across multiple tasks and attack scenarios. The experimental design generally aligns with field standards, but additional details are needed to further enhance the credibility of the conclusions.", "relation_to_broader_scientific_literature": "This paper integrates the manifold hypothesis, intrinsic dimensionality (ID) analysis, and efficient training methods to construct a theoretically coherent and widely applicable adversarial training framework. Its core innovation lies in revealing the decisive role of layer-wise ID distribution in determining adversarial sample properties and, based on this insight, proposing a scalable perturbation strategy.", "essential_references_not_discussed": "There are relatively relevant citations.", "other_strengths_and_weaknesses": "The introduction of the paper is not simple and straightforward enough, and some basic concepts are explained in an overly complicated manner.", "other_comments_or_suggestions": "Adversarial attacks in the visual domain are well known, and it is worth going into more detail about adversarial attacks in the text domain.", "questions_for_authors": "I am unsure about the specific meaning of “data manifold.” If “on-manifold” refers to high-density and “off-manifold” refers to low-density, it may not be necessary to use the term “manifold,” as it could confuse the readers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVxvxC9cYK", "reviewer": "Reviewer_JtkR", "summary": "This paper investigates the intrinsic dimensionality in the layerwise fashion for adversarially trained models. This paper provides a new perspective for adversarial training in different model architectures from manifold conjecture. The off-manifold adversarial examples (AEs) enhance robustness, and the on-manifold AE improves generalization. From an architectural perspective, decoder-based LLM and commonly used vision models exhibit different characteristics from encoder-based LLM. The vision and decoder-based LLMs exhibit low intrinsic dimensionality in earlier layers (favouring off-manifold AEs), whereas encoder-based models do so in later layers (favouring on-manifold AEs). Based on this property, this paper introduced SMAAT: Scalable Manifold Aware Adversarial Training. Experiments demonstrated its effectiveness and efficiency.\n\n---\n\n## update after rebuttal\n\nAfter the rebuttal, I have not changed my rating. The authors have addressed my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments and suggestions. In response, we conducted additional experiments to address the raised concerns and outlined the further revisions planned for the paper.\n\n**Evaluation on vision models:**\n\n> Given that SMAAT is presented as a generalized framework, a more comprehensive evaluation, especially on vision models, would significantly strengthen its contribution.\n\n>  It would be beneficial to add evaluation with vision models on typical benchmarks like RobustBench [1]. \n\n\nAs requested by the reviewer,  we have extended the robustness and generalization analyses of encoder and decoder models to vision models.  Following the approach in Figs. 5 and 6—where one model from each architecture was analyzed—we conducted similar experiments on VGGNet, chosen for its relative computational efficiency. It is important to note that performing a full grid search with adversarial training (AT) across all layers and measuring robustness for each model is highly resource-intensive. \n Specifically, we performed **AT** on VGGNet using the CIFAR-10 dataset by attacking every ReLU layer over 20 epochs, with **ε = 0.031–0.2** and **lr = 0.01–0.001**, evaluating robustness via **RobustBench**. [Results](https://anonymous.4open.science/r/SMAAT-25-DD9F/ICML%20Rebuttal/vgg_curve.pdf) show that attacking lower layers (light blue) improves robustness but reduces generalization (**bottom-right**), while upper layers (dark blue) enhance generalization at the cost of robustness (**top-left**). This shows that vision models more closely resemble decoder models in terms of their generalization versus robustness characteristics.\n\n**New experiments on decoder-based models:** \n\n> Additionally, the current evaluation only focused on encoder-based LLMs; it would be more comprehensive to include decoder-based LLMs as well.\n\nWe conducted new experiments using the PAIR attack on the Llama-2 model.  We performed the robustness versus generalization experiments shown in Fig. 4 under this attack scenario. The corresponding results, for comparison with Fig. 4, can be viewed at [link](https://anonymous.4open.science/r/SMAAT-25-DD9F/ICML%20Rebuttal/llama2_gen_robust_pair.pdf). As seen in the results, we observe the same trend as with the GCG attack. However, while model robustness drops to **30%** under the GCG attack, it remains above **75%** across all setups for the PAIR attack. This further supports our decision to use the GCG attack (i.e., the suffix attack) over PAIR attack in our experiments.\n\n**Efficiency gain assessment:**\n\n>  I suggest discussing the broader benefits of improving adversarial training efficiency, particularly for encoder-based LLMs,\n\nOur analysis of the three most widely used architectures—encoder, decoder, and vision models—reveals a key distinction: encoder-based models uniquely exhibit a decreasing intrinsic dimensionality trend across layers, in contrast to the increasing trend we observed in the other two architectures. The core idea behind our method SMAAT is to apply adversarial training (AT) at the layer with the highest proportion of off-manifold samples to maximize robustness. For encoder models, this corresponds to the last layer, while for decoder and vision models, it aligns with the first layer, effectively making SMAAT equivalent to conventional AT in those cases. In this context, our analysis offers the first explanation for why traditional AT has proven especially effective.\n\n**Emphasizing AT efficiency:**  \n\n> I suggest discussing the broader benefits of improving adversarial training efficiency, particularly for encoder-based LLMs\n\nThank you for this note. Our work fundamentally explores the relationship between layer-wise intrinsic dimensionality (ID) and its effect on the generalization–robustness trade-off.  The proposed SMAAT method leverages these ID-related insights to guide adversarial training. Notably, SMAAT leads to significant improvements for encoder-based models, as highlighted in Fig. 2 of the Introduction. Furthermore, as shown in Table 2, SMAAT introduces no additional overhead beyond standard model training. We will further revise the text to better highlight this aspect.", "claims_and_evidence": "Claims made in the submission are supported by convincing evidence.", "methods_and_evaluation_criteria": "Proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "No new theoretical claims.", "experimental_designs_or_analyses": "The existing experimental designs are technically sound. However, given that SMAAT is more of a generalizable AT framework. It would be beneficial to add evaluation with vision models on typical benchmarks like RobustBench [1]. Additionally, the current evaluation only focused on encoder-based LLMs; it would be more comprehensive to include decoder-based LLMs as well. \n\n[1] Croce, Francesco, et al. \"RobustBench: a standardized adversarial robustness benchmark.\" Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).", "relation_to_broader_scientific_literature": "The findings of this paper are very interesting and could have the potential for a larger impact on broader scientific literature.", "essential_references_not_discussed": "Most of the related works are discussed.", "other_strengths_and_weaknesses": "Strengths\n- The paper provides valuable insights by exploring intrinsic dimensionality in a layer-wise manner across various model architectures, highlighting the important distinctions between off-manifold and on-manifold AEs. These insights could have broad implications and significantly influence future research.\n- Reducing the training overhead is a crucial contribution to adversarial training, particularly given the growing preference for larger, more computationally intensive models.\n\nWeaknesses\n- The evaluation of text-based models appears limited. Given that SMAAT is presented as a generalized framework, a more comprehensive evaluation, especially on vision models, would significantly strengthen its contribution.\n- While the proposed SMAAT framework claims improved efficiency in adversarial training, the current efficiency gains primarily benefit specific architectures (encoder-based LLMs). A broader evaluation across diverse model types, including decoder-based LLMs, vision models, or multimodal architectures, would better demonstrate the generalizability and practical impact of SMAAT.", "other_comments_or_suggestions": "I suggest discussing the broader benefits of improving adversarial training efficiency, particularly for encoder-based LLMs, as it would strengthen the motivation and demonstrate greater practical significance.", "questions_for_authors": "Please address the concerns highlighted in the Experimental Designs and Strengths and Weaknesses sections. Clarifying these points, especially regarding the rigour of the experimental evaluations and the scope of the baseline comparisons, will likely influence my overall assessment of this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies how the interaction between adversarial perturbations and the data manifold influences whether adversarial training boosts generalization or robustness. Based on this insight, the authors propose to strategically apply perturbations at specific layers (perturbing later layers of encoder-based models and earlier layers of vision/decoder models). This approach shows promising results for encoder-based adversarial training. All reviewers agree that this is a good paper; the insight is clear, and the results are promising. Some detailed questions (mostly related to experiments, clarity, and code) were addressed during the rebuttal phase. We thus recommend acceptance of the paper.
Decision: Accept (poster)