Identifying and Understanding Cross-Class Features in Adversarial Training
Authors: Zeming Wei
Steven Y. Guo
Yisen Wang
Venue: ICML 2025 poster (2025)
Keywords: adversarial training
robust overfitting
feature attribution
training dynamics
Primary area: social_aspects->robustness
TL;DR: We present a novel perspective on adversarial training (AT) by identifying the critical role of cross-class features in achieving robust generalization

Abstract: Adversarial training (AT) has been considered one of the most effective methods for making deep neural networks robust against adversarial attacks, while the training mechanisms and dynamics of AT remain open research problems. In this paper, we present a novel perspective on studying AT through the lens of class-wise feature attribution. Specifically, we identify the impact of a key family of features on AT that are shared by multiple classes, which we call cross-class features. These features are typically useful for robust classification, which we offer theoretical evidence to illustrate through a synthetic data model. Through systematic studies across multiple model architectures and settings, we find that during the initial stage of AT, the model tends to learn more cross-class features until the best robustness checkpoint. As AT further squeezes the training robust loss and causes robust overfitting, the model tends to make decisions based on more class-specific features.  Based on these discoveries, we further provide a unified view of two existing properties of AT, including the advantage of soft-label training and robust overfitting. Overall, these insights refine the current understanding of AT mechanisms and provide new perspectives on studying them. Our code is available at https://github.com/PKU-ML/Cross-Class-Features-AT.

Identifying and Understanding Cross-Class Features in Adversarial Training
Zeming Wei
Yiwen Guo
Yisen Wang 34
Abstract
2016; Cohen al., 2019; Chen al., 2024) for defending against this threat have been proposed. Adversarial training (AT) (Madry et al., 2018), which adaptively adds adversarial perturbations to samples in the training loop, has been con- sidered one of the most effective ways to make the DNNs more robust to adversarial attacks (Athalye et al., 2018).
Adversarial training (AT) has been considered one of the most effective methods for making deep neural networks robust against adversarial attacks, while the training mechanisms and dy- namics of AT remain open research problems. In this paper, we present a novel perspective on studying AT through the lens of class-wise fea- ture attribution. Specifically, we identify the im- pact of a key family of features on AT that are shared by multiple classes, which we call cross- class features. These features are typically useful for robust classification, which offer theoreti- cal evidence to illustrate through synthetic data model. Through systematic studies across multi- ple model architectures and settings, we find that during the initial stage of AT, the model tends to learn more cross-class features until the best robustness checkpoint. As AT further squeezes the training robust loss and causes robust over- fitting, the model tends to make decisions based more class-specific features. Based on these discoveries, we further provide a unified view of two existing properties of AT, including the advantage of soft-label training and robust over- fitting. Overall, these insights refine the cur- rent understanding of AT mechanisms and pro- vide new perspectives studying them. Our code is available at https: htth:s...gg/ttttoo/oooo.. com/ hoo--at//roos-ccas--eeteereee
Given the unique success in improving adversarial robust- ness and the complex optimization process of AT, several studies have attempted to interpret AT through different per- spectives like feature visualization (Ilyas et al., 2019; Bai et al., 202 Li et al., 2023) and coverage analysis (Wang et al., 2019). However, there are still a few mysterious prop- erties of AT whose underlying mechanisms remain open research problems. First, AT can lead a phenomenon known as robust overfitting (Rice et al., 2020). During AT, model may achieve its best test robust error at certain epoch, but the test robust error will gradually increase in the latter stage of training. By contrast, the training robust error consistently decreases, resulting in a large robust generaliza- tion gap. Furthermore, although one-hot labels are usually adequate for standard training, integrating soft-label training methods such as knowledge distillation (Hinton et al., 2015) into AT can significantly improve AT whilst mitigating ro- bust overfitting (Chen et al., 2021) (e.g. 41% + 48% on CIFAR- 10 dataset). However, the reasons why soft labels are typically advantageous for AT remain unclear.
In this paper, we explore the mechanisms of AT and offer a unified understanding of the two properties from new aspect of class-wise feature attribution. Specifically, we divide the features learned by the model into cross-class features and class-specific features. The cross-class features are shared among multiple classes in the classification task, e.g. the feature wheels shared by the automobile and truck classes in the CIFAR- 10 dataset (Krizhevsky et al., 2009). We examine how these features are utilized across various stages of AT. Intriguingly, we observe that at the initial stage, the model gradually learns more cross-class features until reaching the most robust checkpoint. In contrast, at later checkpoints where robust overfitting occurs, the model tends to make decisions based more on class-specific features and decreases its dependence on cross-class features. Further- more, we find that models trained with properly learned soft labels, like knowledge distillation, can preserve more cross- class features during AT whilst mitigating robust overfitting.
1. Introduction
As the existence of adversarial examples (Goodfellow et al., 2014) has led to significant safety concerns of deep neu- ral networks (DNNs), a series of methods (Papernot et al.,
'School of Mathematical Sciences, Peking University 'Independent Researcher SSate Key Lab of General Artificial In- telligence, School of Intelligence Science and Technology, Peking University "Institute Artificial Intelligence, Peking University. Correspondence to: Yisen Wang <yisen.wang wang @ pku.edu.cn>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Identifying and Understanding Cross-Class Features in Adversarial Training
Motivated by these observations, we propose novel hy- pothesis of the AT training dynamics. During the initial stage of AT, the model learns both class-specific and cross- class features simultaneously, since these features are both helpful for reducing robust loss (iee, the cross-entropy loss on adversarial examples) when this loss is large. However, as training progresses and the robust loss decreases to a certain degree, the model begins abandon cross-class fea- tures and makes decisions based mainly on class- specisc features, which is caused by cross-class features raising pos- itive logits on other classes and yielding positive robust loss in AT under one-hot labels. Therefore, the model tends to neglect these features to further decrease the robust loss. However, these cross-class features are helpful for robust classification (e.g.. a feature shared by classes y1: 1/2 helps the model distinguish samples in class y1 from other classes and using only class-specific features is insuf- ficient to achieve the best robust accuracy. We discuss this insight in detail in Section 4. As a result, the robust test accuracy (i.e., the accuracy of the model on adversarial examples) gradually decreases, leading to the robust over- fitting issue. In addition, this hypothesis also explains why soft-label training methods typically improve AT as well as alleviate robust overfitting, as their softened labels can preserve more cross-class features during AT than standard one-hot labels.
sure the usage of cross-class features through different stages of AT. We also substantiate these assertions in a synthetic data model with decoupled cross-class and class-specific features.
3. Based our understanding, we further provide uni- fied interpretation of some intriguing properties of AT, like robust overfitting and the advantage of soft-label training, substantiating a novel perspective to study AT that warrants further investigation.
2. Background and Related Work
2.1. Adversarial Training
Adversarial training (AT) (Madry et al., 2018) has been widely recognized as one of the most effective approaches to improving the robustness of models ( (Athalye et al., 2018), 8), which can be formulated as the following min-max optimiza- tion problem:
Bnn, ".....
min 0 N
1
Σ max E(f(0, x, +.)). y1), 16.11
(I)
where @ represents the model parameter, e is the loss func- tion (i.e. cross-entropy loss), (I4, is the i-th sample- label pair in training set, and e is the perturbation bound. For the inner maximization, Projected Gradient Descent (PGD) (Madry et al.. 2018) is generally used to craft the adversarial example:
We provide extensive empirical evidence to support the observations and the hypothesis. First, we propose a met- ric to measure the usage of the cross-class features for a certain model. Then, among various perturbation norms, datasets, and architectures, we show that the best robustness model consistently uses more cross-class features than the robust overfitted ones, showing a clear correlation between robust generalization and cross-class features. We further provide theoretical insights to intuitively understand this effect through a synthetic data model, where we show that cross-class features are more sensitive robust loss, but they are indeed helpful for robust classification. Finally, we extend our study to more scenarios, including discussions on larger training perturbation alternative metrics, standard training, and fast adversarial training (Wong et al., 2020; Andriushchenko & Flammarion, 2020), to further support our insights.
x ++ 1 = (BB(..)) (x' a sign( (((;0''' v))),
(2)
where IT is the function that projects the sample onto an allowed region of perturbation, i.e., B(x, c) = fa' || - x|pp S e). and a controls the step size of gradient ascent. Throughout this thread, numerous variants of AT were proposed from various perspectives, e-g., loss func- tion (Zhang et al., 2019; Wang et al., 2020), computational cost (Shafahi et al., 2019; Wong et al., 2020), and model architecture (Huang et al., 2021; Mo et al., 2022). However, the min-max optimization nature of AT makes its training dynamics black box, and understanding the internal mech- anisms of AT remains an open research area problem (Li & Li, 2024; Wang et al., 2024; Zhang et al., 2024).
Our contributions can be summarized as follows:
2.2. Robust Overfitting
1. We propose a new hypothesis for the training mecha- nism in AT from the perspective of cross-class features. Specifically, the model gradually learss them at the initial stage of AT, and tends to reduce the reliance on them after certain stage. However, these features are actually helpful for robust generalization.
Despite success in improving robustness, AT suffers from a problem known as robust overfitting (Rice et al., 2020). As shown in Figure 1, the model may perform best on the test dataset at certain epoch during AT, but in the later stages, the model's performance on the test data gradually worsens. Meanwhile, the model's robust error on the training data continues to decrease, leading to a significant generalization gap in adversarial training. Moreover, for commonly used
2. We provide both empirical and theoretical evidence to support this understanding. Empirically, we mea-
2

---

Identifying and Understanding Cross-Class Features in Adversarial Training
propose a metric of cross-class feature usage for model in AT. Then, with comprehensive empirical evidence, we demonstrate the dynamics of the model in terms of learning these features during AT, as well as their relationship with robust overfitting and knowledge distillation.
2033
0255
Doach
teemh
3.1. Measuring the Usage of Cross-Class Features
(a) Train robust accuracy
(b) Test robust accuracy
Consider K-class classification task. Let f(-) = W (-(+) represent a classifier, where is the feature extractor with n dimension and W E R is the linear layer. For a given a sample x from the i-th class, the output logit for the i-th class is
Figure 1. Train and test robust accuracy of AT on CIFAR-10 dataset with ex -nom bound E (2, 4, 6, 82/555.
perturbation bound (e.g. [0, 8/255 for (ormo in AT, a relatively large suffers from more severe robust over- fitting. By contrast, for a small E = 2/255, this effect is relatively less pronounced. To address the robust overfit- ting issue in AT, several techniques have been introduced from various perspectives, like data augmentation (Rebuffi et al., 2021; Li & Spratling, 2023) and flatness regulariza- tion (Wu et al., 2020; Yu et al., 2022a). Meanwhile, a series of works attempt to interpret the mechanism of robust over- fitting through data-wise loss (Yu et al., 2022b) and label noises (Dong et al., 2022a). In this work, we provide a new perspective to refine the current understanding of robust overfitting from class-wise feature analysis.
n Σ Wl. j=1
- g(z)
(4)
where Wii] is the i-th row of W. Intuitively, g(x), W [ represents how the j-th feature influences the logit of the i-th class prediction of ((x). Thus we use
(5) as the attribution vector for the sample x on class i, where the j-th element denotes the weight of the j-th feature.
Ax(x) (o(x) Wli IJ.... (i,,).
Characterizing Cross-class Features. We consider the similarity of attribution vectors. If the attribution vectors of samples 21 and øe are highly similar, the model tends to use more features shared by them when calculating their logits for their classe (Bai et al., 2021a; Du et al., 2024). On the other hand, if the attribution vectors of 21 and 22 are almost orthogonal, the model uses fewer shared features, or they just do not share features. Further, this observation can be generalized to K classes. We model the feature attribution vector of a given class as the average of the vectors of the test samples in this class. Further, since we only focus on the feature attribution in the context of adversarial robustness, we only consider the usage of robust features (Tsipras et al., 2019; Ilyas et al., 2019) for classifying adversarial examples. Thus, we craft adversarial examples and analyze their attributions to measure the usage of shared robust features.
2.3. Adversarial Training with Smoothed Labels
Another intriguing property of AT is the advantage of us- ing properly smoothed lahels replace one-hot labels, e-g., leveraging knowledge distillation (Hinton et al., 2015; Chen et al., 2021) or using temporal ensembling (Laine & Aila, 2017; Dong et al., 2022b: Wang & Wang, 2022), For exam- ple, the loss function in Equation (1) of AT with knowledge distillation can be reformulated as
É(O: 0o, x + 6, v) (1 - A)GeR) (J v) - (L 5)T, x ))
(3)
where fcE is the cross-cntropy loss, KL is the Kull- divergence, T is the distillation temperature and 0 is the teacher model. This type of loss function ex- plicitly converts one-hot label into smoothed one, where the model does not necessarily achieve the minimized loss by outputting only a one-hot prediction logit. Motivated by its success in improving AT and mitigating robust overfit- ting, a series of variants of smoothed- AT have been proposed (Zhu et al., 2022; Zi et al., 2021; Huang et al., 2023; Yue et al., 2023; Wu al., 2024), but there is still lack of unified view of how they improve the peak perfor- mance of AT and also mitigate robust overfitting.
As discussed, we can measure the usage of cross-class robust features shared by different classes with the similarity of their attribution vectors. Therefore, we construct the feature attribution correlation matrix using the cosine similarity between the attribution vectors:
A, A₁
Cli.j] = |Ail|2 ||A,112
(6)
The complete algorithm of calculating matrix C' is shown in Algorithm I in Appendix. For two classes indexed by and j. Cli. .) represents the similarity of their feature attribution vector, where a higher value indicates the model uses more features shared by these classes.
3. Cross- Class (Robust) Features
In this section, we elaborate on our proposed understanding of robust overfitting in AT via cross-class features. We first
3

---

Identifying and Understanding Cross-Class Features in Adversarial Training
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
-0.25
-0.25
-0. 50
-0.50
-0. 50
6.75
0.75
0.75
-1.00
-1.00
-1.00
(a) Epoch 70 (Under-fitted)
(b) Epoch 108 (Best-fitted)
(c) Epoch 200 (Over-fitted)
RA= 42.6%, CAS= 18.2
RA= 47.8% CAS= 25.6
RA= 42.5%, CAS= 9.0
Figure2 2. Feature Attribution Correlation Matrix of models at different stages in AT, with their test robust accuracy (RA) and CAS. Class index: airplane (0), automobile (I), bird (2), cat (3), deer (4), dog (5), frog (6), horse (7), ship (8), truck (9).
Numerical Metric. To further support our claims, we pro- pose numerical metric named Class Attribution Similarity (CAS) defined the correlation matrix C:
(RA), the overlapping effect significantly decays, indicat- ing the model substantially neglects cross-class features in its classification. We provide detailed matrices during this training in Figure C in the Appendix.
CAS(C) Σ max max(CCi.,)..0 ((i.5..0) (F
Main hypothesis and Robust overfitting. This intriguing effect motivates us to propose the following hypothesis for the AT mechanism and training dynamics. We identify two kinds of learning mechanisms in AT: (1) Learning class- specific features, i.e., the features that are exclusive to only one class; (2) Learning cross-class features, i.e., the same or similar features shared by more than one class, For example, the wheels shared by categories automobile and truck.
The max function is used since we only focus on the posi- tive correlations, and the negative elements are small (see Figure 2) and do not affect our analysis. As a numerical indi- cator, CAS can quantitatively reflect the usage of cross-class features for a certain checkpoint.
3.2. Preliminary Study
Based on this hypothesis, the overall process of can be roughly divided into two stages. During the initial phase of AT, the model simultaneously learns exclusive class-wiss features and cross-class features. Both of these features help achieve robust generalization and reduce training ro- bust loss. However, once the training robust loss is reduced to a certain degree, it becomes difficult for the model to further decrease it by optimizing cross-class features, since the features shared with other classes tend raise positive logit on the shared classes. Thus, to further reduce the train- ing robust loss, the model begins to reduce its reliance on cross-class features and places more weight on class speeifsc features. Meanwhile, due to the strong memorization ability of AT (Dong et al., 2022b), the model also memorizes the training samples along with their corresponding adversarial examples, which further reduces the training robust error. This overall procedure can optimize training robust error but can also hurt test robust error by forgetting cross-class features, leading to decrease in test robust accuracy and resulting in robust overfitting.
Based on the proposed measurements, we first visualize the feature attribution correlation matrices of vanilla AT (Madry et al., 2018). For the detailed configurations of training, we follow the implementation of (Pang et al., 2021), which pro- vides popular repository of AT with basic training tricks. The model is trained on the CIFAR- 10 dataset (Krizhevsky et al., 2009) using PreActResNet-18 (He et al., 2016) for 200 epochs, and it achieved its best test robust accuracy at the 108th epoch. complete list of hyperparameters for experiments in this Section is presented Appendix B.
Observations. As shown in Figure 2, the model demon- strates fair overlapping effect feature attribution at the 70th epoch (Under-fitted). Specifically, there are several non non-diagonal elements Ci in the correlation matrix C' that exhibit relatively large value (in deeper blue), which indicates that the model leverages more features shared by the classes indexed by and j when classifying adversar- ial examples from these two classes. Therefore, the model has already learned several cross-class features in the initial stage of AT. Moreover, when the model achieves its best robustness at the 108th epoch, the overlapping effect on feature attribution becomes clearer, with more non-diagonal elements in C exhibiting larger values. This also verified by the increase in CAS. However, at the end AT, where the model is overfitted with decreased test robust accuracy
Soft-label AT. Our understanding can also explain why soft- label methods, exemplified by knowledge distillation, are helpful for AT in terms of both best checkpoint robustness and mitigating robust overfitting. In the process of AT with knowledge distillation, the teacher model adeptly captures the cross-class features present in the training data, and

---

Identifying and Understanding Cross-Class Features in Adversarial Training
1.00
1.00
0.75
0.75
Sample
2-
250
0.90
0.25
0.25
0.00
0.00
Best
0.25
0.25
predict:
automobile
truk
horse
Frg
Bird
deer
0.50
0.50
8-
+0.75
-0.75
Last
1,00
1.00
pralist:
dog
alrplatc
(a) AT+KD (Best)
(b) AT+KD (Last)
(c) Saliency maps visualization
RA= 48.1%, CAS= 25.7
RA= 46.2%, CAS= 24.1
Figure 3. (a), (b): matrices for the best and the last checkpoint of AT with knowledge distillation. and their test Robust Accuracy (RA) CAS. (c): Visualization of saliency map with GradCAM. The top row shows original sample, and middle and bottom rows show the saliency map adversarial examples of the best and the last checkpoint, respectively.
then converts the one-hot label into by considering both class-specific and cross-class features. This stands in contrast to vanilla AT with one-hot labels, which primarily emphasize class-specific features and may inadvertently suppress cross-class features in the model weights. Similarly, other smoothed labels, like temporal ensembling, can also effectively mitigate robust overfitting by preserving these crucial features.
precise one
tures being the mane for horse, the frog eyes for frog, the feather for bird, and the antlers for deer. Since the final checkpoint makes decisions based only these limited features, it fails to leverage comprehensive features for clas- sification, making the model more vulnerable to adversarial attacks. More examples on this comparison can be accessed in Appendix D.
Comparing with different perturbation bound As stated in Section 2, the robust overfitting effect is more severe with larger for regular AT (s 8/255), as shown in Figure 1. Intuitively, AT with a larger perturbation bound results in a more rigid robust loss. During AT with a large e, cross-class features are more likely to be climinated by the model to reduce training robust loss, which we prove in Theorem in the next section. In Figure visualize
To support this claim, present comparison between the best and last checkpoint of AT with knowledge distillation in Figure 3 (a) and (b), where no significant differences be- tween the two matrices, nor large gap between their CAS. Therefore, we conclude that AT with knowledge distillation helps by identifying cross-class features and providing more precise labels by considering these features.
3.3. More Empirical Studies
In this section, we conduct more comprehensive studies to support our hypothesis proposed above.
2
05
62
Visualization of saliency map To further interpret the con- cept and role of cross-class features, we present comparisons of the saliency maps on several examples that are correctly classified by the best but misclassified by the last check- point under adversarial attack, as shown in Figure 3 (c). The saliency map derived by Grad-CAN (Selvaraju al., 2017) the true labeled classes. Taking the first column as example, the classes automobile and truck share simi- lar class-specific discriminative regions (highlighted in the saliency map) like wheels. The best checkpoint pays more attention to the overall car including the wheel, whereas the last checkpoint solely focuses on the circular car roof that is exclusive to automobiles. This explains why the last checkpoint misclassifies this sample, for it only identifies this local feature for the true class and does not leverage holistic feature information from the image. The other five samples also exhibit similar effect, with exclusive fea-
(a) E = 2/255
(b) c 4/255 CAS 8.9
A CAS= 4.1
05
00
(c) E 6/255
(d) E 8/255
A CAS= 13.8
A CAS- 16.6
Figure 4. The differences between the feature attribution corre- lation matrices (Cres Cia)) and CAS of the best and the last checkpoint with various training perturbation bound e.
5

---

Identifying and Understanding Cross-Class Features in Adversarial Training
the differences of the feature attribution correlation matrices and CAS between the best and last checkpoint of AT with various perturbation bounds The difference between the two matrices indicates how many cross-class features are abandoned by the model from the best checkpoint to the last. When = 2/255, there is no significant difference between the best and last checkpoint. However, as e increases, AT exhibits more overfitting effects, and the difference becomes more significant. This also verifies that the forgetting of cross-class features is key factor of robust overfitting,
(a)(b), where there are still significant differences between matrices from the two checkpoints of f2-norm AT. Other training configurations are the same as lo -nomm AT above.
Transformer architecture. We show the comparison of the feature attribution correlation matrices of the best and last checkpoints of AT on CIFAR-IO with vision transformer architecture (Deit-Ti (Touvron et al., 2021)) in Figure (c)(d) The observation consistent with other settings.
Notebly, while we mainly focus on AT with practically used c (e.g.. [0, 8/255] for CAT), it is also observed that for extremely large <(> 8/255), the effect of robust overfit- ting begins to decline (Wang et al., 2024; Wei al., 2023). Our interpretation is also compatible with this phenomenon, which we discuss in Section 5.1. In brief, cross-class fea- tures are more sensitive under extremely large 6, making them even harder to learn at the initial stage of AT. Therefore, even at the best checkpoint, they learn fewer cross-class fea- tures, resulting in fewer forgetting of these features in the latter stage of AT.
-0
(a) l2-AT (Best) RA= 69.8%, CAS= 22.1
(b) 2- (Last)
RA= 65.6%. CAS= 10.7
More datasets. We extend our observations by illustrat- ing the comparisons the CIFAR-100 (Krizhevsky al., 2009) and the TinyImagenet (mnmoustafa, 2017) datasets in Figure 5. We can see that there are still significant differ- ences between matrices and CAS derived from the best and the last checkpoint of AT on other datasets, showing this effect still holds for various datasets.
(c) DeiT-Ti (Best) RA= 47.9%, CAS= 25.4
(d) DeiT-Ti (Last) RA= 42.6%, CAS= 16.6
Figure 6. Feature attribution correlation matrices on f--norm AT and Visual Transformer architecture. Color bar scaled to [0, 1].
24
Overall, these empirical findings provide solid justification for our main hypothesis for the learning dynamics of cross- class features during AT. In the following section. we also offer theoretical insights to intuitively understand the role of cross-class features in robust classification.
60
64
60
(a) CIF (Best)
(b) CIFAR- 100 (Last)
RA= 24.7%, CAS= 569
RA= 19.6%, CAS= 352
4. Theoretical Insights
22
In this theoretical framework, we first introduce synthetic data model and then provide insights into our claims.
30
120
08
15
4.1. Data Distribution and Hypothesis Space
350
TM
(c) Tinylmagenet (Best)
(d) Tinylmagenet (Last)
Data distribution We consider a tertiary classification task, where each class owns an exclusive feature attribution 3g and every two classes have a shared cross-class feature attribution #C,j. The attribution for each sample can be formulated as {IFE IC,j S j < 3} E RR.. The data distribution is similar to the model applied in robust and non-robus, features (Tsipras et al., 2019), but we only focus on the inner relation between robust features (class- (class-specific or cross-class) and omit the non-robust features.
RA= 18.0%, CAS= 1548
RA= 14.4%, CAS= 998
Figure 5. Feature attribution correlation matrices on CIFAR-100 and Tiny-ImageNet datasets. Color bar scaled to [-0.75,0. 0.75).
fo-norm AT. We show the comparison of the feature attri- bution correlation matrices of the best and last checkpoints of norm AT (c 128/255) on CIFAR-10 Figure
6

---

Identifying and Understanding Cross-Class Features in Adversarial Training
As discussed above, we model the data distribution of the i-th class Y/ as D;- =
Cross-class features are helpful for robust classification. Although decreasing the value of W2 may reduce the robust training error, we demonstrate in Theorem 2 that using pos- itive w2 is always more beneficial for robust classification than simply setting W2 to 0.
j == xi IC, 10.-- 0,
[N(m.o2),
i
FE
0.
j=i
(8)
Theorem 2. For any class y, consider weights wi > 0.
where i E (1, and p.o a 0. We also assume d VTT to control the variance.
U2 E [0, wil, and e E (0, 5). When sampling x from the distribution of class y, increasing the value of w2 enhances the possibility of the model assigning higher logit to class y than to any other class y' # y under adversarial attack. In other words, the probability
Hypothesis space We introduce a linear model f(x) in this classification task, which gives i-th logit for sample x by f(x), = E, w T.. + Σ, I.j...- However, there are 6 parameters in the data samples, making this linear model hard to analyze. Thus we simplify the model based on the following observations. First, we can simply keep wf, for i * j and wf, = due to the corresponding data distribution is identity to O. Further, we set wf,1 = = = W W1 and wf, wy((= + j) due to symmetry, similar to (Tsipras et al., 2019). Finally, we assume W1, W2 2 0 since j > 0. Overall, the hypothesis space is {fw w = (w), w2), W1, W2 2. 0} and fmo(x) calculates its i-th logit by fw (z); = WIT + wyy(z ( + xC.j2). where {j1. = (1,2,3)((3) Now we consider adversarially training far with le- -nom perturbation bound e < 5. We also add a regularization term 2 || to the overall loss function, which can be modeled as
Pr (fo( + +6), > fa + 8ly, V5: 1811> S c (11) z=D,
monotonically increases with W2 within the range [0, wi].
Smoothed label preserves cross-class features. Finally, we show that smoothed labels can help preserve the cross- class features, which justifies why this method can alleviate robust overfitting. Note that due to the symmetry of distribu- tions and weights among classes, we apply label smoothing to simulate knowledge distillation and rewrite the robust loss as
max f x + +8} + à 11 (12) 110 5e
Eappy (BEND,
(1233) (EEND. max E(w; + +)} + X S
where (us(w; x + 8) is
(9)
2
(1 - 3)1 max (max Suo + 8)5 ful + +8)a]]
11 jti
where
(13)
Σ fw + 8)}.
[(wxx+5)= = max (max fw (Z+)) So Jf4
1aa--8yy-fur ffw (x+)))) (10)
2 jti
4.2. Main results
and 8 < _/2 is the interpolation ratio of label smoothing. In Theorem 3 and Corollary 1, we show that not only does the label smoothed loss (13) enable a larger perturbation bound e for utilizing cross-class features, but also returns larger W2- This explains that preserving the cross-class features is the reason why smoothed labels help mitigate robust overfitting.
Cross-class features are more sensitive to robust loss. We show that under the robust training loss (10), the model tends to abandon de by setting w/2 = 0 if is larger than a certain threshold. However, any E (0, () returns a positive W1, as stated in Theorem 1. This result indicates that cross- class features are more sensitive to robust loss and are more likely to be eliminated in AT compared to class-specific features, even when they share the same mean value μ.
Theorem 3. Consider AT with label smoothing loss (13).
There exists an E (0, 2) with c1 > co derived in Theorem 1, such that for E (0,e1). the output function obtains W2 > O: for E (f1, Ju), the output function returns w2 = O.
Theorem 1. There exists a ED E (0, }u). for AT by optimiz- ing the robust loss ( (10) with e E (0, Eo), the output function obtains W²2 > 0; for AT with e E (eo, }1)), the output func- tion returns w2 = 0. By contrast, AT with e E (0, (u) always obtains w1 >
Corollary 1. Let wy(e) be the value of we returned by AT with (10), and wof(()) be the value of w2 returned by label smoothed loss (13). Then, for e E (0,e1), we have (c) we(e).
This claim is also consistent with our discussion AT with different in Section 3.3. Recall that AT with larger tends to compress more cross-class features as shown in Figure 4. This observation can be verified by Theorem 1 that cross- class features are more likely to be eliminated during AT with larger e, which causes more severe robust overfitting.
All proofs can be found in Appendix E. To summarize, our theoretical analysis demonstrates that cross-class features are more sensitive to robust loss, yet helpful for robust classification. We also present a discussion on extension to higher dimensions in Appendix E.5.
7

---

Identifying and Understanding Cross-Class Features in Adversarial Training
5. Extended Studies and Discussions
ting issue that the test robust accuracy suddenly decreases to near 0% after certain epoch (Kim al., 2021), different from robust overfitting, where the robust accuracy gradually decreases. Our understanding is also compatible with this phenomenon, as discussed in the following.
In this section, we extend our observations to broader sce- narios substantiate our understanding.
5.1. Regarding extremely large 6
Our interpretation is consistent with empirical observations that for commonly used с E 8/,5/2 larger perturba- tion bounds exacerbate robust overfitting. However, it also resolves the seemingly contradictory phenomenon where extremely large (e.g., 8/255) mitigates overfitting (Wang et al., 2024; Wei et al., 2023). To interpret this phenomenon, recall that our main interpretation for ro- bust overfitting is that the model begins to forget cross- class features after a certain stage. Regarding AT with extremely large e, as we proved in Theorem 1, the more rigid robust loss makes the model even harder to learn cross-class features at the initial stage of AT. Given that fewer cross-class features are learned, the forgetting effect of these features is weakened, thus mitigating robust overfit- ting. We support this mechanism with empirical validations.
Epoch 10 CAS=13.8
Best CAS=14.1
After CO CAS=2.1 RA=0.1%
RA=40.0%
RA=41.8%
Figure 7. Feature attribution correlation matrices for fast adver- sarial training at different stages, including epoch 10, best check- point, and after catastrophic overfitting (CO).
We conduct experiments using the FAT method on the CIFAR-10 dataset, with other settings the same as stan- dard AT. The feature attribution correlation matrices and CAS values epoch 10, the best checkpoint, and after catas- trophic overfitting are presented in Figure 7. Similar to the standard AT, the model has already learned certain amount of cross-class features at epoch 10, and achieves better ro- bustness and CAS at the best checkpoint. However, after catastrophic overfitting occurs, the CAS value plummets to 2.1, and the robust accuracy drops to near zero. This sug- gests that during catastrophic overfitting, the model almost completely forgets the cross-class features it had learned ear- lier. Therefore, the forgetting of cross class features is also an underlying mechanism of catastrophic overfitting, which aligns well with our observations on robust overfitting.
Table Comparison RA and CAS on AT with large
Epoch
10
Best
Last
e for AT
CAS / RA
CAS/RA
CAS RA
8/255
25.6/47.8% 16.7/36.9%
9.0/42.5%
12/255
15.6/29.8%
18.9/38.7%
8.7/34.1%
16/255
14.4/23.8%
17.5/31.3%
8.4/28.1%
Specifically, we compare models trained with -nom c E (8/255, 12/255, 16/255] on CIFAR-10, tracking CAS and robust accuracy across epochs (Table 1). At the 10th epoch, models with 12/255 and 16/255 exhibit lower CAS than 8/255, confirming their struggle to learn cross-class features early on. By the best checkpoint, peak CAS values for larger remain markedly lower, indicating limited cross-class feature retention. Crucially, the gap in CAS between the best and final checkpoints shrinks as increases, mirroring the reduced divergence in robust ac- curacy. This trend aligns with our hypothesis: extreme values suppress cross-class feature acquisition from the out- set, leaving fewer features to discard during later stages. Consequently, the attenuated forgetting effect aligns with diminished robust overfitting
5.3. Instance-wise Metric Analysis
In this section, we provide an alternative metric to further support our claims by calculating the feature attribution matrix and CAS instance-wisely. Specifically, when con- sidering classes and j. for each sample x from class i, we identify its most similar counterpart x'' from class j. We then calculate their cosine similarity and average the results over all samples in class i. In this context, x' can be interpreted as the sample in class j that shares the most cross-class features with x among all samples in class j. which pro- vides another way to quantify the utilization of cross-class features. We also attempt to average over all sample pairs (x, x') in classes i and j. but due high variance among samples, each element in the correlation matrix C' hovered near zero throughout all epochs in adversarial training, ren- dering it unable to provide meaningful information. Based on this metric, we conduct a similar study by calculating the matrices and I-CAS for the best and last checkpoints of Lo and Æ2-AT, and the results are shown in Figure
5.2. Regarding catastrophic overfitting
Another intriguing property of AT is the catastrophic over- fitting phenomenon in fast adversarial training (FAT) (Wong et al., 2020; Andriushchenko & Flammarion, 2020), which applics a single-step perturbation during AT for better effi- ciency. However, FAT suffers from the catastrophic overfit-
8

---

Identifying and Understanding Cross-Clast Features in Adversarial Training
5.5. Discussion future applications
Finally, building on our comprehensive study on the critical role of cross-class features in AT, discuss their poten- tial future applications in robust generalization research. First, similar to the robust/non-robust feature decomposi- tion (Tsipras et al., 2019), our cross-class feature model has the potential for more in-depth modeling of adversarial robustness, contributing new tools in its theoretical analysis. Meanwhile, for AT algorithmic design, we list some future perspectives of cross-class feature follows:
(a) 250 (Best) I-CAS=34.9
(b) & (Last) I-CAS=25.6
Data (re)sampling. While generated data is prone to help advance adversarial robustness (Gowal et al., 2021; Wang et al., 2023), it requires significantly more data and computational costs. From the cross-class feature perspective, adaptively sampling generated data with considerations of cross-class relationships may improve the efficiency of large-scale AT.
(c) AT (Best) 1-CAS=27.00
(d) AT (Last) I-CAS=14.9
AT configurations. Customizing AT configurations like perturbation margins or neighborhoods is useful for improving robustness (Wei et al., 2023; Cheng et al., 2022). this regard, customizing sample-wise or class-wise AT configurations based on cross-class rela- tionships may further improve robustness.
Figure& 8. instance-wise feature attribution correlation matrices.
Consistent with the results for class-wise attribution vectors, it is still observed that there is a significant decrease in the usage of cross-class features from the best checkpoint to the last for both to and AT. This observation further substantiates our understanding of cross-class features.
Module design. The model architecture (Huang et al., 2021) and activation mechanisms (Bai et al., 2021b) are crucial in improving robustness. Thus, designing modules that implicitly or explicitly emphasize cross- class features may enhance robustness.
5.4. Regarding Standard Training
We also extend our experimental scope to include standard training. The experimental settings are the same as those outlined in previous sections for CIFAR-10. 10, with the only difference being the absence of perturbations in standard training. We present the matrices and CAS results for epochs (50, 100, 150, 200} in Figure 9. Considering that standard training only focuses on clean accuracy and exhibits negli- gible robustness, we calculate the feature attribution vectors using clean examples. These results reveal a clear lack of differences between them, particularly in the latter stages (150th and 200th), where the training tends converge. This observation is consistent with the characteristic of stan- dard training, which generally does not exhibit significant overfitting (Jiang et al., 2020; Guo et al., 2023). In addition, the numerical magnitude of CAS by these models is signifi- cantly lower than AT (generally > 20), showing that they just use fewer cross-class features standard classification.
6. Conclusion
In this work, we present novel perspective to understand adversarial training (AT) dynamics through the lens of cross- class features. We demonstrate that cross-class features, which are shared across multiple classes, play a critical role in achieving robust generalization. However, as training pro- gresses, models increasingly rely on class-specific features to minimize robust training loss, leading to the forgetting of cross-class features and subsequent robust overfitting. Our empirical analyses across datasets, architectures, and pertur- bation norms, as well as theoretical insights, validate this hy- pothesis that models at peak robustness utilize significantly more cross-class features than overfitted ones. Furthermore, we reveal that soft-label methods like knowledge distillation mitigate overfitting by preserving cross-class features, align- ing with their empirical success. These findings are further substantiated through extended studies like large perturba- tion AT, fast adversarial training, alternative metrics, and comparison with standard training. Overall, our understand- ing provides a unified explanation for robust overfitting and the efficacy of label smoothing in AT, offering new insights for studying robust generalization.
Epoch 50 CAS=7.3
Epoch 100 CAS=8.4
Epoch 150
Epoch 200 1AS=10.2
CAS=9.8
Figure 9. Feature attribution correlation matrices for standard training at different stages. Color bar scaled to [0, 0.5).

---

Identifying and Understanding Cross-Class Features in Adversarial Training
Acknowledgments
Dong, Y., Xu, K., Yang, X., Pang, T., Deng, Z., Su, H., and
Zhu, J. Exploring memorization in adversarial training. In ICLR, 2022b. 3,4
Yisen Wang was supported by National Key R&D Pro- gram of China (2022D00160300), National Natural Science Foundation of China (92370129, 62376010), Beijing Nova Program (20230484344, 20240484642), and BaiChuan AL Zeming Wei was supported by Beijing Natural Science Foundation (QY24035).
Du, T., Wang, Y., and Wang, Y. On the role of discrete tok- enization in visual representation learning, arXiv preprint arXiv:2407.0990 000000 2024.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv: 141.6657, 2014.
Impact Statement
This paper refines the current understanding of adver- sarial training (AT) mechanisms, which could improve the robustness of AI systems in safety-critical appli- cations like autonomous driving and cybersecurity. By identifying the role of cross-class features in AT, our findings may inspire more reliable and gener- alizable defense strategies against adversarial attacks.
Gowal, S., Rebuffi, S.-A., Wiles, O., Stimberg, E., Calian, D. A., and Mann, T. A. Improving robustness using generated data. In NeurIPS, 2021.
Guo, X., Wang, Y., Du, T., and Wang, Y. Contranorm: A contrastive learning perspective oversmoothing and beyond. arXiv preprint arXiv:2303. 06562, 2023, 9
He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In ECCV, 2016.
References
Andriushchenko, M. and Flammarion, Understanding and improving fast adversarial training. NeurIPS, 2020. 2,
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv: htw::5033.025 1503.02331, 2015. 1,3
Athalye, A., Carlini, N., and Wagner, D. Obfuscated gra- dients give false sense of security: Circumventing de- fenses to adversarial examples. In ICML, 2018. 1,2
Huang, B., Chen, M., Wang, Y., Lu, J., Cheng, M., and Wang, W. Boosting accuracy and robustness of student models via adaptive adversarial distillation. In CVPR,
Bai, Y., Yan, X., Jiang, Y., Xia, S.-T., and Wang, Y. Clus- tering effect of adversarial robust models. In NeurIPS, 2021a. 1,3
2023. 3
Huang, H., Wang. Y., Erfani, S. M., Gu, Q., Bailey, J., and Ma, X. Exploring architectural ingredients of adversari- ally robust deep neural networks. In NeurIPS, 2021. 2, 9
Bai, Y., Zeng, Y., Jiang, Y., Xia, S.-T., Ma, X., and Wang, Y. Improving adversarial robustness via channel-wise activation suppressing. In ICLR, 2021b.
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. In NerulPS, 2019. 1,
Chen, H., Dong, Y., Wang, Z., Yang, X., Duan, C., Su, H., and Zhu, J. Robust classification via a single diffusion model. In ICML, 2024.1
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In ICLR, 2020.
Chen, T., Zhang, Z., Liu, S., Chang, S., and Wang, Z. Ro- bust overfitting may be mitigated by properly learned smoothening. In ICLR, 2021. 3,
Cheng, M., Lei, Q. Chen, P.-Y,, Dhillon, I., and Hsieh, C.-.. Cat: Customized adversarial training for improved robustness. IJCAI, 2022. 9
Kim, H., Lee, W., and Lee, J. Understanding catastrophic overfitting in single-step adversarial training. In AAAI, 2021. 8
Cohen, J. M., Rosenfeld, E., and Kolter, J. Z. Certified adversarial robustness via randomized smoothing. In ICML, 2019.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers
features from tiny images. 2009. 1,4,6 6
Laine, S. and Aila, T. Temporal ensembling for semi- supervised learning. ICLR, 2017.
Dong, C., Liu, L., and Shang. J. Label noise in adversarial training: A novel perspective to study robust overfitting. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), NeurlPS, 2022a.
Li, A., Wang, Y., Guo, Y., and Wang. Y. Adversarial exam- ples are not real features. In NeurIPS, 2023.
10

---

Identifying and Understanding Cross-Class Features in Adversarial Training
Li, B. and Li, Y. Adversarial training can provably improve robustness: Theoretical analysis of feature learning pro- cess under structured data. In Mathematics of Modern Machine Learning Workshop at NeurlPS 2024., 2024. 2
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma. X., and Gu, Q- Improving adversarial robustness requires revisiting mis- classified examples. In ICLR, 2020. 2
Wang. Y., Li, L.,, Yang. J., Lin, Z., and Wang, Y. Balance, imbalance, and rebalance: Understanding robust overfit- ting from minimax game perspective. In NeurlPS, 2024. 2, 6, 8
Li, L. and Spratling, M. Data augmentation alone can improve adversarial training. ICLR, 2023.
Madry, A., Makelov, A., Schmidt, I.., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. 1,
Wang. Z.,, Pang, T., Du, C., Lin, M., Liu, W., and Yan, S. Better diffusion models further improve adversarial training. In ICML, 2023. 9
mmmoustafa, M. A. Tiny imagenet. 2017. URL URthttps:// kaggle. com/ competi competitionss witos/itiinniii ieni--mmagetttt
Wei, Z., Wang, Y., Guo, Y., and Wang. Y. Cfa: Class-wise calibrated fair adversarial training. In CVPR, 2023.
Mo. Y., Wu, D., Wang, Y., Guo, Y., and Wang, Y. When adversarial training meets vision transformers: Recipes from training to architecture. In NeurIPS, 2022. 2
Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free:
Revisiting adversarial training. In /CLR, 2020. 2, 8
Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight per- turbation helps robust generalization. In NeurIPS, 2020. 3
Pang. T., Yang, X., Dong, Y., Su, H., and Zhu, J. Bag of tricks for adversarial training. In ICLR, 2021.
Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. Distillation as defense adversarial perturbations against deep neural networks. In SP, 2016.
Wu, Y.-Y., Wang, H.J., and Chen, S.-T. Annealing self- distillation rectification improves adversarial training. In ICLR, 2024. 3
Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, Data augmentation can improve
Yu, C., Han, B., Gong, M., Shen, I.., Ge, S., Du, B., and Liu, T. Robust weight perturbation for adversarial training. In IJCAI, 2022a. 3
O., and Mann, robustness. NeurIPS, 2021.
Rice, L., Wong, E., and Kolter, Z. Overfitting in adversari- ally robust deep learning. In ICML, 2020. 1.2
Yu, C., Han, B., Shen, L., Yu, J., Gong, C., Gong, M., and Liu, T. Understanding robust overfitting of adversarial training and beyond. In ICML, 2022b.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 5
Yue, X., Mou, N., Wang, Q. and Zhao, L. Revisiting adversarial robustness distillation from the perspective of robust fairness. In NeurlPS, 2023. 3
Shafahi. A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! NeurlPS, 32, 2019. 2
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019. 2
Touvron, H. Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention. In ICML, 2021.
Zhang, Y., He, H., Zhu, J., Chen, H., Wang, Y., and Wei, Z. On the duality between sharpness-aware minimization and adversarial training. In ICML, 2024. 2
Zhu, J., Yao, Han, B., Zhang, J., Liu, T., Niu, G., Zhou, J., Xu, J., and Yang, H. Reliable adversarial distillation with unreliable teachers. In ICMI, 2022. 3
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. ICLR, 2019. 3, 6, 7,9
Zi, B., Zhao, S., Ma, X., and Jiang, Y.-G. Revisiting ad- versarial robustness distillation: Robust soft labels make student better. In ICCV, 2021.
Wang, H. and Wang. Y. Self-ensemble adversarial training for improved robustness. In ICLR, 2022. 3
Wang, Y. Ma, X., Bailey, J., Yi, J., Zhou, B., and Gu, Q. On the convergence and robustness of adversarial training. In ICML, 2019.

---

Identifying and Understanding Cross-Class Features in Adversarial Training
A. Algorithm for calculating the feature attribution correlation matrix
present the complete algorithm for calculating the feature attribution correlation matrix in Algorithm falg. For each class, we first calculate the feature attribution vectors for each test adversarial sample, then calculate the mean of these vectors as the feature attribution vector of this class. Finally, we calculate the cosine similarity of the vectors as the measure of cross-class feature usage for each pair of two classes.
Algorithm Feature Attribution Correlation Matrix
Input: A DNN classifier f with feature extractor g and linear layer W; Test dataset D (Dy y E y): Perturbation margin
Output: A correlation matrix C' measuring the cross-class feature usage
* Record robust feature attribution
for y E y do
A +- (0,...,)) 0) initialization as a n-dim vector
for x E D, do
8 arg 18 St (cu(f(x+ /* untargeted PGD Attack
*/
A += g(x + @ W /* point piint-wise multiplication
x
A + A 1 /* Average
for 1 ii | do
Cii + - /+ Cosine similarity
A'
return C
B. Detailed training hyperparameters
A complete list of training hyperparameters for AT models is shown in Table 2. For more implementation details, please refer to our code repository https: /ttp::///iitee /ttpt/ggtttt/////////e/eheehhe hww.hhuu...ooeoeeeeeeeeeeeeeee Cross- Class Features-Aa AT.
Table 2, Hyperparameters for AT
Parameter
Value
Train epochs
200
SGD Momentum
0.9
batch size
128
weight decay
5e-4
Initial learning rate
0.1
Learning rate decay Learning rate decay rate training PGD steps training PGD step size (foo) training PGD step size (f2)
100-th, 150-th epoch
0.1
10
e/ (e is the perturbation bound)
e/ /8 (e is the perturbation bound)
C. More feature attribution correlation matrices at different epochs
We present more feature attribution correlation matrices different epochs in Figure 10, and the test robust accuracy is aligned with Figure ((b) (red line, 8/255). From the matrices we can see that at the initial stage of AT (10th- 90th Epochs), the model has already learned several cross-class features, and the overlapping effect of class-wise feature attribution achieves the highest at the 10th epoch among the shown matrices. However, for the later stages, where the model starts overfitting, this overlapping effect gradually vanishes, and the model tends to make decisions with fewer cross-class features.
12

---

Identifying and Understanding Cross-Class Features in Adversarial Training
600
Epoch 10
Epoch 30
Epoch 50
Epoch 70
Epoch 90
CAS- 16.7 RA=36.9%
CAS= 17.8
CAS= 17.9
CAS= 18.2
CAS- 19.7
RA=11.2%
RA=41.5%
RA=42.6%
RA=42.8%
435
375
60
Epoch 110
Epoch 130
Epoch 150
Epoch 170
Epoch 190
CAS= 23.6
CAS= 18.9 RA=46.4%
CAS= 15.6 RA=44.7%
CAS= 13.8 RA=43.3%
CAS 9.1 RA=42.8%
RA=47.5%
Figure 10. Feature attribution correlation matrices, and their corresponding robust accuracy (RA), CAS at different epochs.
D. More saliency map visualizations
We include more visualization examples (ordered by original sample ID) in Figure 11, where many saliency maps of these examples still exhibit such properties discussed in Section 3.3. However, acknowledge that not all samples enjoy such clearly interpretable features (e.g., wheels shared by automobiles and trucks), since features learned by neural networks are subtle and do not always align with human intuition, including cross-class features.
E. Proof of theorems
E.1. Preliminaries
First, we present some preliminaries and then review the data distribution, the hypothesis space, and the optimization objective.
Notations
Let N(u, o) be the normal distribution with mean and variance o². We denote <(x) e o
and
V2x
((x) = -_ V2n
e f dt Pr..r.WW PF.((O00,)) < x) as its probability density function and distribution function.
Data distribution
For E (1, the sample of the i-th class is
NE,,,
R
(14)
follows distribution
EE (3) j) O²,
Joci(Mi (1i * -N,, sc.i (v: = =
(15)
FF (y/ # j) 0
and μ, a > We also assume VTp to control the variance.
Hypothesis space
The hypothesis space is (Sw (w1, 12)) w1, w2 20 and fo (x:) calculates its i-th logit
(x), = TF + W2(C.h + #CJ.): where [j1,22} = (1,2.
(16)
Optimization objective Consider adversarially training fwo with 6oo norm perturbation bound c < We hope that given sample ~ D., under any perturbation {6 :|66| S <). the f(r + 6); is larger than any f(x + 8)3 as much as possible. We also add regularization term to the loss function.
13

---

Identifying and Understanding Cross-Class Features in Adversarial Training
Sample
Sample
Best
Best
predict:
dog
ship
autommbbia airplene
predict:
dierr
truck
dog
truck
Last
Last
predict:
dog
ship
truck
airplane
deer
froa
predict: car
doer
dbar
truck
dog
bruck
Sample 0-5
Sample 6-11
Sample
Sample
Best
Best
predict:
frug
homse
truck
smip
day
honse
predict::
truek
frg
autormci le airplene
wrroleme
truck
Last
predict:
frog
horse
bruck
s-ip
deg
horse
predict:
truck
frog
automeble airplane
airalane
truck
Sample 12-17
Sample 18-23
Figure 11. More saliency maps visualization ordered by sample ID in CIFAR-IO.
Overall, the loss function can be formulated as
C(fw) = E, E.EDD, max (max fwll + 8) (x 0),1] + 1111
(17)
| Se ffi
E.2. Proof for Theorem
Theorem There exists a E (0, (u), for AT by optimizing the robust loss (17) with e E (0, en), the outpul function obtains W2 > 0; for AT with E (Eo. 2)), the output function returns W2 = 0. By contrast, AT with e E (0, (4) always obtains w1 0.
To prove Theorem fthatrain robust, need the following lemmas.
1 Lemma 1. Suppose that X, Y ~ N'0, 1), and they are independent. Let = max(X, YJ, then E|Z]
V
proof. Let p(-) and F(-) be the probability density function and distribution function of Z, respectively. Then, for any 2 E IR,
F(2) = Pr(Z < 2) = Pr(mmx(XXXX)) < Pr(X < < < < = =x)(2).
(18)
and we have
p(=) = F'(=) = ( (2)) 20())
(19)
14

---

Identifying and Understanding Cross-Class Features in Adversarial Training
Thus,
+000
E
2z0(2) 2z)(2)((z)dz
&
2
di)dz
2
v2x
-00
00
2
T dt)d(e-
(20)
1
+00
dt) -00
130
+
e
2 dz
π
π -00
1
to0
1
0+
dz
V
Lemma 2. Given (
FE,,, TE,,, CC ~ D1, E (0, 5) and un = (wi, W2). then ō
TC,2,
(-E,,,E,,,E, -, -) is solution for å = arg max [max fw (x + 8)5 - fwl + 8]].
| x j71
proof. Denote 5 = (Ō 8F,2, 81,3, oc, 60,2, ).aa). Note that for x ~ D1, we have TF.2 DF. = ZC,1 = 0. Then,
max j#1
Sa (x + 8), la( + 5)1
max [w15g,2 + w2d0,1 + w2(103 + 6c,3), w wigg, + w2dc,1 + w2(2222 + 8c,2)l jE(2,3)
- Wi (1E,1 + ŌE,1) - w2(2022 + 60,2 + Ic,3 + 5c,3)
(21)
=w28c,1 + max [1] + w2(#03 + 6c,3), w widE, + w2(#022 + ,c,2)l
;E(2.3)
wi (Σ + ) - w2(2022 + 6, + IC, + 5c,3)-
Since W1, W2 2 0, it is clear that 8g,1 c, 6g, = وg,3 = 6c, = are the optimal values for maximizing (21). As for ō and 6c,3, prove that dc,2 6c,2
the optimal values, by variable simplification (a' 60,2, V/ 60,3) and
dividing by wy only need to show that
max + a', 8'} - a' - 1/ S max b - c) + 2
(22)
under the constraint
|a'|
S and [3'] S Note that (22) is equivalent to
+ + d' - s max( +e + max((a + b 6'} < max(a, b) + a' + + te b') < max(a + a' + b' + +c, d' + bb +c].
(23)
< maxx(a++0
Since [6'] < have // + 2 0 and hence a a' Sata' a' + b +e S max + d' + + be b' b+ max(a+dd +b' b a' + +c} and finally we have max(a+d) .' bbbb) - Clearly when a' = 1/ -c, the equal sign holds.
+e). Similarly,
+0 +ollelttt''ttttttoooeeee +c}.
Proof for Theorem 1. First, due to symmetry, optimizing (17) is equivalent to optimize
En max (max (x 0), Sw (x: + o) | j#1
(24)
Further, by Lemma flemmardelta we can replace 8 with its optimal value and transform the optimization objective above as
X |20 1
E
EaND, (max fao (0)5 - San (+)+)) 3#1
(25)
where Dj is the adversarial data distribution:
15

---

Identifying and Understanding Cross-Class Features in Adversarial Training
EEs N( j=1
&cs (Mu- 1#1
(26)
i#1'
j=1
Now we calculate the expectation in (25).
E BD [max (8); - mm (+):) ] + X 1
=E E, [max(wic+ + w2e + u23c,3, WiC + uye + w2Fc.2) w1 w2(0022 &c,3)] 11001
=bgnb, [Wot + u2e + wy max (c22) wI u2(40.22 ft,3) + =2 1101
(27)
=W16 + u2e + w2 [max fc22)] + E. BBDD -11 u2(00,,2 + +.,,3) + ||
X w 2.
=wiet + wye + wyBBabb [max(0o,3 fc22) + [- - - 2w02 c)]
2
Finally, since Sc,,, fc2 ~ (μ - e, ) and they are independent, by Lemma flemma:z we have
E Elmax( BC.3 - (1r -c) c) CC. - (1s
(28)
VF
hence E, [max(fc,,,, tc,2)) = - VF.
Therefore, the optimization objective can be simplified as
X
(((w) (- 26)wi + (-u 2e
Jul +
[w) + +
(29)
V
2
For 02, we have
Oc
-
Aw2
(30)
Owz
V
Recall that < VT.. Let 6g ( -)
E (0, By analyzing the sign (30), it is clear that for E (0, 60), the
optimal Wa for minimizing the loss function (29) is
/ 2e V
W2
(31)
However, for e E (co. 1), ac is always negative, thus the returned by AT is W2 = 0 under the constraint W/ 2 0.
Owy
By contrast,
ac - 2e Aw1,
(32)
Ow
and for (0, 4), the optimal wa for minimizing the loss function (29) is always positive:
W1 /1 0.
(33)
This ends our proof.
E.3. Proof for Theorem 2
Theorem 2 For any w > 0 and e E (0, the adversarial examples from any other given class.
)) w2 E [0, wi). a larger w2 increases the possibility of the model distinguishing
To prove Theorem 2, we need the following lemma.
16

---

Identifying and Understanding Cross-Class Features in Adversarial Training
Lemma 3. Suppose that X, K,Y (1, of) and they are independent, 01 > Let Z X + IY where L > 0. Denole
((t) = Pr(Zz 0), then u(t) is monotonically increasing t for E [0,
proof. Note that Z, = x + tY ~ NV + {))))
Thus, the distribution function of Ze is 0e(z) 2 ((1)))))) and
-1-t $
1+t
u(t) 1 (0))) Φ0
V1 P.0
100001
d'(t) pl
1tt
VI+ P-1 (1+
vI+K
101
p(
1+t
(1+ (1+
(34)
VI+to1
(1+8)0}
VI+ (1+ v
1+t
1-t
pl V1+ + t01 (1+ t
Therefore, for t E (0, (t) 0 and u(t) is monotonically increasing t for to E [0, 1]. Proof for Theorem 2. Due to symmetry, it's suffice to show that given w1, for w2 E [0, wil, the probability
(T) > (i)22)), a DD
(35)
is monotonically increasing at w2. Note that
fw (+) fw (8)2 = w1 ( -3,2) w2($c,2 &c,1),
g dg ~ NNI - 2c, 20,
(36)
.c fc,1 NN - 2e, "12,-2,20" 20
By dividing w1 (n- 2c), and let i probability
w2 X wi
$8.1- and Y 20.--0.1 from Lemma flemma:t we know that the p-2e
2-
(r(f (2)2 > 0)
(37)
monotonically increasing at w2 wi
and hence increasing at W2. This ends our proof.
E.4. Proof for Theorem 3 and Corollary
Simplification of knowledge distillation as label smoothing. In this context, the term 'symmetry' specifically refers to the symmetry of logits for the other two classes when taking the expectation in the loss function (equation 10). When considering data from class y, both the distribution of features 3g, and æc, for the other two classes, as well as their respective weights W1 and W2, exhibit symmetry respectively. Consequently, after applying knowledge distillation, the expectation for logits of the other two classes in the objective loss function becomes identical. To simplify this process, we can employ label smoothing.
We prove Theorem 3 and Corollary 1 in the following. Recall that we define the robust loss under knowledge distillation as
Lis(fw) = E,(EE-D (1 - I)
[ max (max fw + 8), - faw (x + +))11 116 1108 x Se jfi
Σ fw + +))g} + 2 w 2 jイ
(38)
Theorem 3 Consider AT with knowledge distillation loss (38). There exists an > co, such that for e E (0, e1 the output function obtains W2 > 0; for e E (f1, 44), the output function returns W2 = O.
Proof for Theorem 3. Similar to the proof for Theorem 1, the optimization objective (38) can be simplified as
X
(is(fw) = (1 ()((3+((+++))) + 2e)w1 + (-p + 2 + V
Juzl - -Blew1 + [wwl]
2 (wf + w3)
(39)
2 (w/ wē).
= [11 - B8p + (2 33)c|ws w1 [[1 - 3)
/ u/wu
2 4 + Awa,
Thus
Lis (1 (1- B)(e
σ
(40)
v
002
17

---

Identifying and Understanding Cross-Class Features in Adversarial Training
and let 61 ( ( cg, similar to the analysis for eg, have for (1, the output function obtains w2 >>
for (c1, 21). the output function returns W2 0. This ends proof.
Corollary 1 Let (E) be the value of w2 returned by AT with (17), and 2 be the value of w2 returned by label smoothed loss (38). Then, for E E1), we have w (e) > (E).
Proof for Corollary 1. For e E (0, f1), by analysing the sign of (40), we have
# 32 + 7F
(e)
(41)
and recall that in the proof for Theorem 1 we have
(G( - (2e + v
(42)
thus is clear that
B(2c + V
u2 ( (c) w5(())
>0.
(43)
This ends our proof.
E.5. Extension to higher dimensions
In Section 4, into cross-class features, since binary classification is incapable of handling cross-class features, they do not influence the binary classification result. Therefore, we explored a ternary classification problem with minimum dimensions in our framework.
6-dimensional data representation because it is the smallest dimension for illustrating insights
When considering extension to higher dimensions, our theory can be extended to more feature dimensions for ternary classification. For each original feature FF. or JC,j we can extend them = or =č.; where 1, 1,2, K, thus resulting in 6K feature dimensions. Accordingly, we also have corresponding parameters w and w for h: = 1,2, K. Based on this extended model, we can derive similar results in Theorems and 2, where the bounds are set for w and wy. This can be easily derived through calculating the optimal perturbation e with Lemma 2, calculating the optimizing objectives like Equation (26), and finally deriving the solution of w and w for minimizing the objectives like equations (27) and (29).
18


Reviews:
[{"review_id": "ThsZr6YfY7", "reviewer": "Reviewer_HgFV", "summary": "This paper proposed a novel perspective to understand adversarial training. By splitting features into cross-class features and class-specific features and investigating model learning behaviors on cross-class features, this paper demonstrated the importance of cross-class features in improving model robustness. Based on that, this paper also provided an interpretation of robust overfitting and the advantage of soft-label in adversarial training based on cross-class features.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer HgFV,\n\nThank you for your valuable feedback. We address your concerns below.\n\n---\n\n**Q1**: It would be better if authors could discuss, based on the findings in this work, any possible ways to develop advanced adversarial training methods.\n\n**A1**: Thank you for the thoughtful comment. We list some potential directions for further applications of our theory as follows:\n\n- **Data (re)sampling**. Since more generated data is prone to be helpful for advancing adversarial robustness [1], it requires significantly more data and computational costs. From the cross-class feature perspective, adaptively sampling generated data **with considerations of class-wise relationship** may improve the efficiency of large-scale AT and decrease the forgetting of cross-class features.\n- **AT configurations**. Customizing AT configurations like perturbation margins or neighborhoods is useful for improving robustness [2]. Since cross-class features are more sensitive against robust loss, customizing AT configurations for different samples or classes **based on class-wise relationships** may mitigate this sensitivity and further improve robustness, as shown in Theorem 1.\n- **Robust module design:** The architecture of a model and the mechanisms of activation play a crucial role in adversarial robustness [3]. Therefore, designing modules that either implicitly or explicitly emphasize cross-class features may enhance robustness. For example, calibrating channel activation can improve robustness [4], thus creating activation mechanisms that preserve more cross-class features can further contribute to this improvement.\n\nIn addition to these AT algorithms, we would also like to highlight the **theoretical modeling potential** of our work. Similar to the robust/non-robust feature decomposition [5], which has been applied in many subsequent theoretical works, e.g. [6,7,8], our cross-class feature model has the potential for more in-depth modeling of adversarial robustness, contributing new tools in its theoretical analysis.\n\n[1] Better Diffusion Models Further Improve Adversarial Training. ICML 2023\n\n[2] CAT: Customized Adversarial Training for Improved Robustness. IJCAI 2022\n\n[3] Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. BMVC 2023\n\n[4] Improving Adversarial Robustness via Channel-wise Activation Suppressing. ICLR 2021\n\n[5] Adversarial examples are not bugs, they are features. NeurIPS 2019\n\n[6] On the Tradeoff Between Robustness and Fairness. NeurIPS 2022\n\n[7] Understanding the Impact of Adversarial Robustness on Accuracy Disparity. ICML 2023\n\n[8] Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data. ICLR 2025\n\n---\n\n**Q2**: As authors' claimed, after some checkpoint, the trained model reduce its reliance on cross-class features, which result in the reduced robustness on test data; so could we understand this phenomenon in another way, i.e. some class-specific features are naturally contradicted with cross-class features, as cross-class features are helpful for model robustness, the robustness of model will be reduced if model starts learning those contradicted class-specific features after some checkpoint.\n\n**A2**: Thank you for the alternative interpretation of our theory, which aligns well with our understanding. There may indeed be some contradiction between cross-class and class-specific features, caused by various factors such as model capacity or feature overlap. As a result, learning more class-specific features for lower training robust loss can lead to a decrease in cross-class features, as discussed in Section 3.2. Thank you once again for your insightful comment; we will incorporate this discussion into our revision.\n\n---\n\nWe truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "The main hypothesis of this work is that cross-class features are very helpful for improving robust generation of model trained by adversarial training. However, during adversarial training, the model only learns cross-class features at the initial stage and gradually ignores these features after some checkpoint. Authors conduced both theoretically analysis and empirically studies to support this hypothesis.", "methods_and_evaluation_criteria": "This work utilized benchmark datasets to conduct empirical studies and proposed a new metric Class Attribution Similarity (CAS) to evaluate the usage of cross-class features during adversarial training.", "theoretical_claims": "I quickly went through proofs provided in the supplementary material.", "experimental_designs_or_analyses": "Both theoretical analysis and empirical studies strongly support authors' hypothesis about the role of cross-class features played in adversarial training.", "relation_to_broader_scientific_literature": "This work provided a novel perspective to understand adversarial training, which might be able to inspire future works to develop more advanced adversarial training based method to improve model robustness.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Pros: \n1. This work provided a novel perspective to understand adversarial training and presented a hypothesis about it by investing cross-class features.\n2. Both theoretical analysis and empirical studies strongly support authors' hypothesis and also explain the robust overfitting problem and the advantage of soft-label in adversarial training.\n\nCons: \n1. It would be better if authors could discuss, based on the findings in this work, any possible ways to develop advanced adversarial training methods.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. As authors' claimed, after some checkpoint, the trained model reduce its reliance on cross-class features, which result in the reduced robustness on test data; so could we understand this phenomenon in another way, i.e. some class-specific features are naturally contradicted with cross-class features, as cross-class features are helpful for model robustness, the robustness of model will be reduced if model starts learning those contradicted class-specific features after some checkpoint.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UO11ZWkwgn", "reviewer": "Reviewer_oCW6", "summary": "While successful at defending models against adversarial examples, the dynamics of adversarial training (AT) are poorly understood. This paper attempts to explain two properties of AT: robust overfitting, and the utility of soft labels over one-hot labels. These properties are studied through the lens of cross-class features, which are features used by the model for classification which are shared by multiple classes. It is shown that a well-fit robust model displays significant correlations between features for different classes; however, when the model is overfit, these cross-class features largely disappear. It is hypothesized that this contributes to the increase in test loss that is characteristic of robust overfitting. Furthermore, it is hypothesized that soft-label methods like knowledge distillation prevent the loss of cross-class features and therefore mitigate robust overfitting. Additional experimental results are provided to support these hypotheses. A theoretical model for adversarial training is thes described which illustrates the utility of cross-class features in robust classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oCW6,\n\nThank you for your valuable feedback. We address your concerns below.\n\n---\n\n**Q1**:  Implementation details\n\n**A1**: Thank you for your careful reading. We are committed to publishing our code upon publication. For implementation details regarding model training, we utilize the default hyperparameters according to a well-known adversarial training repository [1] listed below:\n\n| Parameter | |\n| --- | --- |\n| Train epochs | 200 |\n| SGD Momentum | 0.9 |\n| weight decay | $5\\times 10^{-4}$ |\n| initial learning rate | 0.1 |\n| learning rate decay | 100, 150-th epochs (decay rate=0.1) |\n| training adversary | 10-step PGD |\n\nWe will make these details clear in our revision. Thanks again for the reminder!\n\n[1] Bag of Tricks for Adversarial Training, ICLR 2021, https://github.com/P2333/Bag-of-Tricks-for-AT.\n\n---\n\n**Q2**: Saliency map visualizations.\n\n**A2**: Thank you for your thoughtful comment. To show these examples are not cherry-picked, we will include a full page of visualization examples (ordered by original sample ID) in the appendix of our revision, where many saliency maps of these examples still exhibit such properties. However, we acknowledge that **not all** samples enjoy such clearly interpretable features (e.g., *wheels* shared by *automobiles* and *trucks*), since features learned by neural networks are subtle and do not always align with human intuition, including cross-class features. Thus, we will tone down related claims made in that paragraph and highlight that the saliency maps are only presented to help understand the concept of cross-class features.\n\n---\n\n**Q3**: The theoretical data model in Section 4 is fixed to 6 dimensions. What was the reason for choosing a 6-dimensional data representation? Would any complications arise when trying to extend the theoretical results to higher dimensions?\n\n**A3**: Thanks for the thoughtful comment. First, we acknowledge that using a  6-dimensional data representation is the smallest dimension for illustrating our insights into cross-class features, since binary classification is not capable of handling cross-class features as they do not influence the binary classification result. Therefore, we explored a ternary classification problem with minimum dimensions in our framework.\n\nFollowing your suggestion, we attempted to extend our theory to more feature dimensions for ternary classification . For each original feature $x_{E,j}$ or $x_{C,j}$, we can extend them to $x_{E,j}^k$ or $x_{C,j}^k$, where $k=1,2,\\cdots,K$, thus resulting in $6K$ feature dimensions. Accordingly, we also have corresponding parameters $w_1^k$ and $w_2^k$ for $k=1,2,\\cdots, K$. Based on this extended model, we can derive similar results in Theorems 1-3, where the bounds are set for $w_1^k$ and $w_2^k$. The proof sketch includes deriving the optimal perturbation $\\epsilon$ with Lemma 2, calculating the optimizing objectives like equation (26), and finally derive the solution of $w_1^k$ and $w_2^k$ for minimizing the objectives like equation (27) and (29). Due to space limitations, we are unable to present all proof details here, but we will include this extension in the appendix of our revision. Thank you again for the valuable suggestion!\n\n---\n\n**Q4**: The term \"feature\" feels overloaded in different sections of this paper.\n\n**A4**: Thank you for your careful reading. We clarify the definition of “features” as the activation in a particular layer (feature extractor), as stated in Section 3.1. Thus, the feature is model-dependent under this definition.\n\nRegarding saliency maps and Section 4, we apologize for the potential confusion and will replace the term “features” as follows. For saliency maps, we will use the term “class-specific discriminative regions” that was used in the original paper of [2] GradCAM for the highlighted regions. For $x_{E,j}$ and $x_{C,j}$ in Section 4, we will define them as “attribution” instead of features to distinguish them. Thanks again for your kind reminder, we will revise these terms in our revision.\n\n[2] Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. IJCV 2019\n\n---\n\nWe truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "I think the major claims made in this paper are backed up by the evidence presented.", "methods_and_evaluation_criteria": "I think the methods and evaluation are largely consistent with the goals of the paper, but some minor issues I have are included in the strengths and weaknesses section below.", "theoretical_claims": "I didn't read the proofs in detail, but I did not find any obvious errors.", "experimental_designs_or_analyses": "I think the experimental design and analyses presented here are sound. Multiple datasets, architectures, adversarial attacks, and training methods are tested. However, many implementation details are missing, and I don't see a link to a repository with code, so it might be difficult to replicate the experiments.", "relation_to_broader_scientific_literature": "This paper studies the problem of adversarial training, which is the focus of a large body of work, and looks specifically at the phenomenon of robust overfitting. This paper connects robust overfitting to the model's use of robust features, discussed in Tsipras et al. and Ilyas et al., and makes use of feature similarity and visualization techniques to explore this connection.", "essential_references_not_discussed": "I am not aware of essential references that are not discussed here.", "other_strengths_and_weaknesses": "Strengths\n- The description of cross-class features is novel and, according to the experimental results presented, appears to at least partially explain the phenomenon of robust overfitting.\n- The feature attribute correlation matrices are a very clear visualization of the phenomenon being described.\n- The hypotheses presented are backed up by both experimental and theoretical results.\n- The insights from this paper could help inspire further advances in adversarial training. For example, new knowledge distillation methods could be designed that better encourage the maintenance of cross-class features.\n\nWeaknesses\n- I find the saliency map visualizations to be unconvincing. It's not clear from figure 3 that these are not cherry-picked examples, and it's difficult to assess the rigor of the claims made in that paragraph.\n- The theoretical data model in Section 4 is fixed to 6 dimensions. While the results support the hypotheses posed in previous sections, there isn't any justification as to why we would expect these results to hold for different dimensions.\n- The term \"feature\" feels overloaded in different sections of this paper. In section 3.1, features are defined to be the activations of a feature extractor, while in the saliency map experiments a feature is some abstract quality of the input (i.e. the wheels on a car), while in section 4 a feature is a column in the dataset. I think it would benefit the paper to have a concrete definition of what a feature is in this context (and in particular, whether features are model-dependent).", "other_comments_or_suggestions": "The paper is well written, I didn't find any typos.", "questions_for_authors": "1. What was the reason for choosing a 6-dimensional data representation? At the moment, I'm assuming it's just because that's the smallest dimension with an interesting cross-class feature structure.\n2. Would any complications arise when trying to extend the theoretical results to higher dimensions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1RYtNmlD0", "reviewer": "Reviewer_v6k6", "summary": "This paper explores a unique characteristic of adversarial training from the perspective of class-wise feature attribution. Specifically, it highlights that data often contain **cross-class features**, such as the feature of wheels shared by the automobile and truck classes in the CIFAR-10 dataset. \n\nThe authors discover that as training progresses and the robust loss decreases beyond a certain threshold, the model begins to abandon cross-class features. Consequently, the model makes decisions primarily based on class-specific features rather than cross-class ones, which contributes to improved robustness.\n\nThrough their experiments, they demonstrate that this phenomenon can be observed across various adversarial training setups.\n\n## update after rebuttal\n\nThe insights are valuable and merit inclusion in the main paper, though their earlier presence—particularly the discussion on applications—would have strengthened the submission. Therefore, I maintain my score in favor of acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer v6k6,\n\nThank you for your valuable feedback. We address your concerns below.\n\n---\n\n**Q1**: **A more solid explanation for Figure 8 is needed**\n\n**A1**: Thank you for your thoughtful comment. First, we would like to clarify that Figure 8 aims to show that our theory for adversarial training (AT) is compatible with standard training (ST), where the overfitting issue is far less than robust overfitting in AT. This is supported by the fact that both the accuracy and CAS of ST do not change as significantly as AT in the latter training stage. As for the overfitting in ST stated in [1], it may have other mechanisms like sample memorization [A], but we kindly note that such understandings are not within the scope of our study, since our theory focuses on particular properties of AT. We will add this discussion in our revision.\n\n[A] The Pitfalls of Memorization: When Memorization Hurts Generalization. ICLR 2025\n\n---\n\n**Q2**: **The potential use of cross-class features**\n\n**A2**: Thanks for the insightful comments. Due to space limitations, please kindly refer to our [response](https://openreview.net/forum?id=FvBYG5jA7k&noteId=ryP80kF51O) to Q1 by Reviewer HgFV for a discussion on the future directions of cross-class features.\n\n---\n\n**Q3**: **Regarding catastrophic overfitting**\n\n**A3**: Thank you for the insightful comment. Following your suggestion, we implemented a 200-epoch fast adversarial training with 1 step PGD attack, $\\ell_\\infty$-norm $\\epsilon=8/255$ on PreActResNet-18 (same as the main observation experiment in our paper). The CAS for different stages is presented as follows:\n\n| Epoch | 50 | 100 | 150 | 200 |\n| --- | --- | --- | --- | --- |\n| CAS | 16.4 | 18.5 | 7.3 | 2.8 |\n| Robust Accuracy (PGD-10) % | 38.1 | 40.9 | 27.4 | 0.0 |\n\nThe results also validate that the forgetting of cross-class features (measured by CAS) can be detected in the catastrophic overfitting of fast adversarial training, which aligns with the main discovery of our paper. We will include these results and related code, figures in our revision. Thanks again for raising this point!\n\n---\n\n**Q4**: Class names displayed in the figures\n\n**A4**: Thanks for the kind suggestion. We will add these class labels to the saliency maps in our revision.\n\n---\n\n**Q5**: Please ensure consistency in notation.\n\n**A5**: Thanks for the careful reading. We will unify them as $A_i$ in our revision.\n\n---\n\nWe truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "The main claim of the paper can be summarized as follows:\n\n**(Main claim)** Cross-class features are ignored after the optimal point during adversarial training, leading to adversarial overfitting.  \nThe main evidence supporting this claim is presented in Figure 2 and Figure 8. Since the paper focuses on the unique characteristics of adversarial training, Figure 8, which illustrates the tendency of standard training, is highly significant. The results demonstrate that adversarial and standard training exhibit different behaviors (so I personally recommend that the authors mention Figure 8 earlier in the paper). Furthermore, the definition of Class Attribution Similarity (CAS) is quite reasonable. The experiments cover various models and datasets, which validates their claim.\n\nI have several questions for the authors:  \n1) **A more solid explanation for Figure 8 is needed**\nIn Section 5.3, the authors explain why standard training does not exhibit a similar tendency:  \"This observation is consistent with the characteristic of standard training, which generally does not exhibit overfitting.\" However, as empirically demonstrated in [1], standard training also suffers from overfitting. Please provide a more solid explanation for this observation.  \n\n2) **The potential use of cross-class features**\nWhile I acknowledge the importance of this observation, the paper does not provide the benefits of knowledge distillation (or soft-labeling) in terms of adversarial robustness. While they can mitigate the observed phenomenon, it seems they are not effective in improving the robsut accuracy based on Figure 3. Furthermore, class-aware features are difficult to identify, as noted in [2]. Therefore, they cannot be directly used as a regularizer or training trick for adversarial training. How, then, can this insight be leveraged to improve adversarial robustness? If there exists at least a potential direction, it would be valuable to explore these phenomena further.  \n\n3) **Regarding catastrophic overfitting**\nWhile the current observation on multi-step adversarial training is quite interesting, catastrophic overfitting is a well-known issue in adversarial training [3-4]. Can the observed phenomenon also be detected in the single-step adversarial training framework?  \n\n**Suggestions:**\n1) Instead of using numerical labels in all figures, using class labels (e.g., \"truck\") would provide a more intuitive understanding for readers. As the authors mentioned, automobile (label=1) shows high Class Attribution Similarity (CAS) with truck (label=9). If class names were displayed in the figures, the observations would be clearer.  \n2) In Equation (5), \\( A \\) is indexed as \\( A_i \\), but in Equation (6), it changes to \\( A^i \\). Please ensure consistency in notation.  \n\n\n\n- [1] Jiang, Yiding, et al. \"Fantastic generalization measures and where to find them.\" arXiv preprint arXiv:1912.02178 (2019).\n- [2] Tsipras, Dimitris, et al. \"Robustness may be at odds with accuracy.\" arXiv preprint arXiv:1805.12152 (2018).\n- [3] Wong, Eric, Leslie Rice, and J. Zico Kolter. \"Fast is better than free: Revisiting adversarial training.\" arXiv preprint arXiv:2001.03994 (2020).\n- [4] Kim, Hoki, Woojin Lee, and Jaewook Lee. \"Understanding catastrophic overfitting in single-step adversarial training.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021.", "methods_and_evaluation_criteria": "The experiments cover various models and datasets, which validates the observation of the paper.", "theoretical_claims": "I've checked all theoretical claims and verified there is no problem.", "experimental_designs_or_analyses": "Refer to Claims And Evidence.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Refer to Claims And Evidence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oO3nAvd4SI", "reviewer": "Reviewer_RcTN", "summary": "Adversarial Training (AT) is a widely adopted technique for enhancing the robustness of deep learning models against adversarial examples. However, a critical challenge associated with AT is robust overfitting. As training progresses, the robust accuracy on the training set continues to improve, yet the robust accuracy on the test set stops increasing and instead begins to decline.\nIn this paper, the authors investigate the phenomenon of robust overfitting through the lens of class-wise feature attribution. They observe that the decline in robust accuracy occurs when adversarial training (AT) shifts its focus away from cross-class features and instead relies solely on class-specific features. Based on this observation, they hypothesize that robust overfitting arises due to the reduced reliance on cross-class features during AT. To support their hypothesis, they provide both theoretical analysis and extensive empirical evidence.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer RcTN,\n\nThank you for your valuable feedback. We address your concerns below.\n\n---\n\n**Q1**: A concern I have is the paper’s discussion on its significance for future research. While previous work may not have fully understood how AT utilizes class features, we still have an intuitive understanding of the underlying logic. Even with the clearer evidence provided, the challenge remains in how to preserve reliance on cross-class features while continuing to improve the robustness of the model.\n\n**A1**:  Thank you for the thoughtful comment. We list some potential directions for further applications of our theory as follows:\n\n- **Data (re)sampling**. Since more generated data are prone to be helpful for advancing adversarial robustness [1], it requires significantly more data and computational costs. From the cross-class feature perspective, adaptively sampling generated data **with considerations of class-wise relationship** may improve the efficiency of large-scale AT and decrease the forgetting of cross-class features.\n- **AT configurations**. Customizing AT configurations like perturbation margins or neighborhoods is useful for improving robustness [2]. Since cross-class features are more sensitive against robust loss, customizing AT configurations for different samples or classes **based on class-wise relationships** may mitigate this sensitivity and further improve robustness, as shown in Theorem 1.\n- **Robust module design:** The architecture of a model and the mechanisms of activation play a crucial role in adversarial robustness [3]. Therefore, designing modules that either implicitly or explicitly emphasize cross-class features may enhance robustness. For example, calibrating channel activation can improve robustness [4], thus creating activation mechanisms that preserve more cross-class features can further contribute to this improvement.\n\nIn addition to these AT algorithms, we would also like to highlight the **theoretical modeling potential** of our work. Similar to the robust/non-robust feature decomposition [5], which has been applied in many subsequent theoretical works, e.g. [6,7,8], our cross-class feature model has the potential for more in-depth modeling of adversarial robustness, contributing new tools in its theoretical analysis.\n\n[1] Better Diffusion Models Further Improve Adversarial Training. ICML 2023\n\n[2] CAT: Customized Adversarial Training for Improved Robustness. IJCAI 2022\n\n[3] Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. BMVC 2023\n\n[4] Improving Adversarial Robustness via Channel-wise Activation Suppressing. ICLR 2021\n\n[5] Adversarial examples are not bugs, they are features. NeurIPS 2019\n\n[6] On the Tradeoff Between Robustness and Fairness. NeurIPS 2022\n\n[7] Understanding the Impact of Adversarial Robustness on Accuracy Disparity. ICML 2023\n\n[8] Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data. ICLR 2025\n\n---\n\n**Q2**: I suggest that the authors provide more detailed explanations of the concepts introduced in their paper. For instance, the terms \"robust accuracy\" and \"robust loss\" are used without further clarification. I assume these terms refer specifically to the accuracy and loss measured on adversarial examples, but it would be helpful for the authors to explicitly define them for clarity.\n\n**A2**: Thanks for your careful reading. Your assumption is correct, we will clarify these definitions in our revision as follows:\n\n- **Robust accuracy** is the accuracy of the model on adversarial examples.\n- **Robust loss** is the average cross-entropy loss of the model on adversarial examples.\n\n---\n\nWe truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "I find the authors’ evidence convincing, as they provide solid theoretical proofs for each theorem they propose. Additionally, they validate their findings through extensive experiments conducted across multiple datasets.", "methods_and_evaluation_criteria": "The metrics and benchmark datasets used in their experiments are appropriate.", "theoretical_claims": "I have not specifically verified the correctness of their theoretical proofs.", "experimental_designs_or_analyses": "The authors conduct their experiments on various classification datasets, including CIFAR-10 and CIFAR-100, to visualize feature attribution correlations at different training stages. Their experimental results effectively support their findings, demonstrating that AT tends to reduce reliance on cross-class features as the model becomes overfitted.", "relation_to_broader_scientific_literature": "This work primarily explores why traditional AT suffers from robust overfitting and how AT with smooth label can mitigate this issue. By highlighting the importance of preserving cross-class features, the study provides valuable insights that could guide future research toward improving AT and enhancing model robustness.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "The strength of this paper lies in its solid evidence and thorough explanation of why Adversarial Training (AT) suffers from robust overfitting. It also emphasizes the importance of utilizing cross-class features, demonstrating how this approach can help alleviate the issue and improve the effectiveness of AT. However, a concern I have is the paper’s discussion on its significance for future research. While previous work may not have fully understood how AT utilizes class features, we still have an intuitive understanding of the underlying logic. Even with the clearer evidence provided, the challenge remains in how to preserve reliance on cross-class features while continuing to improve the robustness of the model.", "other_comments_or_suggestions": "I suggest that the authors provide more detailed explanations of the concepts introduced in their paper. For instance, the terms \"robust accuracy\" and \"robust loss\" are used without further clarification. I assume these terms refer specifically to the accuracy and loss measured on adversarial examples, but it would be helpful for the authors to explicitly define them for clarity.", "questions_for_authors": "none", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel perspective on adversarial training by identifying and analyzing cross-class features—features shared across multiple classes—and their role in robust model behavior. The study combines theoretical insights with comprehensive empirical evaluations, showing that models rely on cross-class features early in training but tend to abandon them in later stages, contributing to robust overfitting. Reviewers found the hypothesis compelling and well-supported, with strengths including clear visualizations, a new attribution metric (CAS), and relevance to known AT phenomena such as soft-label benefits. Some concerns were raised about practical applicability, overloaded terminology (particularly around "features"), and the interpretability of saliency maps. In the rebuttal, the authors addressed these points effectively by clarifying definitions, outlining concrete application directions, and extending experiments to single-step AT to show generality. Given the conceptual novelty, strong empirical support, and thoughtful engagement with reviewer feedback, the overall recommendation is to accept. To strengthen the final version, the authors are encouraged to improve clarity in terminology, integrate practical applications into the main text, and ensure visual aids are more intuitive.
Decision: Accept (poster)