Understanding Catastrophic Overfitting in Fast Adversarial Training From a Non-robust Feature Perspective
Authors: Chaoning Zhang
Chenshuang Zhang
Kang Zhang
Axi Niu
In So Kweon
Chang D. Yoo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Fast Adversarial Training
Catastrophic Overfitting
Non-robust Feature

Abstract: To make adversarial training (AT) computationally efficient, FGSM AT has attracted significant attention. The fast speed, however, is achieved at the cost of catastrophic overfitting (CO), whose reason remains unclear. Prior works mainly study the phenomenon of a significant PGD accuracy (Acc) drop to understand CO while paying less attention to its FGSM Acc. We highlight an intriguing CO phenomenon that FGSM Acc is higher than accuracy on clean samples and attempt to apply non-robust feature (NRF) to understand it. Our investigation of CO by extending the existing NRF into fine-grained categorization suggests: there exists a certain type of NRF whose usefulness is increased after FGSM attack, and CO in FGSM AT can be seen as a dynamic process of learning such NRF. Therefore, the key to preventing  CO lies in reducing its usefulness under FGSM AT, which sheds new light on understanding the success of a SOTA technique for mitigating CO.


Under review as a conference paper at ICLR 2023
UNDERSTANDING CATASTROPHIC OVERFITTING IN FAST ADVERSARIAL TRAINING FROM A NON-ROBUST FEATURE PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
To make adversarial training (AT) computationally efficient, FGSM AT has at- tracted significant attention. The fast speed, however, is achieved at the cost of catastrophic overfitting (CO), whose reason remains unclear. Prior works mainly study the phenomenon of significant PGD accuracy (Acc) drop to understand CO while paying less attention its FGSM Ace. We highlight intriguing CO phenomenon that FGSM Ace is higher than accuracy on clean samples and attempt apply non-robust feature (NRF) understand it. Our investigation of CO by extending the existing NRF into fine-grained categorization suggests: there exists a certain type of NRF whose usefulness is increased after FGSM attack, and CO in FGSM AT can be seen as a dynamic process of learning such NRF Therefore, the key preventing CO lies reducing its usefulness under FGSM AT, which sheds new light on understanding the success of SOTA technique for mitigating CO.
INTRODUCTION
Despite impressive performance, deep neural networks (DNNs) (LeCun et al. 2015 He et al. 2016
Huang et al., 2017 Zhang et al. 2019a 202T) are widely recognized to be vulnerable to adversarial examples (Szegedy et al. 2013 Biggio et al. 2013 Akhtar & Mian 2018). Without giving false sense of robustness against adversarial attacks (Carlini & Wagner 2017 1 Athalye et al. 2018 Croce & Hein 2020), adversarial training (AT) (Madry et al. [2018] /hhng et al. (219cc has become the de facto standard approach for obtaining an adversarially robust model via solving a min-max problem in two-step manner. Specifically, it first generates adversarial examples by maximizing the loss, then trains the model on the generated adversarial examples by minimizing the loss. PGD-N AT (Madry classical AT method, where N the iteration steps when generating the adversarial samples in inner maximization. Notably, PGD-N AT is N times slower than its counterpart standard training with clean samples. straightforward approach make AT faster is to set N to 1, i.e reducing the attack in the inner maximization from multi-step PGD to single-step FGSM (Goodfellow al. 2015). For simplicity, PGD-based AT and FGSM-based fast AT are termed PGD AT and FGSM AT. respectively.
al., 2018 Zhang et al. 2019c)
FGSM AT often fails with a sudden robustness drop against PGD attack while maintaining its robustness against FGSM attack, which is called catastrophic overfitting (CO) (Wong et al. 2020). With Standard Acc denoting the accuracy on clean samples while FGSM Acc and PGD Acc indicating the accuracy under FGSM and PGD attack, emphasize that CO model is characterized by two main phenomena as follows.
Phenomenon 1: The PGD Acc drops a value close zero when CO happens (Wong et al. 2020 Andriushchenko & Flammarion 2020).
Phenomenon 2: FGSM Acc higher than Standard Ace for a CO model (Kmm et al. 2020
Andriushchenko & Flammarion 2020).
Multiple works (Wong et al. 2020 Kim et al. 2020: Andriushchenko & Flammarion 2020 have
focused understanding CO by explaining the drop of PGD Acc in Phenomenon however, they pay less attention to Phenomenon 2 regarding FGSM Acc. Specifically for Phenomenon 1, FGSM-RS (Wong et al. 2020) attributes it to the lack of perturbation diversity in FGSM AT, which

---

Under review as a conference paper at ICLR 2023
is refuted by follow-up GradAlign (Andriushchenko & Flammarion 2020) by demonstrating co-occurrence of local non- non-linearity and the PGD Acc drop. However, these understandings cannot explain why FGSM Acc is higher than Standard Acc for CO model in Phenomenon 2.
In the context of adversarial learning, numerous works (Goodfellow et al., 2013 Tabacof & Valle 2016 Tanay & Griffin 2016 Koh & Liang 2017 Nakkiran [019 Athalye et al. 2018 Zhang et al.
2020) have attempted to explain why adversarial examples exist from different angles, among which
non-robust feature (NRF) (Ilyas et al.)
2019) a popular one which also aligns well with all other
explanations (Goodfellow et al. 2015: Tabacof & Valle 2016, Tanay & Griflin 2016, Koh & Liang
2017 Nakkiran 201 Athalye et al., 2018). Such compatibiliyy suggests that the NRF perspective
constitutes an essential tool for understanding adversarial vulnerability, to which CO is also directly
related. Specifically, the authors of (Ilyas et 2019) define the positive-correlatin between features and true labels as feature usefulness (see Section 3.1 for more detailed definitions). Therefore, the adversarial vulnerability of DNNs attributed the existence of non-robust features (NRFs), which can be made anti-correlated with the true label under adversary. This understanding of NRFs in (Ilyas) ct al. 2019) well aligns with the fact that a CO model achieves close to zero robustness against PGD attack, and thus motivates to believe that the NRF perspective might be auspicious direction for understanding CO in FGSM AT.
The NRF in Qlyas et al. 2019) is defined with PGD attack, which is followed in this work; how- ever, we extend their NRF framework by additionally considering FGSM attack for fine-grained categorization. Considering the difference of adversarial attack strength between FGSM and PGD) attack, GradAlign (Andriushchenko & Flammarion 2020) explains Phenomenon 1 by demonstrating how well the attack variant (FGSM PGD attack) can solve the inner maximization problem in AT. We start our investigation by providing an alternative interpretation of this adversarial strength difference between the two attack variants within the NRF framework (Iyass et al., 2019). named strength- NRF categorization. Despite aligning well with Phenomenon 1, We find that this strength-basee categorization cannot explain Phenomenon since the usefulness of these NRFs is decreased under FGSM attack and leads to an decrease (instead of increase in Phenomenon 2) of classification accuracy on FGSM adversarial examples than clean samples.
To understand Phenomenon 2 in CO from the NRF perspective, we conjecture that there exists type of NRF whose usefulness is increased under FGSM attack, thus can lead to a higher FGSM Acc than Standard Acc (Phenomenon 2). In other words, if such type of NRFs (NRF2 in the following categorization) exists, Phenomenon 2 can be justified. Considering whether the usefulness is decreased or increased under FGSM attack, we propose direction-based NRF categorization where NRF2 (NRF1) leads to the increase (decrease) of classification accuracy under FGSM attack. To prove the existence of NRF2,
follow the procedure of verifying the existence of NRF in (Iyaas) cet al. 2019). Moreover, we show that NRF2 can cause a significant PGD Acc drop. which also helps justify Phenomenon I in CO.
Overall, towards understanding CO in FGSM AT, our contributions are summarized as follows:
Our work shifts the previous focus on PGD Acc in Phenomenon I FGSM Acc in Phenomenon 2 for understanding CO. Given NRF as a popular perspective on adversarial
vulnerability, we are the first to attempt at applying to explain Phenomenon 2.
We extend the existing NRF framework under PGD attack (Ilyas et al.] 2019) to more fine-grained NRF categorization by FGSM attack. We verify the existence NRF2 and show that its existence well justifies Phenomenon 2 (as well as Phenomenon 1).
Very recent works show that adding noise on the image input achieves SOTA performance for FGSM AT. However, their mechanism of such simple technique preventing CO remains not fully clear, for which our NRF2 perspective shed new light on its success.
2 PROBLEM OVERVIEW AND RELATED WORK
2.1 FGSM AT AND EXPERIMENTAL SETUPS
Let D denote data distribution with (r, y) pairs and f(:0) parameterized by A denote a deep model. For standard training, the model f(, 0) is trained on D by minimizing ((z.))>D(((((() 0), v)]. where I indicates a cross-entropy loss for typical multi-class classification task. Adversarial training
2

---

Under review as a conference paper at ICLR 2023
(AT) (Madry al. 2018) for obtaining robust model is formalized as min-mas optimization problem:
arg min E ((y,))DD
max max((f(x +000),+6;000)))) 6;0),
(I)
0
663
where S is a perturbation limitation (e with the lse constraint in this work). The outer minimization problem in AT is often the same standard training; however, AT has an unique inner maximization problem that seeks a perturbation inside the S for maximizing the optimization loss. PGD AT and FGSM AT are two typical adversarial training methods with PGD attack and FGSM attack solving the inner maximization problem, respectively.
Experimental setups. Unless specified, we follow the settings in GradAlign (Andriushchenko & Flammarion 2020) during training and evaluation. The experiments are conducted on CIFARTO with PreAct ResNet-18, 18, trained for 30 epochs with cyclic learning rates and half calf-precision training. We adopt SGD optimizer with weight decay 5 x 10 10-4, and the maximum learning rate is set to 0.2. foo attack with perturbation constraint e=8/255 is applied in both training and evaluation. Following (Wong et a 20020; Andriushchenko & Flammarion 2020), we calculate the Standard accuracy (Standard Acc) on clean samples, FGSM accuracy (FGSM Acc), and PGD accuracy (PGD Acc) under PGD-50-10 10 attack (performing PGD-50 attack with ten restarts and step size = e/4) for evaluation.
2.2 CATASTROPHIC OVERFITTING IN FGSM AT
What are the CO Phenomena? Notably, a model trained only on adversarial examples generated by FGSM attack in FGSM AT still has robustness against PGD attack. In practice, this robustness is only slightly lower than that of much more computationally expensive PGD AT. However, this robustness level can often not be maintained till the end of training as a classical PGD AT. Specifically, as the FGSM AT evolves, the model robustness against PGD attack first increases but then enters a phase where the robustness quickly drops and stays at zero. Following (Wong al.. 2020). this phase is termed catastrophic overfitting (CO). Another intriguing phenomenon related to CO is that for model at the phase of CO, it achieves higher FGSM Acc than Standard Ace (Kim al. 2020 Andriushchenko & Flammarion 2020). We term these two phenomena regarding CO as Phenomenon I and Phenomenon respectively, yyy as in Section [
How to explain the CO phenomena? With the finding that random initialization of perturbation helps alleviate CO (Wong et al., 2020). a tempting explanation suggests that the CO FGSM AT lies in the lack of perturbation diversity, which has been refuted by (Andriushchenko & Flammarion 2020). Instead, it attributes the reason for the PGD Ace drop to local non- non-linearity, which is quantified by the gradient alignment: cos(V, ( y; 0), vzl(x + 7), y; 0)). The local non-linearity (low gradient alignment) indicates a low linear approximation quality of FGSM perturbations to PGD perturbations. In other words, local non-linearity means that the inner maximization problem in Eq T cannot be solved accurately by FGSM. It is demonstrated in (Andriushchenko & Flammarion, 2020) that local linearity decreases significantly when CO happens in FGSM AT. Their perspective mainly dependent on the co-occurrence between non-linearity and the drop of PGD Acc. In other words, the non-line perspective exclusively focuses on explaining Phenomenon 1, for which this work provides an alternative NRF explanation (see Section 3). More importantly, our work fills the gap to explain Phenomenon 2 from NRF perspective (see Section
How to prevent CO? With the focus on Phenomenon 1, numerous works have attempted to prevent CO. Fast AT (Wong et al., (2200) the first show FGSM AT can achieve comparable robustness as PGD AT of "free" variants (Shatahi et al., 2019; (Zhang et al. 2019b). A follow-up work (An- driushchenko & Flammarion] (020) shows that CO still occurs in (Wong et al. 2020) when the step size increases and introduces regularization loss (GradAlign) for maximizing local linearity to avoid CO. Other successful attempts for avoiding CO include adaptive perturbation size (Kim et al. 2020). dynamic dropout scheduling (Vivek& & Babul 2020) and detection-based alternating strategy (Li et al. 2020). Intriguingly, very recent works (Zhang et al. 2022 de Jorge et al. 2022) have shown that adding noise on the image input is sufficient for preventing collapse and achieves SOTA performance. However, the reason for its success remains not fully clear, for which our NRF perspective with direction-based categorization provides explanation (see Section 5).
3

---

Under review as a conference paper at ICLR 2023
3 NON-ROBUST FEATURE PERSPECTIVE ON ADVERSARIAL TRAINING
Before investigating CO from the NRF perspective, we first revisit the definition and methodology of robust and non-robust features defined in (Iyyss al. 2019) (Fig. (Fig.Ta)). Considering the difference of attack strength between FGSM attack and PGD attack, we extend the non-robust features defined in Ilyas al. 2019) fine-grained categorization under FGSM attack (strength-based categorization
in Fig. b)) and discuss its relationship with CO phenomena.
3.1
BACKGROUND ON FEATURE USEFULNESS AND ROB USTUNSS
Here, we revisit the definitions and methodology of DNN features introduced in (Ilyas et al. 2019). According to (lyas et al. 2019), a feature is defined as function mapping from the input space X to real numbers, i.e X R. where IR can be the label space in classification task. Therefore, a DNN classifier can be perceived as function utilizing set of useful features for label prediction (yyas et al.] 2019). where useful features (llyas et al. 2019) are characterized by their positive correlation with true label, defined
p-useful features: A feature is p-useful (p 0) if it is correlated with the true label in expectation, shown as follows:
I (xxyi [x)] 2 p.
To understand adversarial vulnerability, Myyys et al. 2019) further proposes to dichotomize the above useful features into robust features (RFs) and non-robust features (NRFs), defined as follows:
Robust feature (RFs): usefuul feature f is robust if there exists a 7y > 0 for it to be -robussly useful under some specified set of valid perturbations A, shown follows:
E (xx))~p inf y f(x + 8)] 2 7. SEA(w)
Non-robust feature (NRFs): a useful feature f is non-robust if 7 0 does exist.
Adversarial vulnerability be attributed to the existence of NRFs (Myas et al.) 2019).
RF
NRF
DNRF
SNRF
As discussed in [Ilys et al., 2019). adversarial vulnerability is caused by the presence of NRFs which are useful and predictive. According to IIlas et al., 2019), "in the presence of an adver- sary, any useful but non non-robust features can be made anti-correlated with the true label, leading to adversarial vulnerability" (Ilyss et al. 2019). Therefore, adversarial training obtains robust model by discouraging from learning NRFs. In practice, finding worst-case perturbation under a certain budget for Eq. 33s s not feasible since it often an NP-hard problem (Katz al. 2017
+
No Attack
PGD Attack
FGSM Attack
(b)
Figure 1: Strength-based NRFs categorization. (a) Definitions RFs and NRFs in (Iyaas et. 2019), where the plus(+)mminus( -) sign indicate that the features are positive-correlate or anti-correlated with true label, respectively. (b) Definitions of DNRF. Considering the attack strength, DNRF are made anti-correlated by both FGSM and PGD attack, while SNRF is made anti-correlated by PGD attack but still pexitive-correlated with true labels under FGSM attack.
Weng et al. 2018), 8), and thus (Ilyas et al. 2019) uses multi-step PGD attack to approximate such a worst-case solution when investigating NRFs.
Fig. [
(a) summarizes the feature definition in indicates the useful features which has positive correlation with true labels, while the minus sign (-) indicates anti-correlated features under PGD attack.
(Ilya al., 2019). Specifically, the plus sign (+)
Verifying the existence of NRFs (Ilyas et al. 2019). The procedure verifying the existence of NRFs in (yyas et al. (2019) is summarized in Fig. aa by three steps. At Step 1, it trains model MM with standard training on the original training set (Xxrrain y), where Xtrain and indicate the training sample and its corresponding true label, respectively. At Step 2, it first randomly picks random label Irand for each training sample to ensure that the training set X Xtrain has features with positive correlation with the random label yrand- After that, perturbation ó is generated by PGD attack on Mi by making sample prediction f(x 8) close to Brand- This step aims generate perturbation g
4

---

Under review as a conference paper at ICLR 2023
(a/Veriffcation of (b) Verification of NRFs
(c) Verification of NRF2
DNRF
MMg by standard training
Mr by standard training
JM by FGSM AT
Step Train model Mr with original dalaset (X y)
Step Generate perturbation by attacking M4, that (x + 8) is close to/far from Yrant.
PGD attack close to yrand
FGSM attack close to Yrand
FGSM attack far from yranu
Step 3: Train model M6 with new datasel (Ntvatn + 6, yranu)), and evaluate X62 with original caraset (Xeest. y)
Experiment procedures for verifying the existence of non-robust features with three basic steps. (a) verifies the existence of NRF (mainly NRFI) in (Ilyas et al. 2019; (b) verifies the existence of DNRF; and (c) verifies the existence of NRF2.
Figure 2:
which includes NRTs related Brand At Step 3, model M2 is trained on the new dataset (X + 8, Drand) generated at Step 2. and then evaluated on the original test dataset with true labels (Xrest- y). According to [lyyss et al. 2019). the perturbation 5 is the only connection between X + 6 and yrand since there is no positive correlation between Xtrain and yrand- Therefore, if model MM achieves higher accuracy than random prediction (e.g 10% for CIFAR10) the original test dataset (Keests y) with true labels, the existence of NRFs in 5 is verified. We re-implement this experiment in (Ilyas al. 2019), and M2 achieves a accuracy of 16.16% (with five independent runs), as shown in Table which verifies the existence of NRFs as in (Ilyss et al.
2019
3.2 STRENGTH-BASED NRF CATEGORIZATION
It is widely known that PGD attack is stronger than FGSM attack, which is supported by the finding that FGSM Acc is higher than PGD Acc under the same lx perturbation budget (Madry et al. 2018). Thus, PGD Acc is often adopted AT is faster than PGD AT but at the cost of a mildly lower PGD Acc (than PGD AT) even when CO does not happen in FGSM AT. When CO occurs, the PGD Acc drops to a value close zero (Phenomenon 1). Since the difference between PGD AT and FGSM AT lies in the attack variant, GradAlign (Andri Flammarion, (020) explains their difference based on how well the adopted attack can solve the inner maximization problem. Specifically, FGSM AT yields lower robustness because FGSM attack cannot solve the problem as accurately as PGD attack because PGD attack is stronger than its FGSM counterpart. The following discussion provides alternative interpretation of the attack strength-basee explanation in (Andriushchenko & Flammarion 2020 from the NRF perspective.
common metric evaluate the model robustness. FGSM
Intuitive categorization. Considering the attack strength difference, the NRFs can be divided into two types, as shown in Figure [b). The first type of NRFs is named double non-robust feature (DNRF) since it can be made anti-correlated with the true labels by both FGSM and PGD attack. The existence of DNRF explains why FGSM AT yields more robust model than standard training against PGD attack during evaluation. By contrast, the other type of NRFs is called single non-robust feature (SNRF) since is made anti-correlated with true labels by PGD attack but is still positive-correlated with true labels under FGSM attack.
Experimental verification of DNRE This setup follows the procedure in (llyas et al. 2019) (Fig. aa) with a small modification. With its definition, DNRF has the property of being made anti-correlated with true labels under both PGD attack and FGSM attack. Therefore, the existence of DNRF ensures that the test acc will also be higher than random guess (10% for CIFAR10) if replace the PGD attack Step 2 with FGSM attack, as shown in Fig. (b... This is confirmed by an accuracy of 20.01 % on the original test set, see Table 1
Table I: Verifying the existence of DNRF. The experimental procedure follows Fig. aa) and (b), with PGD and FGSM attack at Step 2 respectively.
Features
Attack at Step 2
Test Acc of Mr
NRFs (Ilyss et al. 2019 DNRF
PGD (iig.
2
48. 18.1615.112
(Fig
ZD))
10.01+1.16
5

---

Under review as a conference paper at ICLR 2023
On SNRF and its relationship with CO phenomena. is challenging to directly verify the existence of SNRF. The phenomenon that 20.01% % (FGSM attack) is lower than 48.61% (PGD attack) in Table can be seen as indirect evidence for the existence of SNRFs which can be extracted by PGD attack but not FGSM attack. Even though direct empirical verification of SNRF is challenging, its theoretical existence is straightforward weaker the FGSM attack (compared with PGD attack), the more SNRF. FGSM AT cannot effectively discourage the model from learning SNRF PGD AT, and thus we can attribute the lower PGD Acc of FGSM AT than PGD AT to the existence of SNRF. However, SNRF under strength- baaet NRF categorization might (at most) partly explain CO Phenomenon but cannot justify CO Phenomenon 2. The reason is that the model might have a very low PGD Acc, but FGSM Acc cannot be higher than Standard Acc even in an extreme case when the NRFs become SNRF due to very weak FGSM attack. The following section introduces a new NRF categorization to better explain CO phenomena, especially Phenomenon 2.
long as FGSM attack is weaker than PGD) attack. Moreover, the
4 DIRECTION-BASED NRF CATEGORIZATION FOR UNDERSTANDING CO PHENOMENA
NRF1
NRF2
Direction-based NRF categorization. Similar to the above strength-based NRF categorization, the categoriza- tion here considers FGSM attack but differs by key assumption: whether the usefulness of certain NRFs is decreased or increased under FGSM attack. We call this NRF categorization as direction-based, which is defined follows:
PGD Attack
FGSM Attack
Figure 3: Change of usefulness under attack for NRFI and NRF2. The 1/4 indicate the increase/decrease of feature usefulness under attack, respectively. NRF1: NRFI is type of NRF whose usefulness is decreased after FGSM attack, thus can be exploited by FGSM attack to decrease the classification accuracy after FGSM attack.
NRF2: NRF2 is a type of NRF whose usefulness is increased after FGSM attack, thus can be exploited by FGSM attack to increase the classification accuracy after FGSM attack.
NRFI and NRF2 still follow the definition of NRF regarding PGD attack. In other words, the usefulness of both NRFI and NRF2 is decreased after PGD attack. The change of their usefulness after attacks is summarized in Fig. 3 where the increase and decrease of feature usefulness are denoted by the 1 and 4, respectively.
4.1 ON NRF2 EXISTENCE AND ITS EXPLANATION FOR PHENOMENON 2
When we discuss DNRF and SNRF in Section 3.2, by default, we assume that their usefulness is decreased after FGSM attack, and thus they can be seen NRFI. In other words, the existence of NRF1 is straightforward; however, it is unclear whether NRF2 actually exists.
Conjecture 1: We conjecture that there exists NRF2, and the FGSM attack in AT encourages the model to learn NRF2.
Differences between verifyyng NRFI and NRF2. The experimental procedure of verifying NRF2 is shown in Fig. c). The key reason why procedures in Fig. 2 can verify the existence of certain NRFs that the generated perturbation o is the only connection between Xtrain + 8 and Grand. and should include certain NRFs related to Yrand- In other words, the usefulness of certain NRFs should be increased after attack at Step of Fig. 2 For FGSM attack, f(x + 5) is optimized to be far from the true label by maximizing the loss 10 (x: + 5), y). and the usefulness of NRF1 and NRF2 are decreased and increased by definition, respectively(see Fig. Therefore, to increase the usefulness of NRFI Step 2, the optimization goal should be close to Frand, shown in Fig. 2].). By contrast, to verify the existence of NRF2, the optimization goal at Step should follow that FGSM attack, i. far from yrand, as shown in Fig. C), which increases the usefulness of NRF2.
Verification of Conjecture 1. As discussed above, verifying the existence of NRF2 requires an opposite optimization goal with that of NRFI at Step 2 (see Fig. (())). For the model M₁ at Step 1, we adopt FGSM AT with the results reported in Table When M, at Step is set to a CO model with FGSM AT, our model M2 evaluated on the original test achieves accuracy of around

---

Under review as a conference paper at ICLR 2023
17.52% + 1.59% (with five independent runs), which verifies the existence of NRF2 since it is higher than 10% (random prediction for CIFARIO), Given that the usefulness of NRF2 is increased after FGSM attack, the existence of NRF2 in model after CO justifies why FGSM Acc can be higher than Standard Acc (CO Phenomenon 2). Moreover, we experiment with setting the Mi at Step to one with standard training. the Test Acc of M2 is close random prediction, suggesting that it is the FGSM attack in AT that encourages the model to learn NRF2.
Does NRF2 exist in model before CO in FGSM AT? It is interesting to investigate whether NRF2 exists for the
Table 2: Verifying the existence of NRF2 shown in Figure [().. Ablation stud- ies of different M at Step I are also implemented.
FGSM AT model before CO. To this end, we set M1 at Step 1 of Fig. [[c) to a FGSM AT model saved before CO, which yields an M2 with an accuracy close to ran- dom prediction see FGSM AT(Before CO) in Table ( This indicates that NRF2 mainly exists in the model after CO in FGSM AT, which further confirms the relationship between CO and NRF2.
Model Mi
Test Acc of MM
Standard Training FGSM AT (After CO)
10.42:1:.774
17.52::1.59
FGSM (Before CO)
9,48:12.01
Can NRF2 be exploited by FGSM attack to decrease FGSM accuracy? (Kim et al. 2020) reports
that FGSM Acc is higher than Standard Ace when CO happens in FGSM AT. Here, we show that this is not always the case if we evaluate FGSM Acc of the CO model with different step sizes, as shown in Table 3 Note that the result with step size of zero indicates the Standard Acc. We find that FGSM Ace is higher (lower) than Standard Acc when the step size is relatively large (small). The results suggest that NRF2 can still be exploited by an FGSM attack with smaller step size to be anti-correlated with the true label. In other words, step size plays a non-trivial role when FGSM attack exploiting NRF2. The above results well explain why CO only occurs when the step size is set relatively large value (Wong al. 2020).
Table 3: Evaluate FGSM Acc of CO model under different step sizes.
step size (/255)
0
2
FGSM Acc (%)
85.77
37.27
41.57
66.78
86.93
95.16
96.94
96.63
94.72
A dynamic view on the CO from the NRF2 perspective. Prior works analyzing CO mainly focus on Phenomenon I about low PGD Ace, which seems to be a pseudo- static state since the PGD Acc stays at zero after CO. Here, we investigate CO model further by analyzing FGSM Acc at differ- ent epochs, as shown in Figure 4 Fig. 4 show that the FGSM Acc under large step sizes consistently gets higher with more training epochs, suggesting the model continues to rely more on NRF2. In other words, CO can be perceived as a dynamic state of learning NRF2, which does not stop after the drop of PGD Acc. This is reasonable because NRF2 can be very useful features under FGSM attack.
TOOR HOMM
Figure 4: FGSM Acc with different step sizes. (a) FGSM AT where CO occurs at epoch 24; (b) GradAlign model where CO does not occur.
4.2 CAN NRF2 AL.SO JUSTIFY PHENOMENON 1?
The above analysis verifies the existence of NRF2 in CO model, which well justifies the improved accuracy after FGSM attack (Phenomenon 2). Here, we discuss whether it can be used to justify Phenomenon 1. Regarding the relationship between NRF2 and PGD Acc, we formulate the following conjecture.
Conjecture 2: We conjecture that NRF2 can be cause of a significant PGD Acc drop.
Verification of Conjecture 2. To verify Conjecture 2, we finetune a robust model on a training dataset with and without such NRF2, respectively. and evaluate the PGD Acc on the original test set with true labels (Xtests y). To minimize the influence of other NRF types, we adopt a model pretrained by PGD AT, which mainly has RFs, for the finetuning experiment. Specifically, the generated new training dataset (X + 8, yrand) at Step 2 of Fig.2]o as the one with NRF2. For the counterpart dataset without NRF2, we remove the added perturbation 8, and thus (X, yrand) is
adopt
7

---

Under review as a conference paper at ICLR 2023
used for training. The basic loss is set to cross-entropy (CE) encourage learning the features, if any, in the generated dataset. However, the accuracy will quickly reduce to zero due to the random choice of Vrand- Thus, a KL loss, which encourages the output of the finetuned model to be close to that the pretrained mode, is added on top of the CE loss to encourage the model in the finetuning process maintain the original RFs. The total loss is shown follows:
LosS finetune CE(f(x + 6;0), Brand) + (L(f(z + 8; 0 (:retroin); f(x + 6)0)).
(4)
where [(z; Opretrain) and [xx; indicate the pretrained PGD model and finetuning model, respectively. A detailed setup for this experiment is reported in the Appendix. The results with X set to 5 are shown in Fig. [ We observe that the PGD Ace can be maintained around 25% after 30 epochs of finetuning for the dataset (X, Brand) which contains no features. By contrast, under the same setting, the PGD Acc quickly decreases to value close to zero for the generated dataset (X + 8, Brand) which contains NRF2. The contrasting results verify the claim in Conjecture 2.
0.5
Pinsture on (atraa
Fanslur V2
$
3.3
0.2
0.0
5
10
15
30
25
30
Eauch
Figure 5: Finetuning pre- trained PGD AT model.
Additional results with other values in Equation 4 are report in Fig. 6 As gets larger, the model finetuned on (X, Brand) maintains more RFs learned in pretrained weights 6 ppretrains leading to an increase in accuracy. However, the PGD Ace for the model finetuned on (X + yrand) (with NRF2) is zero for wide range of À values, which is much lower than the model finetuned on (X, Drand) (without NRF2). This further verifies the claim in Conjecture 2. Interestingly, the result in Fig. [6 can also be viewed another proof for Conjeture 1. The Standard Acc, evaluated on the original test set (Xrest, 3). is higher for the model finetuned on (X + 5, Brand) is higher than its counterpart on (X, yrand) for all in Fig. b.b.. This finding aligns with Conjecture that there exists a type of NRF, which can be encouraged under FGSM attack.
25
Finetune on (Xoo))))
20
Finetune on + 0.Vane)
Standard Acc
70
15
RGD
40
Finetune en (K.Yurel Finetune on (X Gyau)
20
2
7
0
8
Figure 6: Accuracy of the finetuned model with different lambda. x=0 indicates no KL term is introduced, and the model maintains more features of pretrained weights if is larger.
Discussion on the drop speed of PGD Ace from the NRF2 perspective. As demonstrated in Section 4.1 CO can be perceived as a dynamic process of learning NRF2. With this insight, learning occur suddenly, which is supported by the finding that FGSM Acc still
NRF2 in FGSM AT does increases even after CO happens. If this is the case, how can we justify the sudden drop of PGD Acc within one epoch? At first sight, it seems that NRF2 can only explain the PGD Acc drop but its drop speed. However, we argue that the sudden drop of PGD Acc is due to the worst-case property of PGD attack. Note that PGD attack seeks the most effective adversarial perturbation with multiple iterations to fool the model by exploiting the most vulnerable features in the model. In other words, the model already vulnerable to PGD attack even if only learns a small amount NRF2 (one epoch regarding CO, for instance). After the PGD Acc drops to zero, the model continues to learn more NRF2, leading to a higher FGSM Acc.
NRF2 HELPS EXPLAIN HOW SOTA METHODS PREVENT CO
recent work (Zhang et al. 2022) outperforms prior methods in FGSM AT by a large margin
without additional computation overhead. Specifically, it shows that adding noise to the input (instead of initializing the adversarial perturbation with noise as in (Wong et al.) (020 is critical for its success (Zhang al. 2022). A similar finding has also been reported in another recent work (de Jorge
8

---

Under review as a conference paper at ICLR 2023
et al. 2022). However, why such simple technique of adding noise on the images is so effective remains not fully clear. Here, we show that NRF2 sheds new light on their success.
Intuitivelyy the model tends to learn those features that are useful for prediction. Therefore, PGD AT mainly Iearns RFs because NRFs are not useful under PGD attack. With FGSM AT, the model is encouraged to learn NRF2 because FGSM attack increases its usefulness. Moreover, with our analysis in Section 4 CO can be seen as dynamic process of learning NRF2. Therefore, the key to preventing CO in FGSM AT lies in decreasing the NRF2 usefulness under FGSM attack. Regarding why adding noise to the image input prevents CO, we establish the following hypothesis.
Conjecture 3: We conjecture that adding noise the input decreases the usefulness of NRF2 under FGSM attack (indicated by FGSM Acc).
Verification of Conjecture 3. For facilitating the discussion, we divide all types of features into NRF2 and non-NRF2. A CO model has both NRF2 and non-NRF2, while a non-CO model mainly has NoR- We evaluate the performance on a model without or with random noise added to the input and calculate the noise-induced change of Standard Acc and FGSM Acc (Table J. Note that for FGSM Ace with noise, the noise is added to the imput before the FGSM attack following (Zhang ctt al. 2022). For the model before CO, the noise has almost the same influence on the change of Standard Acc and FGSM Acc, ie SSA of -0.70% (Standard Acc change) is close to VFGSM of 1.30%. We further conduct the same experiment on a CO model. Before adding noise, the FGSM Acc (94.67%) is higher than its standard Acc (85.77%) which can be attributed to NRF2 as in Conjecture 1, After adding noise, this trend is reversed 37% < 84.55%), suggesting Phenomenon 2 disappears in this setup. Moreover, FGSS (-37.30 .0%) is much more significant than SS (- 1.2%%). Such a significant drop of FGSM Acc (V rGSSM) on a CO model (with NRF2) suggests that the NRF2 usefulness under FGSM attack is significantly decreased. Fig. 7 visualizes VIGSM and v S of different noise sizes, which shows the same trend with Table 4 that V FrGSM of CO model is the most significant change among all settings. Therefore, Conjecture 3 is verified, which provides a new understanding on why input noise prevents CO.
Iinal mode bnfppess Tha
105/ mode burtee CD Prom
Hoal mode Tu
rede Cox Reas
Figure 7: Adding Uniform noise on input images during evaluation. Noise scale indicates the multiple of
Table 4: Adding noise on the input images of FGSM AT model before and after CO during evaluation. SA and V FGSM indicate the accuracy drop after adding Uniform noise, with noise scale 2x (2x8/255 2555 = 16/255).
Evaluation model
NRF2
non-NRF2
Stundard Acc with noise
FGSM with nosse
V PoSS VsA
original
ViA
original
FGSM model before CO FGSM model after CO
No
Yes
83
66.13 84,55
-0.70
41.95
40.65
-1.30
-0.60
85.77
-1.22
57.37
-3730
-36,08
More discussion on NRF2 explaining earlier attempts of mitigating CO. Even though we mainly apply our NRF2 to understanding the SOTA technique of input noise in recent works (Zhang et al.) 2022, de Jorge et al. (022). it also well justifies earlier successful attempts. For example, the success of random initialization in (Wong et al. 2020) is conceptually similar to adding the noise on the input but the noise magnitude is limited by the allowable perturbation size. (Kim et al.) (020) alleviates
CO by limiting the step size, which aligns well with our finding in Table 3 (L et al. 2020) avoids CO by switching to PGD AT after detecting the occurrence of CO, the success of which is expected since PGD attack can effectively discourage the model from learning NRF2.
CONCLUSION
The reason for CO in FGSM AT remains not fully clear despite various attempts to mitigate it. In contrast to prior works mainly studying PGD Acc drop to understand CO, our work focuses another intriguing phenomenon that FGSM Acc is higher than Standard Acc. We have found that there exists NRF2 whose usefulness is decreased under FGSM attack and CO can be seen as dynamic process of learning such a type of NRF. Our investigation has also provided a new understanding of successful attempts on how to mitigate CO in recent works.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey, IEEE Access, 2018.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. NeurlPS, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICMI, 2018.
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Smniić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, 2013.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected. In ACM Workshop on Artificial Intelligence and Security, 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter- attacks. In ICML,, 2020.
Pau de Jorge, Adel Bibi, Riccardo Volpi, Amartya Sanyal, Philip HS Torr, Grégory Rogez, and Puneet K Dokania. Make some noise: Reliable and efficient single-step adversarial training. arXiv preprint arKiv:2202.01 181, 2022.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International conference on computer aided verification, pp. 97-117. Springer, 2017.
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. arXiv preprint arXiv:0010001999 2020.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 2015.
Maosen Li, Cheng Deng, Tengjiao Li, Junchi Yan, Xinbo Gao, and Heng Huang. Towards transferable targeted attack. In CVPR, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Prectum Nakkiran. A discussion 'adversarial examples are not bugs, they are features': Adversarial examples are just bugs, too. Distill, 2019.
5.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NewrIPS, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 131iv: 2013.
10

---

Under review as a conference paper at ICLR 2023
Pedro Tabacof and Eduardo Valle. Exploring the space of adversarial images. In I/CNN, 2016.
Thomas Tanay and Lewis Griffin. A boundary tilting persepective the phenomenon of adversarial examples. arXiv preprint arXiv:1608.07690 2016.
BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling, In CVPR, 2020.
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning, pp. 5276-5285, PMLR, 2018.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. ICLR, 2020.
Chaoning Zhang. Francois Rameau, Seokju Lee, Junsik Kim, Philipp Benz, Dawit Mureja Argaw, Jean-Charles Bazin, and In So Kweon. Revisiting residual networks with nonlinear shortcuts, In BMVC, 2019a.
Chaoning Zhang, Philipp Benz, Tooba Imtiaz, and In-So Kweon. Understanding adversarial examples from the mutual influence of images and perturbations. In CVPR, 2020.
Chaoning Zhang. Philipp Benz, Dawit Mureja Argaw, Seokju Lee, Junsik Kim, Francois Rameau, Jean-Charles Bazin, and In So Kweon. Resnet or densenet? introducing dense shortcuts to resnet. In WACV, 2021.
Chaoning Zhang, Kang Zhang, Axi Niu, Chenshuang Zhang, Jiu Feng, Chang Yoo, and So Kweon. Noise augmentation is all you need for fgsm fast adversarial training: Catastrophic overfitting and robust overfitting require different augmentation. arXiv preprint arXiv:2202.05488, 2022.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In NeurIPS, 2019b.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019c.
A APPENDIX
Experimental setups for NRF categorization in Fig. 2 At Step 1, we follow the settings in (Andriushchenko & Flammarion 2020), and train Mi on CIFAR10 for epochs and cyclic learning rate with the maximum learning rate 0.3. Both attack radius sizes for training at Step I and perturbation generation at Step 2 are set as 8/255. Based on the new dataset (X + 6, yrand), M2 is trained for 30 epochs with constant learning rate 0.015.
Experimental setups for the finetuning experiment Section 4.2, The first two steps of the finetuning experiment follow the same settings of that in Fig. generating new dataset (.X + 8, Vrand). at Step 3, we first follow the settings of PGD AT in (Andrushchenkoo & Flammarion 2020) and train robust Mpgd- Based on the new dataset (X + ō, Drand), M2 trained by finetuning Mpgd for 30 epochs with a constant learning rate 0.005.
11


Reviews:
[{"review_id": "xbOdalb58pc", "reviewer": "Reviewer_LEgj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a more fine-grained (direction based) classification of non-robust feature (NRF) to explain two phenomena of CO: 1) the PGD accuracy drops to a value close to zero when CO happens; 2) FGSM accuracy is higher than standard accuracy for a CO model. The authors think that there is an NRF with increasing usefulness after FGSM attack. Based on this, an explanation is provided for a simple technique for CO mitigating.", "summary_of_the_review": "I appreciate the authors exploring CO from the perspective of NRF, but the main claim of the paper is questionable, so I am more inclined to reject.", "strength_and_weaknesses": "+Provides phenomenon 2 that has been overlooked before: FGSM accuracy is higher than standard accuracy for a CO model.\n\n+Extend the perspective of non-robust feature to FGSM AT to explain CO. \n\n-The core of this paper is the direction-based categorization of NRF, say, NRF1 and NRF2. But the verification of NRF2 is not convincing. Table 2 shows that NRF2 exists after CO, and not before CO, which is obviously contradictory. As far as I know, non-robust features are inherent properties of the data (derived from patterns in the data distribution [1]) that do not change with the training process. For example, the semantic features are RF and the background is NRF, and we cannot say that there is no NRF in the data after PGD AT training. Based on this point, the subsequent opinions and analysis seem to be self-justifying and not convincing.\n\n[1] Ilyas, Andrew, et al. \"Adversarial examples are not bugs, they are features.\" Advances in neural information processing systems 32 (2019)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JbA3uaRLYxQ", "reviewer": "Reviewer_x5Nw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the phenomenon of catastrophic overfitting (CO) in adversarially robust training with a single step perturbation method FGSM. It brings ones's attention to an intriguing property of CO, namely the accuracy on FGSM attacked samples is higher than that on clean samples. The authors try to explain this phenomenon from a robust/non-robust feature perspective. They conjecture the existence of a special type of non-robust feature that is particularly useful to the classification, and verify it based on controlled experiments. They also show that the non-robust feature perspective can potentially explain the recent practice on combating CO, namely adding random noise on the inputs.", "summary_of_the_review": "This paper made some interesting observations about catastrophic overfitting in adversariral training. But its contribution is hard to measure without comparative discussion with previous work studying this phenomenon. The clarity and quality also need to be improved as mentioned above.", "strength_and_weaknesses": "This paper is well-organized and presents the problem clearly. It focuses on an important problem in adversarially robust learning, and made interesting observations.\n\nMy biggest concern of this paper is that it might miss an important citation. [1] has discussed in detail about the phenomenon studied in this paper, namely accuracy against FGSM adversarial examples are higher than standard accuracy. In their paper,  the phenomenon is dubbed as the \"label leaking\" effect. They provide a very simple explanation to this phenomenon. One-step attack methods that use the true label perform a very predictable transformation that the model can learn to recognize. The generated adversarial examples thus may inadvertently leak information about the true label. I believe this explanation is not discussed anywhere in this paper. Furthermore, such a simple explanation and the observations made in this paper are not contradictory. In fact, the observations in this paper can further support the 'label leaking' explanation.\n\nNevertheless, some observations made in this paper are indeed novel and interesting. For example, in section 4.1, at \"verification of conjecture 1\", the authors verified the existence of a particular type of non-robust feature (NRF2) by showing the adversarial examples generated by a CO model can be learned by another model and achieve non-trivial standard accuracy. Compared to the original CO observation, this further shows that the information about the true label leaked in the adversarial example is in fact transferable, which is worth further investigation.\n\n[1] Adversarial Machine Learning At Scale. Kurakin et al.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5kWscYxwCIO", "reviewer": "Reviewer_xEW8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new explanation for catastrophic overfitting based on the existence of a special type of non-robust features which FGSM makes easier to learn. To demonstrate this, the authors reuse the experiments of (lyas et al. 2019) to validate their conjecture by showing that a training on a dataset containing only this type of non-robust features yields non-trivial accuracy. They later discuss how their new hypothesis can explain other observations regarding CO with special emphasis in explaining the success of injecting noise during fast-AT. Their working hypothesis, which they validate on some experiments, is that adding noise makes learning the non-robust features harder.", "summary_of_the_review": "Overall, I believe this is a borderline paper. The main hypothesis has merit and it is an interesting contribution to the community. However, this work only partially corroborates its truth based on weak evidence on a single dataset, and fails to explain prior observations. The text is  not easy to read, and some experiments have too many complicated design choices that do not manage to isolate the right effects. \n\nAt this stage, I am leaning towards rejection, but if the authors can convincingly show evidence that their findings can be replicated in other contexts and clarify my concerns of **weakness 5**, I might be willing to increase my score and lean to accept.", "strength_and_weaknesses": "# Strengths\n\n1. **Interesting idea**: The idea that CO could be caused by the existence of certain kinds of non-robust features in the data is very compelling and valuable.\n2. **Good empirical methodology** I really appreciate the presentation of the experiments in this work in which each experiment is motivated by a research question and a plausible hypothesis.\n3. **Interesting insights for fast AT methods** personally, I believe that the insights in Sec. 5 can be relevant to the community as they give more information as to why fast AT methods based on noise injection work.\n\n# Weaknesses\n\n1. **Only partial and weak evidence** One of the main weaknesses of this work is that the main experiment in Sec 3 only provides some weak signal in support of the main hypothesis. 17% accuracy is indeed slightly higher than trivial accuracy, but it is significantly lower than the original 48% used by Ilyas et al. to corroborate their conjecture. In this regard, ss the hypothesis presented in this work is rather complex, one would expect to have more convincing evidence that it is true in all settings. Something that would alleviate my concerns would be seeing a replication of this experiment on other datasets (e.g. SVHN, CIFAR100, ImageNet100…) and other models which showed that the weak numbers observed for CIFAR10 are at least consistent in other settings.\n2. **Poor clarity** In general, I have found the central sections of this work very hard to read. The constant use of acronyms, references to observations as non-descriptive Phenomenon 1 or 2, and in general, the convoluted writing to describe an already-complex hypothesis, make this paper very hard to read.\n3. **Unpolished experiments** I find some experimental setups in the text a bit strange and full of unjustified moving pieces that just add further complexity to test complex conjectures. For example, I am not sure I understand why the regularisation term of Eq. 4 is fully necessary, or why as robust model is needed to verify conjecture 2.\n4. **Some speculation** While I certainly believe this work provides partial evidence to connect CO with the features of the data, I believe it does not do the same to explain the sudden onset of CO in fast AT. The explanations of the authors in this regard are not substantiated by clear evidence and, thus, remain speculations in my mind.\n5. **No clear refutation of alternative hypothesis** Although in their introduction, the authors argue that the non-linear hypothesis of Andriushchenko & Flammarion cannot explain Phenomenon 2, there is no clear evidence in this work that explains why it is wrong. There could be other additional hypothesis on top of their non-linear conjecture that could explain that phenomenon, However, the NRF hypothesis of this work does fail to explain the key observations of Andriushchenko & Flammarion which is that after CO the models become highly non-linear. In the view of the authors, how does the NRF theory then explain this phenomenon, and why does GradAlign succeed in preventing CO?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "20t479lkr1D", "reviewer": "Reviewer_ykUA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on the finding of Non Robust Features (NRF) in the previous work (Ilyas et al., 2019), this paper tries to explain the phenomenon of  Catastrophic Overfitting (CO) in FGSM-based adversarial training. It categorizes NRFs into different types and tried to design experiments to show that the two commonly observed phenomena of CO, i.e., 1) the PGD Acc drops to a value close to zero when CO happens, and 2) FGSM Acc is higher than Standard Acc, is due to a specific type of NRFs.", "summary_of_the_review": "The paper studied an interesting problem. However, the proposed conjectures lacks theoretical and empirical justifications, making them unconvincing.\n\nFirst, the authors define DNRF as the features that are anti-correlated with the true labels by both FGSM and PGD attack. However, the authors did not explain why Figure2(b) can verify DNRF. It seems that the authors assume the features that are anti-correlated with the true labels by FGSM attack are the subset of the features that are anti-correlated with the true labels by PGD attack, but there’s no explanation or proof about this assumption. Note that “PGD attack is stronger than FGSM attack” does not explain the assumption. As far as I know, PGD is not the same as FGSM even with only one iteration. The relationship between DNRF and NRF2 is unclear to me. How do you make sure that NRF2 also exists in DNRF?\n\nThe existence of NRF2 also lacks justification. I don't understand why NRF2 can be exploited by Figure2(c) and have the following questions: \n1) Why the perturbation of FGSM attack far/close from y_rand can increase/decrease the usefulness of NRF?\n2) How to justify that the acc increment really comes from NRF2 instead of other features or side effects, such as the binary signing, in FGSM? \n3) Why there’s no counterpart NRFs in PGD, i.e., why there’s no NRF whose acc is decreased/increased after PGD attack thus can be exploited by PGD attack to decrease/increase the acc after PGD attack?\n\nThere seems to be a contradiction between the definition of NRF2 and the experiments results shown in Table3, where the former defines NRF2 are the NRFs that the usefulness (correlation with true label, following Eq 2) increases while the latter concludes that NRF2 can be anti-correlated with true label under smaller step size.\n\nThere’s no evidence of the existence of NRF2. There’s also no visualization nor ground-truth injection of NRF2. Merely showing gaps in Acc is not sufficient to explain that CO phenomena happen as the ways you thought. Even though we assume that DNRF and NRF2 exist, they do not explain why these features increase only FGSM Acc but not Standard Acc, given that the FGSM Acc is known to be higher than Standard Acc when CO happens.\n\nThe experiment of verification of Conjecture 2 is also confusing. Why not just evaluate the PGD Acc variation with (X_train, y) and (X + δ, y) against a PGD AT model? If NRF2 really hurts the performance, (X + δ, y) should give worse Acc than (X_train, y). Why is the fine-tuning necessary?\n\nLast but not the least, the authors tried to explain why adding noise prevents CO by “adding noise to the input decreases the usefulness of NRF2 under FGSM attack.” Again, there's no justification. I don't understand why adding noise to the input decreases the usefulness of NRF2 under FGSM attack. Your verification of Conjecture 3 is only a reproduction of (Zhanget al., 2022)'s experiments. And Figure 7 only shows that adding noise can prevent CO, which has nothing to do with NRF2.\n\n", "strength_and_weaknesses": "S1. The paper is well-written in general\nS2. The idea of categorizing NRFs to understand CO is new\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject