Catastrophic overfitting is a bug but it is caused by features
Authors: Guillermo Ortiz-Jimenez
Pau de Jorge
Amartya Sanyal
Adel Bibi
Puneet K. Dokania
Pascal Frossard
Grégory Rogez
Philip Torr
Venue: Submitted to ICLR 2023 (2023)
Keywords: Adversarial robustness
catastrophic overfitting
understanding deep learning
single-step adversarial training
FGSM
fast adversarial training

Abstract: Adversarial training (AT) is the de facto method to build robust neural networks, but it is computationally expensive. To overcome this, fast single-step attacks can be used, but doing so is prone to catastrophic overfitting (CO). This is when networks gain non-trivial robustness during the first stages of AT, but then reach a breaking point where they become vulnerable in just a few iterations. Although some works have succeeded at preventing CO, the different mechanisms that lead to this failure mode are still poorly understood. In this work, we study the onset of CO in single-step AT methods through controlled modifications of typical datasets of natural images. In particular, we show that CO can be induced when injecting the images with seemingly innocuous features that are very useful for non-robust classification but need to be combined with other features to obtain a robust classifier. This new perspective provides important insights into the mechanisms that lead to CO and improves our understanding of the general dynamics of adversarial training.

Under review as a conference paper at ICLR 2023
CATASTROPHIC OVERFITTING IS A BUG BUT IT IS CAUSED BY FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial training (AT) is the de facto method to build robust neural networks, but it is computationally expensive. To overcome this, fast single-step attacks can be used, but doing so is prone to catastrophic overfitting (CO). This is when networks gain non-trivial robustness during the first stages of AT, but then reach a breaking point where they become vulnerable in just few iterations. Although some works have succeeded at preventing CO, the different mechanisms that lead to this failure mode are still poorly understood. this work, we study the onset of CO in single- step AT methods through controlled modifications of typical datasets of natural images. In particular, we show that CO can be induced when injecting the images with seemingly innocuous features that are very useful for non-robust classification but need to be combined with other features to obtain a robust classifier. This new perspective provides important insights into the mechanisms that lead to CO and improves our understanding of the general dynamics of adversarial training.
INTRODUCTION
Deep neural networks are sensitive to imperceptible worst-case perturbations, also known as adversar- ial perturbations (Szegedy et al. 2014). As a consequence, training neural networks that are robust to such perturbations has been an active area of study in recent years (see Ortiz-Jiménee et al. (202T) for a review). In particular, a prominent line of research, referred to as adversarial training (AT), focuses on online data augmentation with adversarial samples during training. However, it well known that finding these adversarial samples for deep neural networks is an NP-hard problem (Weng) et al. 2018). In practice, this is usually overcome with various methods, referred to as adversurial attacks that find approximate solutions to this hard problem. The most popular attacks are based on projected gradient descent (PGD) (Madry et al. 2018) a computationally expensive algorithm that requires multiple steps of forward and backward passes through the neural network to approximate the solution. This hinders its use in many large-scale scale applications motivating the use of alternative
efficient single-step attacks (Goodfellow al. 2015 Shafahi et al. 2019 Wong al. 2020
The use of the computationally efficient single-step attacks within AT, however, comes with con- cerns regarding its stability. While training, although there is an initial increase in robustness, the networks often reach breaking point beyond which they lose all gained robustness in just few iter- ations (Wong et al., 2020). This phenomenon is known catastrophic overfitting (CO) (Wong et al.) 2020 (Andrushchenko & Flammarion 2020). Nevertheless, given the clear computational advantage of using single-step attacks during AT, significant body of work has been dedicated to finding ways to circumvent CO via regularization and data augmentation (Andriushchenko & Flammarion 2020
Vivek & Babu 2020 Kim et al. 2021 Park & Lee| 2022 Golgooni et al. 2021 de Jorge et al. 2022).
Despite the recent methodological advances in this front, the root cause of CO remains poorly understood. Due to the inherent complexity of this problem, we argue that identifying the causal mechanisms behind CO cannot be done through observations alone and requires active interven- tions Iyas et al. (2019). That is, we need to be able to synthetically induce CO in settings where would not naturally lly happen otherwise.
In this work, we identify one such type of intervention that allows to perform abundant experiments to explain multiple aspects of CO. Specifically, the main contributions of our work are: (i) We show that CO can be induced by injecting features that, despite being strongly discriminative (i.e. useful for standard classification), are not sufficient for robust classification (see Fig. [ (ii) Through

---

Under review as a conference paper at ICLR 2023
Data intervention
Clean accuracy
Robust accuracy
100%
100%
+
903
90%
6056
v(v)
85%
Injectod features
20%
80%
0%
V
0
10
15
20
25
10
20
25
R0/255)
(3(2555)
8/255
Depiction of our modified dataset that injects simple, discriminative features. Right: Clean and robust performance after FGSM-AT on injected datasets Dzz, We vary the strength of the synthetic features 3 (B = O corresponds to the original CIFAR-10) and the robustness budget e (train and test). We observe that for e E {4/255, 6/255} our intervention can induce CO when the synthetic features have strength 3 slightly larger than while training on the original data does not suffer CO. Results are averaged over 3 seeds and shaded areas report minimum and maximum values.
Figure 1:
Left:
extensive empirical analysis, discover that CO is connected the preference of the network to learn different features a dataset. (iii) Building upon these insights, we describe and analyse a causal chain of events that can lead to CO. The main message of our paper is:
Catastrophic overfitting is learning shortcut used by the network to avoid learning complex robust features while achieving high accuracy using easy non-robust ones.
Our findings improve our understanding of CO by focusing on how data influences AT. Moreover, they also provide insights the dynamics of AT, in which the interaction between robust and non-robust features plays key role.
we give overview of the related work on CO. Section 3 presents our main observation: CO can be induced by manipulating the data distribution. In Section 4 we perform an in-depth analysis of this phenomenon identify the causes of CO. Finally, in Section we use our new perspective to provide new insights the different ways can prevent CO.
Outline In Section 2
2 PRELIMINARIES AND RELATED WORK
Let fo Rd - y denote a neural network architecture parameterized by set of weights 0 E Ik" which maps input samples x Fd y E y cl....,ch. The objective of adversarial training (AT) is to find the network parameters E R" that optimize the following min-max problem:
min E (DWN max |101,
+
0
where D is some data distribution, 8 E Rdd represents an adversarial perturbation, and p, characterize the adversary. This is typically solved by alternately minimizing the outer objective and maximizing the inner one via first-order optimization procedures. The outer minimization is tackled via some standard optimizer, e.g., SGD, while the inner maximization problem is approximated with adversarial attacks like Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2015) and Projected Gradient Descent (PGD) (Madry et al.) 2018). Single-step AT methods are built top of FGSM. In particular, FGSM solves the linearised version of the inner maximization objective. When = oc, this leads to:
8PGSM argmax C(fo(z) 5 ( = esign (V (V=((fo(x).))
Sr
Note that FGSM is very computationally efficient as only requires single forward forward-backwarr step. Unfortunately. FGSM-AT generally yields networks that are vulnerable to multi-step attacks such as PGD. In particular, Wong et al. (2020) observed that FGSM-AT presents a characteristic failure mode where the robustness of the model increases during the initial training epochs, but, at a certain point in training, the model loses all its robustness within the span of a few iterations. This is known catastrophic overfitting (CO). They further observed that augmenting the FGSM attack with random noise seemed to mitigate CO. However, Andriishchennoo & Flammarion (2020) showed that this method still leads to CO larger e. Therefore, they proposed combining FGSM-AT with
2

---

Under review as a conference paper at ICLR 2023
smoothness regularizer (GradAlign) that encourages the cross-entropy loss to be locally linear. Although GradAlign succeeds in avoiding CO in all tested scenarios, optimizing it requires the computation of second-order derivative, which adds a significant computational overhead.
Several methods have been proposed that attempt to avoid CO while reducing the cost of AT. However these methods either only move CO larger radii (Golgooni et al. (02T), are more computationally expensive (Shafahi et al. [019 Li et al. 2020), or achieve sub-optimal robustness (Kang & Moosavi
Dezfooli 2021 Kim et al. [202]] p. Recently, de Jorge al. (2022) proposed N-FGSM that successfully
avoids CO for large radii while incurring only a fraction of the computational cost of Grad
On the more expensive side, multi-step attacks approximate the inner maximization in Eq. ① with several gradient ascent steps (Kurakin et al., [017 Madry et al.) 20188] Zhang et al. 2019). Provided they use sufficient number of steps, these methods do not suffer from CO and achieve better robustness. Nevertheless, using multi-step attacks in AT linearly increases the cost of training with the number of steps. Due to their superior performance and extensively validated robustness in the literature (Madry et al. 20188 Tramer et al.) 2018 Zhang et al. 2019 Rice et al. 2020) multi-step methods, such as PGD, are considered the reference AT.
Aside from proposing methods to avoid CO, some works have also studied different aspects of the training dynamics when CO occurs. [Wong et al. (2020) initially suggested that CO was a result of the networks overfitting to attacks limited to the corners of the fx ball. This conjecture was later dismissed by Andriushchenko & Flammarion (2020) who showed that AT with PGD attacks projected to the corners of the for ball does not suffer from CO. Similarly, while AAdriushchennoo & Flammarion (2020) suggested that the reason that Wong et al. (2020) avoids CO is the reduced step size of the attack, de Jorge et al. (2022) showed they could prevent CO with noise augmentations while using a larger step-size. On the other hand, it has been consistently reported (Andriushchenko & Flammarion 2020) Kim et al. 202T) that networks suffering from CO exhibit highly non- lon-linear loss landscape with respect to the input compared to their CO-free counterparts. As FGSM relies on the local linearity of the loss landscape. this sudden increase in non- non-liicarrtty of the loss renders FGSM practically ineffective (Kim et al., 2021 Kang & Moosavi-Dezfoolij 202Tp. This provides plausible explanation for why models are not fooled by FGSM after CO. However, none of these works have managed to identify what pushes the network to become strongly non-linear. In this work, we address this knowledge gap, and explore a plausible mechanism that can cause single-step AT to develop CO.
3 INDUCING CATASTROPHIS OVERFITTING
Our starting point is well known observation: while robust solutions can be attained with non-trivial training procedures, e.g., using AT, they are not the default consequence of standard training. That is, robust solutions are harder to learn and are avoided unless explicitly enforced (e.g. via adversarial training). On the other hand, we know robust classification requires leveraging alternative robust features that are not learned in the context of standard training (Iyaa et al. 2019 Sanyal et al. [022] and, when CO happens, the robust accuracy plummets but the clean and FGSM accuracies do not drop; on the contrary, they keep increasing (Wong et al. 2020; Andriushchenko & Flammarion, 2020).
Bearing this in mind, we pose the following question: Can CO be a mechanism to avoid learning the complex robust features? ? If this true, then the network could be using CO as a way to favour the learning of some very easy and non-robust features while ignoring the complex robust ones.
Directly testing this hypothesis, however, requires identifying and characterizing these two sets of features (robust vs rob-sobust) in real dataset. This is a challenging task (as it is basically equivalent to solving the problem of adversarial robustness) that is beyond our current capabilities. For this reason, as is standard practice in the field (Arpit et al. 2017] [lyas et al., 2019; Shah et al. 2020 Ortiz-limenez et al. 2020a), we take an alternative approach that relies on controlled modifications of the data. Conducting experiments the manipulated data, we are able to make claims about CO and the structure of the data. In particular, we discover that if we inject a very simple discriminative feature on standard vision datasets then, under some conditions, we can induce CO at much lower values of e than those for which it naturally happens without our intervention. This is a clear sign that the structure of the data plays big role in the onset of CO.
Our injected dataset Let (x. 3) ~ D be an image-label pair sampled from a distribution D. Our intervention modifies the original data xr by adding an injected label-dependent feature vly). We
3

---

Under review as a conference paper at ICLR 2023
construct family of injected datasets Dg where the label-dependent feature is scaled by 8:
(ã. 3) D3= == == 3 (yy) with (x, ~ D,
(3)
Moreover, we design (yy such that | = for all y E y and are linearly separable with respect to y. Since CO has primarily been observed for &g perturbations, we mainly use oo but also present some results with p = 2 in Appendix [ We denote the set of all injected features as y = (n(y) y E 3/}. The scale parameter / > 0 is fixed for all classes and controls the relative strength of the original and injected features, i.e,, a and mvy) respectively (see Fig. (left)).
This construction has some interesting properties. Since the injected features are linearly separable and perfectly correlated with the labels, a linear classifier relying only on V can separate D3 for large enough B. Moreover, as 33 also controls the classification margin, if (3 > this classifier is also robust. However, if x: has some components in sVan the interaction between and v(y) may decrease the robustness (or even clean accuracy) of the classifier for small B.. We rigorously illustrate such a behaviour for linear classifiers in Appendix A In short, although uvv) is casy-to-learn in general (we will empirically see that in Section 4.1. the discriminative power (and robustness) of a classifier that solely relies on the injected features V will depend on 3.
With the aim to control the interactions between x and v(y), we design V by selecting vectors from the wwwffreuency components of the 2D Discrete Cosine Transform (DCT) (Ahmed et al., 1974) as these have large alignment with the space of natural images that we use for our experiments (e.g.. CIF)R-10). To ensure the norm constraint, we binarize these vectors so that they only take values in =1, ensuring a maximal per-pixel perturbation that satisfies [b)()) 1. These two design constraints also help to visually identify the alignment of adversarial perturbations with vyy) as these patterns are visually distinctive (see Fig.
Injection strength (3) drives CO We train PreActResNet188 (He et al.) (016) different intervened versions of CIFAR-IO (Krizhevsky & Hinton 2009) using FGSM-AT for different robustness budgets c and different scales 3. Fig. D (right) shows a summary of these experiments both in terms of clean accuracy For the clean accuracy, Fig. (right) shows two distinct regimes. First, when B < e, the network achieves roughly the same accuracy by training and testing on D3 as by training and testing on D (corresponding to =). This is expected as FGSM does not suffer from CO in this setting (see Fig- (right)) and effectively ignores the added feature v(y). Meanwhile, when 3 >> e, the clean test accuracy is almost 100% (which is larger than of accuracy on CIFAR-10) indicating that the network heavily relies on the injected features. We provide further evidence for this in Appendix E
Intevvoe suncls (r
fyoEs berveen CT)
and robustness
after CO)
Figure 2: Different samples of the injected dataset Da, and FGSM perturbations before and after CO. While prior to CO perturbations focus on the syn- thetic features, after CO they become noisy.
The behaviour with respect the robust accuracy is more interesting. For small c = 3/255) the robust accuracy shows the same trend as the clean accuracy, albeit with lower values. For large
e (e = 8/255), the model incurs CO for most values of 3. This not surprising as CO has already been reported for this value of & on the original CIFAR-10 10 dataset (Wong et al. 2020). However, the interesting setting is for intermediate values of E (e E {4/265, 6/255)) }). For these settings, Fig. I (right) distinguishes between three distinct regimes. The first two regimes are the same as for
2/255:
(i) when the strength of the injected features is weak (3 < c), the robust accuracy is similar to that trained on the original data (3- = 0) and (ii) when it is strong (3 > c), the robust accuracy is high as the network can use only v(y) to classify T robustly. Nevertheless, there is a third regime where the injected features are mildly robust, i.e.
e. Strikingly, in this regime, the training suffers from CO and the robust accuracy drops to zero. This is significant, since training on the original dataset DD (8 0) does not suffer from CO for this value of e; but it does so when 3
Thrrughout the paper, robustness is measured against strong PGD attacks with 50 iterations and 10 restarts.
4

---

Under review as a conference paper at ICLR 2023
We replicate these results for different V's and for l2 perturbations with similar findings in Appendix D Results for other datasets and networks, as well as further details of the training protocol are given in Appendices C and respectively From these observations, conclude that there is indeed a link between the structure of the data and CO. In the following section, we delve deeper into these results to better understand the cause of CO.
ANALYSIS OF INDUCED CATASTROPHIE OVERFITTING
Since now have a method intervene in the data using Eq. 3 and induce CO, we can use it to better characterize the mechanisms that lead to CO. In particular, we explore how different features in the dataset influences the likelihood of observing CO in FGSM-AT.
4.1 ROBUST SOLUTIONS COMBINE EASY- AND HARD-TO--EARR FEATURES
The previous section showed that when 3 < or 3 > c, our data intervention does not induce CO. However, for = c FGSM-AT consistently experiences CO. This begs the question: what makes special? We show that for original dataset 'D and the injected features in V to achieve a high robust accuracy on the injected dataset Dz. However, when trained without any adversarial constraints i.e., for standard training, the network only uses the features in and achieves close to perfect clean accuracy.
network trained using AT uses information from both the
In order to demonstrate this empirically, we perform standard, FGSM-AT, and PGD-AT training of a PreActResNet188
the injected dataset D (as described in Section 3) with 8 $/255 and c= 6/255. First, note that Fig. (right) shows that an FGSM-AT model suffers from CO when trained on this injected dataset. Next, to identify the different features learned by these three models, construct three different test sets and evaluate the clean and robust accuracy of the networks on them in Fig. 3 The three different test sets are: (i) CIFAR-10 test set with injected features (Dp), (ii) original CIFAR-10 test set (D), and (iii) CIFAR-IO
with shuffled injected features (D+(2)) where the additive signals are correlated with a permuted set of labels, i.e.
(-((), .y) ~ P=(B): (-))_
(v)) with (æ, v) D and E V.
Here, V + D is fixed permutation operator that shuffles the labels. Note that evaluating these networks (trained Dz) on data from P=(3) exposes them contradictory information, since x and (x)) are correlated with different labels in DB. Thus, if the classifier only relies on V the performance should be high D, and low on D and (x())) while if it only relies DD the performance should remain constant for all injected datasets.
PGD training We can conclude from Fig. lleft) that the PGD-trained network achieves robust solution using both D and V. We know it uses D as it achieves better than trivial accuracy on D (containing no information from V) as well as on Da(,8) (where features from V are uncorrelated with the correct label; see Eq. ( ). On the other hand, we know the PGD-trained network uses V as the network achieves higher accuracy on D3 (containing information from both D and V) than D (only contains information from D). Further, it suffers drop in performance on Dx(9) (where information from D and V are anti-correlated). This implics that the robust PGD solution effectively combines information from both the original and injected features for classification.
Standard training Standard training shows a completely different behaviour than PGD (see Fig. (center)). In this case, even though the network achieves excellent clean accuracy on D3. its accuracy on D is nearly trivial. This indicates that with standard training the model ignores the information present in 2 and only uses the non-robust features from V for classification. This is further supported by the observation that on DR(3), where the labels of the injected features are randomized, its accuracy is almost zero. From these observations we conclude that the injected features easy to learn i.c. is preferred by our optimiser and neural networks.
FGSM training The behaviour of the FGSM-AT in Fig. riihht highlights the preference for the injected features even further. In the studied setting, FGSM-AT undergoes CO around epoch 8 when the robust accuracy on D. suddenly drops zero despite high clean accuracy on Dj.. Next, seen
5

---

Under review as a conference paper at ICLR 2023
PGD-10 training
Standard training
FGSM training
100%
100%
100%
CO
Cleta uccurccy
300
5076
10%
098
095
061
10
20
30
10
20
60%
6005
6006
CO
Robart accuracy
40%
4076
40%
200
9055
0%
0956
019
30
100
Epoch
Eporls
Epoch
Injerted (Dx)
CIFAR1O (D)
Shuffedd (D.st)
Figure 3: Clean (top) and robust (bottom) accuracy on 3 different test sets: (i) the original CIFAR-10
(D), (ii) the dataset with injected features D3 and (ii) the dataset with shuffled injected features D Da( All training runs use 3 = $/255 and e = 6/255 (where FGSM-AT suffers CO). The blue shading denotes when the network exploits both D and V and the yellow shading when it learns only V.
in Fig. 3 (top right), FGSM-AT presents two distinct phases during training: (i) Prior to CO, when the robust accuracy on D3 is non-zero, the network leverages features from both D and V, similar to PGD. (ii) However, with the onset of CO, both the clean and robust accuracy on D and Dx(3) drops, exhibiting behavior similar to standard training. This indicates that, post-CO, akin to standard training, the network forgets the information from D and solely relies features in V.
Why does FGSM change the learned features after CO? From the behaviour of standard training concluded that the network has preference for the injected features V, i.e., they easy to learn. The behaviour of PGD training suggests that when the easy features are not sufficient to classify robustly, the model combines them with other (harder-to-learn) features e.g., in DD become robust. FGSM initially learns a robust solution leveraging both and V similar to PGD. However, the FGSM attacks are rendered ineffective, the robustness constraints are essentially removed. This allows the network to revert back to the simple features and the performance on the original dataset D drops. This is. exactly, what occurs with the onset of CO around epoch 10 (further discussed in Section 4.2) Yet, why does CO happen in the first place? In the following, we will see that the key to answer this question lies in the way learning each type feature influences the local geometry of the classifier.
4.2 CURVATURE EXPLOSION DRIVES CATASTROPHIC OVERHTTING
Recent works have shown that after the onset of CO the local geometry around the input becomes non-linear (Andriushchenko & Flammarion, 2020 Kim et al. [021 de Jorge et al. 2022). Motivated by these findings, and with the aim identify how data intervention causes CO, now investigate how the curvature of the loss landscape evolves with different types of training. We the average maximum eigenvalue of the Hessian N 100 fixed training points Xmua E
m=1 Amax C(fo ((,) ) to estimate the curvature, as suggested in Moosavi- -ezzavil et al. (2019). and record it throughout training. Fig. 1lett) shows the result of this experiment for FGSM- AT (orange line) and PGD-AT (green line) training on D) with 3 8/255 and 6/255. Recall that this training regime exhibits CO with FGSM-AT around epoch 8 (see Fig. 1efo))
Two-phase curvature increase Interestingly, we observe that even before the onset of CO, both FGSM-AT and PGD-AT show nearly similar steep increase in curvature (the y-axis is in logarithmic scale). While right before the 8th epoch, there a large increase in the curvature for PGD-AT, stabilizes soon after and remains controlled for the rest of training. Prior work has observed that PGD-AT acts as a regularizer on the curvature (Moosavi-Dezfood) et al.) [019 Qin et al., 2019) which explains how PGD-AT controls the curvature explosion. However, curvature second-order
Results for other parameters and for the original D are provided in Appendix

---

Under review as a conference paper at ICLR 2023
Curvataze (Amor)
CIFAR-10 feature aceuacc
C0
702
10P
607
10
Plane
30%
10"9
0
10
15
25
30
15
20
225
30
Epoch
Epoch
PGSM-AT (Dg)
PGD-AT (Ds)
POSM-AT (P))
PGELAT (0))
Figure 4: Evolution of different metrics for FGSM-AT and PGD-AT 2 datasets: (i) with injected fea- tures (Dz) and (ii) with orthogonally projected features, i.e. with interaction between the original and injected features (D)). AT is performed for B $/255 and e 6/255 (where FGSM suffers CO).
property of the loss surface and unlike PGD-AT, FGSM-AT is based on coarse linear (first order) approximation of the loss. Therefore, FGSM-AT is not as effective at regularising the curvature. Indeed, we see that FGSM-AT cannot contain the curvature increase, which eventually explodes around the 8th epoch and saturates at a very large value. Quite remarkably, the final curvature of the FGSM-AT model is 100 times that of the PGD-AT model.
High curvature leads to meaningless perturbations The fact that the curvature increases rapidly during CO, when the attack accuracy also increases, agrees with the findings of (Anrriushhhenko & Flammarion (2020), that the loss becomes highly non-linear and thus reduces the success rate of FGSM. To show that CO indeed occurs due to the increased curvature breaking FGSM, we visualise the adversarial perturbations before and after CO. As observed in Fig. 2 before CO, the adversarial perturbations point in the direction V. albeit with some corruptions originating from æ. Nonethe- less, after CO, the new adversarial perturbations point towards meaningless directions; they do not align with even though the network heavily reliant on this information for classifying the data (cf. Section 4.1 This reinforces the idea that the increase in curvature indeed causes breaking point after which FGSM is no longer an effective adversarial attack. We would like to highlight that this be- haviour of the adversarial perturbations after CO is radically different from the behaviour standard and robust networks (in the absence of CO) where adversarial perturbations and curvature are strongly aligned with discriminative directions (Fawzi et al. 2018 Jetley et al.) 22018 llyas et al. 2019).
4.3 CURVATURE INCREASE IS A RESULT OF INTERACTION BETWEEN FEATURES
But why does the network increase the curvature in the first place? In Section we observed that this is a shared behaviour of PGD-AT and FGSM-AT, at least during the initial stage before CO. Therefore, it should not be a mere "bug". : We conjecture that the curvature increases is result of the interaction between features of the dataset which forces the network to increase its non-linearity in order to combine them effectively to obtain robust model.
Curvature does not increase without interaction To demonstrate this, we perform a new experiment in which we modify 'D again (as in Section However, this time, we ensure that there is no interac- tion between the synthetic features vyy) and the features from D. We do so by creating D such that:
a Pya (x) + Byy with (x, D and vyy) EV
(5)
where PyL denotes the projection operator onto the orthogonal complement of V. Since the synthetic features vyy) are orthogonal to D, a simple linear classifier relying only on v(y) can robustly separate the data up to radius that depends solely on 3 (see the theoretical construction in Appendix A).
Interestingly, we find that, in this dataset, none of the (3, configurations used in Fig. 5 induce CO. Here, we observe only two regimes: one that ignores V (when 3 < c) and one that ignores D (when e). This supports our conjecture that the interaction between the features of æ and (y) is the true cause of CO in Dz. Moreover, Fig. 4 (left) shows that, when performing either FGSM-AT (light blue) or PGD-AT (dark blue) on D 33. the curvature is consistently low. This agrees with the fact that in this case there is no need for the network to combine the injected and the original features to achieve robustness and hence the network does not need increase its non-linearity separate the data.
7

---

Under review as a conference paper at ICLR 2023
Clean accuracy
Robast accuracy
10005
100%
9062
00%
20%
1005
09
(1 (1235))
/8 (1956)
2/255
4/255
Figure 5: Clean (left) and robust (right) accuracy after FGSM-AT on dataset with orthogonally injected features D g i.c interaction between original and injected features. We train while varying the strength of the injected features and the robustness budget 6.
Non-linear feature extraction Finally, we study the relationship between the information learned in the features and the network's curvature. Using similar methodology as Shi et (2022), train multiple logistic classifiers on the the feature representations of 'D (output of the penultimate layer) of networks trained on Dz Note that the accuracy of these classifiers strictly depends on how well the network (trained on D&)) has learned both and We will call this metric fearuue accuracy. Figure Iigght) shows the evolution of the feature accuracy of the networks during training. Observe that, for PGD-AT (green), the feature accuracy on DD progressively grows during training. The high values of feature accuracy indicate that this network has learned to meaningfully extract information from D, even if it trained Dg. Moreover, we note that the feature accuracy closely matches (left). Meanwhile, for FGSM-AT the feature accuracy has two phases: first, it grows the same rate for the PGD-AT network, but when CO happens, it starts to decrease. Note, however, that the curvature does not decrease. We argue this is because the network using a shortcut ignore D. Specifically, if the curvature very high, FGSM rendered ineffective and allows the network to focus only on the easy non-robust features. On the other hand, if we the features from networks trained on D low. This reinforces the view that the network is increasing feature representation. In D the network does not need to combine information from both and to become robust, and hence it does not learn to extract information from D.
the curvature trajectory in Fig. 4
observe that the accuracy on D is always curvature in order to improve its
4.4 A MECHANISTIC EXPLANATION OF CO
To summarize, we finally describe the chain of events that leads to CO in our injected datasets:
ii To learn robust solution, the network attempts to combine easy, non-robust features with more complex robust features. However, without robustness constraints, the network strongly favors learning only the non-robust features (see Section 4.1).
(ii When learning both kinds of features simultaneously, the network increases its non-linearity to improve its feature extraction ability (see Section 4.3).
(iii) This increase con-lincarity provides a shortcut to break FGSM which triggers CO. This allows the network avoid learning the complex robust features while still achieving a high accuracy using only the easy non-robust ones (see Section 4.2h.
Aside from empirical study, the intuition that classifier needs combine features to become robust can be formalized in certain settings. For example, in Appendix we mathematically prove that there exist many learning problems in which learning robust classifier requires leveraging additional non-linear features on top of the simple ones used for the clean solution. Moreover, in Section 5 we leverage these intuitions explore how data interventions on real datasets can prevent CO.
FURTHER INSIGHTS AND DISCUSSION
Our proposed dataset intervention, defined in Section 3 allowed us gain better understanding of the chain of events that lead to CO. In this section, we focus our attention on methods that can prevent CO and analyze them in the context of our framework provide further insights.
8

---

Under review as a conference paper at ICLR 2023
Robust accuracy
Curvature (Aroae)
10.
50%
00%
107
40%
13"
0%
10
12
14
10
20
25
30
(/206)
Epoch
PGD-10 10
GoudAlign 2 0.112)
GradAlign
N. PGPG (# 10c)
N.GSM 2c]
Figure 6: Left: Clean and robust performance after AT with GradAlign, N-FGSM and PGD-10 on DA at é 6/255. Results averaged over three random seeds and shaded areas report minimum and maximum values. Right: Curvature evolution when training on D3 at e 6/255 and B 8/235.
GradAlign and N-FGSM prevent CO on D3 In Fig, 6, we show the robust accuracy (left) and curvature (right) of models trained with GradAlign (Andriushchenko & Flammarion 2020) and N-FGSM (de Jorge al. 2022) on Di for varying 3. Figure (left) shows that both methods able to prevent CO on our injected dataset for suitable choices of the regularisation parameter i.e., À for GradAlign and & for N-FGSM. This suggests that the mechanism by which our intervention induces CO is similar to how it occurs in real datasets. However, for certain values of B, both GradAlign and N-FGSM require stronger regularization. Thus, the regularization strength not only function of c, as discussed in their respective manuscripts, but also of the signal strength B. As / increases, vvv) becomes more discriminative creating a stronger incentive for the network to use it. We argue that this increases the chances for CO as the network (based on our observations in Section 4 will likely increase the curvature to combine the discriminative injected features with others in order to become robust. Moreover, Fig. shows that the curvature of N-FGSM and GradAlign AT stabilizes with trend similar to PGD-AT and stronger regularizations dampens the increase of the curvature even further. This further shows that preventing the curvature from exploding can indeed prevent CO.
Can data modifications avoid CO? Section shows that it is possible induce CO through data manip- ulations. But is the opposite also true that CO can be avoided using data manipulations? We find that this is indeed possible on CIFAR-10. 10. Table I shows that removing the high frequency components of D con- sistently prevents CO at 8/255 (where FGSM-AT fails). Interestingly,GGrabnn Grabinski et al. (0022) in concur- rent work, applied this idea to the pooling layers and showed they could prevent CO. Surprisingly, though, we have found that applying the same low-pass tech- nique at e = 16/255 does not work. We conjecture this is because the features which are robust at 8/255 in the low pass version of CIFAR- 10 might not be robust at = 16/255, therefore forcing the network to combine more complex features. Although this method does not work in all settings, it is an interesting proof of concept that shows that removing some features from the dataset can indeed prevent CO. Gen- eralizing this idea other kinds of features and datasets can be promising avenue for future work.
Table 1: Clean and robust accuracies of FGSM-AT and PGD-AT trained networks on CIFAR-10 and the low pass version described
in Ortiz-Jimenez et al. (2020b) at different e.
Original
Low pass
Method (c)
Clean Robust
Clean
Robust
FGSM (*/256)
85.6
0.0
81.1
47.0
PGD (*/255)
80.9
50.6
80.3
49.7
FGSM (16/25))
76.3
0.0
78.6
0.0
PGD (16/258)
68.0
29.2
66.9
28.4
CONCLUDING REMARKS
In this work, we have presented a thorough empirical study to establish a causal link between the fea- tures of the data and the onset of CO in FGSM-AT. Specifically, using controlled data interventions we have seen that catastrophic overfitting is a learning shortcut used by the network to avoid learning hard- to-learn robust features while achieving high accuracy using easy non-robust ones. This new perspec- tive has allowed us to shed new light on the mechanisms that trigger CO, as it shifted our focus towards studying the way the data structure influences the learning algorithm. We believe this opens the door to promising future work focused on understanding the intricacies of these learning mechanisms. In gen- eral, we consider that deriving methods for inspecting the data and identifying how different features of dataset interact with each other as in [lyas al. (2019) is another interesting avenue for future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
N. Ahmed, T. Natarajan, and K. R. Rao. Discrete Cosine Transform. IEEE Transactions on Computers, 1974.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In Neural Information Processing Systems (NeurIPS). 2020.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger. Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien A closer look at memorization deep networks. In International Conference on Machine Learning (ICML), 2017.
James Aspnes, Richard Beigel, Merrick Furst, and Steven Rudich. The expressive power of voting polynomials. Combinatorica, 1994.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
Pau de Jorge, Adel Bibi, Riccardo Volpi, Amartya Sanyal, Philip H. S. Torr, Grégory Rogez, and Puneet K. Dokania. Make some noise: Reliable and efficient single-step adversarial training. 18x1v:2202.01181, 2022.
Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfoolii Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zeinab Golgooni, Mehrdad Saberi, Masih Eskandar, and Mohammad Hossein Rohban. Zerograd: Mitigating and explaining catastrophic overfitting in fgsm adversarial training. arXiv:2103. 15476 2021.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR), 2015.
Julia Grabinski. Steffen Jung, Janis Keuper, and Margret Keuper. FrequencyLowCut Pooling Plug & Play against Catastrophic Overfitting. In European Conference on Computer Vision (ECCV), 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016.
Andrew Hyas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Neural Information Processing Systems (NeurIPS), 2019.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? Neural Information Processing Systems (NeurIPS), 2018.
Peilin Kang and Seyed-Mohsen Moosavi-Dezfooli. Understanding catastrophic overfitting in adver- sarial training. arXiv:2 2021.
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfiuting in single-step adversarial training. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Klim Kireev, Maksym Andriushchenko, and Nicolas Flammarion. On the effectiveness of adversarial
training against common corruptions. In James Cussens and Kun Zhang (eds.), Proceedings of the Thirth-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings
of Machine Learning Research, pp. 1021, PMLR, 01-05 Aug 2022. URL https:// proceedings www.mirrreesseeee press / v ww1////iieee2222 html
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
10

---

Under review as a conference paper at ICLR 2023
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations 2017.
Bai Li, Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial training. arXiv:2006.03089 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robust- ness via curvature regularization, and vice versa. In IEEE Conference on Computer Vision and Pultern Recognition (CVPR), 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems (NeurIPS), Workshops, 2011.
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi, and Pascal Frossard. Neural anisotropy directions. Neural Information Processing Systems (NeurIPS), 2020a.
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohse Moosavi, and Pascal Frossard. Hold me tight! influence of discriminative features on deep network boundaries. Neural Information Processing Systems (NeurIPS), 2020b.
Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Optimism in the face of adversity: Understanding and improving deep learning through adversarial robustness. Proceedings of the IEEE, 2021.
Geon Yeong Park and Sang Wan Lee. Reliably fast adversarial training via latent adversarial perturbation. In International Conference Learning Representations (ICLR), Workshops, 2021.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Neural Information Processing Systems (NeurIPS), 2019.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference Machine Learning (ICML), 2020.
Amartya Sanyal, Puneet K. Dokania, Varun Kanade, and Philip Torr. How benign is benign overfitting In International Conference on Learning Representations (ICLR), 2021.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Neural Information Processing Systems (NeurIPS), 2019.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Neural Information Processing Systems (NeurIPS), 2020.
Yuge Shi, Imant Daunhawer, Julia E Vogt, Philip HS Torr, and Amartya Sanyal. How robust pre-trained models to distribution shift? arXiv preprint arXiv:2206.0888 08871, 2022.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations (ICIR), 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018.
BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling. In /EEE Conference Computer Vision and Patlern Recognition (CVPR), 2020.

---

Under review as a conference paper at ICLR 2023
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference Machine Learning (ICML), 2018.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In bmve, 2016.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.
Theoretically principled trade-off between robustness and accuracy. International Conference Machine Learning (ICML), 2019.
Richard Zhang. Making convolutionall neural networks shift-invariant again. In International Conference on Machine Learning (ICML), 2019.
12

---

Under review as a conference paper at ICLR 2023
A ANALYSIS OF THE SEPARABILITY OF THE INJECTED DATASETS
With the aim to illustrate how the interaction between D and V can influence the robustness of a classifier trained D now provide a toy theoretical example in which we discuss this interaction. Specifically, without loss of generality, consider the binary classification setting on the dataset (ar v) DD where y {-1, [-1,+1) +1} and | l2 1, for ease. Let's now consider the injected dataset Do and further assume that v(+1) = w and v(-1) = uu with u E I& and || la = 1, such that + Byu. Moreover, let y E [0, 1] denote the interaction coefficient between D and V, such that
S x
u S 7/-
We are interested in characterizing the robustness of classifier that only uses information in V when classifying Da depending on the strength of the interaction coefficient. In particular, as we are dealing with the binary setting, will characterize the robustness of linear classifier h Rd - {-1,+1) that discriminates the data based only on V, i.e., h())) = =iin(u (u" Φ). In our setting, we have
Σ = ==-- x Bu"
x B if +1
x x Bu uuaa x B if
Proposition (Clean performance). 3 > %, then achieves perfect classification accuracy on D3-
Proof. Observe that if y i.. the features from original dataset D do not interact with the injected features V, the dataset is perfectly linearly separable. However, if the data from D interacts with the injected signal u, i.e. non zero projection, then the dataset still perfectly separable but for sufficiently larger 3, such that u x + B > 0 when = +1 and u x + B < 0 when -1. Because -7 S x u S y this achieved for 3 >%
Proposition (Robustness). If B > 7, the linear classifier h is perfectly accurate and robust to adversarial perturbations in an b2lball of radius e S 3 7V. Or, equivalently, for to be e-robust, the injected features must have a strength
e
Proof. Given E. seek to find the minimum distance to the decision boundary of such classifier. minimum distance problem can cast as solving the following optimization problem:
min rEB
||
x
subject to r T u 0,
which can be solved in closed form
u T
E |u
yB8l
The robustness radius of the classifier h will therefore be inf @esupp(Da) (( which in our case can be bounded by
inf (a.))
c'(x)
min S
Iu x + y
F y 8-7- 7
|u |
Based on these propositions, we can clearly see that the interaction coefficient y reduces the robustness
of the additive features V. In this regard, if 8 - robust classification radius can only be achieved by also leveraging information within
13

---

Under review as a conference paper at ICLR 2023
B ROBUST CLASSIHICATION CAN REQUIRE NON-LINEAR FEATURES
We now provide a rigorous theoretical example a learning problem that provably requires additional complex information for robust classification, even though it achieve good clean performance using only simple features.
Given some p E N, let R be the input domain. A concept class, defined over R is set of functions from Re++ to {0, }.. A hypothesis h is s-non-linear if the polynomial with the smallest degree that represent h has degree (largest order polynomial term) of
Using these concepts we now state the main result.
Theorem 1. For any p. k E N. < 0.5 such that p. there exits a famity of distributions Dy over R and a concept class H. defined over RP+ such that
1. H. is PAC learnable (with respect to the clean error) with linear (degree 1) classifier. However, H is not robustly learnable with any linear classifier.
2. There exists efficient learning algorithm, that given dataset sampled ii.d from distribution D Da robustly learns H.
In particular, the algorithm returns k-non-linear classifier and in addition, the returned classifier also exploits the linear features used by the linear non-robust classifier.
Proof. We now define the construction of the distributions in Dr. Every distribution D in the family of distribution Da is uniquely defined by three parameters: a threshold parameter E (4de: / E {0.... , k]) (one can think of this as the non-robust, easy-to-learn feature), dimensional bit vector E 1) given p and (and which we discuss when necessary and ignore from the notation for simplicity), we can define the distribution De.p We provide an illustration of this distribution for 2 in Figure 7
1)"h such that |c]]| = & (this is the non-linear but robust feature) and Therefore,
Sampling the robust non-linear feature To sample a point (x. E RR++1 from the distribution De.P, first, sample random bit vector & E IRP from the uniform distribution over the boolean hypercube (0, 13". Let Σ Tii] c/i] (mod 2) be the label of the parity function with respect to e evaluated on &. The marginal distribution over G. if sampled this way, is equivalent to the Bernoulli distribution with parameter 2. To see why, fix all bits in the input except one (chosen arbitrarily from the variables of the parity function), which distributed uniformly over {0, 1}. easy to see that this forces the output of the parity function be distributed uniformly over {0, 1} as well. Repeating this process for all dichotomies of 1 variables of the parity function proves the desired result. Intuitively, & constitutes the robust non non-linear feature of this distribution.
Sampling the non-robust linear feature To ensure that å is not perfectly correlated with the true label, we sample the true label y from Bernoulli distribution with parameter 2 the ron-ro feature 2; as follows
1 Then we sample
Unif (X-
y = 0 A 9
Unif (x+)
y = 1 A Φ = 1
Unif (X,
y = 0 OAS Φ
Unif (X²)
y 1 1A 7
where
x = [p, + and X² = (( + 2e, p Xx, = pp-
1 1
and X₂ = 110 - 3e, 2c)]
Finally, we return (æx,) y) where x (æ1; àt) is the concatenation of and ac
Linear non-robust solution First, we show that there is a linear, accurate but non- robust solution to this problem. To obtain this solution, sample m-sized dataset Sn, = {(11 111 .i........ (mm: }n]} E IR++ x [-1,1} from the distribution Dr-e. Ignore all, except the first coordinate, of the covariates of the dataset to create o {x1 [0] v1)..... ... (z [0] ym )
where S; [j] indexes the jth coordinate of the 7th element of the dataset. Then, sort 8 on the basis of the covariates (i.e. the first coordinate). Let á be the largest element whose label is 0.
14

---

Under review as a conference paper at ICLR 2023
[1]
(0,1) 1)
OOI 2006
[0, 1)
*
(0. 1
(0..0)
(1.0)
(0.,
(1,0)
(0.0)
æ: (0)
Fur [o] - S
For m/0 pl >
Figure 7: Illustration of one possible distribution De in three dimensions. The data is linearly separable in the direction [0] but has a very smalll margin in that direction. Leveraging x[1] and a additionally, we see how the data can indeed be separated robustly, albeit non-linearly.
Define fin as the linear threshold function on the first coordinate i.e. Jin, (œ) I [=/00] 2 P)- By construction, Jinn,p accurate classifies all points in Sm. The VC dimension of linear threshold function in 1 dimension is 2. Then, using standard VC sample complexity upper bounds for consistent learners, if m 2 Kσ log + a have that with probability least 1 B,
1 log (+)): where Ko is some universal constant, we
Err (fin.p; DPP)) S
Non-linear robust solution Next, we propose an algorithm to find robust solution and show that this solution has non-linearity of degree k. First, sample the m-sized dataset S,,, and use the method described above to find P. Then, create a modified dataset S by first removing all points from Sm, such that [z [0] - أ) and then removing the first coordinate of the remaining points. Thus, each element in S belongs IRP {0, 1} dimensional dataset.
Note, that by construction, there is a consistent (i.e., an accurate) parity classifier on S. Let the parity bit vector consistent with S be Ć. This can be learned using Gaussian elimination. Consequently, construct the parity classifier Spar,a = Ei= a [i] [(] (mod 2).
Finally, the algorithm returns the classifier Ia.e. which acts as follows:
[ > X -
gá.a (a)
[<0[0] < 0
(6)
fpar, (x)
O.w.
where क round (ar [1,....P] pl) obtained by rounding off æ starting from the second index till the last. For example, if [0.99, 0.4, 0.9, 0.8).
0.2, and c: 10, 0, 1, then T [0, 1, 1] and 90.5.e [ = 1. Finally. it is easy to verify that the classifier gp.e is accurate on all training points and the number of total parity classifiers is less than 2 (hence finite VC dimension), as long as m 2 K1 (1 log
3 + log l2)) where 11 is some universal constant, we have that with
probability least B,
Err (gp.e: D S a.
Robustness of ga.e As x [0] is distributed uniformly in the intervals [p c, c, pl c), we have that 1p A v Eir p; DC.P) I 4ea. Therefore, when m is large enough (m = poly (t)) such that a < 32 we have that p) < S. Intuitively, this guarantees that gp.é uses the linear threshold function on [0] for classification in the interval [e,p + e] [e, p-p-6 E] and fpar. in the [p 2c, p + 3c] U [p- 2c, 3c). A crucial property of 9a.e is that for all x E Supp (Dr:e), the classifier Jr.e does alter prediction in an loo ball radius We show this by studying four separate cases. First, we prove robustness along all coordinates except the first.
1. When 0 أ 2 + as shown above, ga.a invariant to all [i] for all i 0 and is thus robust against all foo perturbations against those coordinates.
cs. Tectures//ectured0:
15

---

Under review as a conference paper at ICLR 2023
2. When x - < c 8. due Equation 6 we have that gp.c Spar.2 (Ø) where ã = round (x [I...., pl) is obtained by rounding off all indices of a except the first. As the rounding operation less than 0.5, we have that Ip.e is robust to all lo perturbations of radius less than 0.5 on the support of the distribution Do.
the boolean hypercube is robust any foc perturbation of radius
Next, we prove the robustness along the first coordinate. Let < 8 < e erprrssen an adversarial pertur- bation. Without loss of generality, assume that [0] ý similar arguments apply for the other case.
1. Consider the case x [0] S Á + c 8. Then, x [0] 8- -- s c + - 5 S c c+ and
hence, by construction, 9a.e (a) = -a.e (10 [0] ò; [x] [T....,p]]] -Pl). On the other hand, for all 6, we have that Ip.c ([x [0] [d] [1, . pll) lifggee (z)
2. For the case x: [0] 2 أ + + c 82 the distribution is supported only on the interval [o + 2e, 3e]. When a positive 8 is added to the first coordinate, the classifier's prediction does not change and it remains 1. For all S 5. when the perturbation is subtracted from the first coordinate, its first coordinate is still larger than ő + e + g and hence, the prediction is still 1.
This completes the proof of robustness of ga.e along all dimensions to loo perturbations of radius less than e, Combining this with its error bound, we have that AdV (9p,6) De.P) S a.
To show that the parity function is non-linear, we use classical result from Aspnes et al. (1994). Theorem 2.2 in Aspnes et al. (1994) shows that approximating the parity function in k bits using a polynomial of degree é incurs at least Σn (+) where fo- 8-6-1 mistakes. Therefore, the lowest degree polynomial that can do the approximation accurately at least k.
2
This completes our proof of showing that the robust classifier is of non-linear degree k while the accurate classifier is linear. Next, we prove that no linear classifier can be robust. We show this by contradiction.
No linear classifier can be robust Construct set Z of s (to be defined later) points in Re+1 by sampling the first coordinate from the interval [p, + <] and the remaining p coordinates uniformly from the boolean hypercube. Then, augment the set by subtracting e from the first coordinate while retaining the rest of the coordinates. Note that this set can be generated, along with its labels, by sampling enough points from the original distribution and discarding points that do not fall this interval. Now construct adversarial examples of each point in the augmented set by either adding or subtracting c from the negatively and the positively labelled examples respectively and augment the original set with these adversarial points. For large enough s this augmented set of points can be decomposed into multiset of points, where all points in any one set has the same value in the first coordinate but nearly half of their label is zero and the other half one.
Now, assume that there is a linear classifier that has a low eтrоr on the distribution Dr.P, Therefore the classifier is also accurate on these sets of points as the classifier is robust, by assumption, and the union of these sets occupy a significant under the distribution D However, as the first coordinate of every point within a set constant despite half the points having label one and the other half zero, the coefficient of the linear classifier can be set to zero without altering the behavior of the classifier. Then, effectively the linear classifier is representing a parity function on the rest of the p coordinates. However, we have just seen that this is not possible as a linear threshold function cannot represent a parity function on & bits where le > 1. This contradicts our initial assumption that there is a robust linear classifier for this problem.
This completes the proof.
"There is slight technicality as might not obtain points that exact reflections of each other around p but that can be overcome by discretising upto certain precision
16

---

Under review as a conference paper at ICLR 2023
C EXPERIMENTAL DETAILS
In this section we provide the experimental details for all results presented in the paper. Adversarial
training for all methods and datasets follows the fast training schedules with a cyclic learning rate introduced in Wong et al. (2020). We train for 30 epochs on CIFAR Krizhevsky & Hinton (2009)
and 15 epochs for SVHN Netzer et al. (20TT) following Andrishhhennoo & Flammarion (020))
When we perform PGD-AT we use 10 steps and step size a 2/253; FGSM uses a step size
of a = 6. Regularization parameters for GradAlign AAnriushchhenoo Flammarion (2020) and N-FGSM de Jorge et al. (2022) will vary and are stated when relevant in the paper. The architecture employed is a PreactResNett
He et al. (2016). Robust accuracy is evaluated by attacking the trained models with PGD-50-100 10, ie. PGD with 50 iterations and 10 restarts. In this case we also employ step size a = 2/255 as in Wong et al. 2020). All accuracies are averaged after training and evaluating with 3 random seeds.
The curvature computation is performed following the procedure described in Moosavv-Dezzooll (2019). As they propose, loss with respect to the input, i.e.,
Dezfooli et al. finite differences to estimate the dircctional second derivative of the
= Vol(fola
+ tw), - vaL(fo(o tw),
w T V2 (2(((()))
21
with >> 0 and use the Lanczos algorithm to perform partial cigendeccoposition of the Hessian without the need to compute the full matrix. In particular, we pick t 0.1.
All our experiments were performed using a cluster equipped with GPUs of various architectures. The estimated compute budget required to produce all results in this work is around 2, 000 GPU hours (in terms of NVIDIA V100 GPUs).
D INDUCING CATASTROPHIC OVERFITTING WITH OTHER SETTINGS
In Section 3 we have shown that CO can be induced with data interventions for CIFAR-10 10 and fox. perturbations. Here we present similar results when using other datasets (i.e. CIFAR-100 and SVHN) and other types of perturbations (i.e. &2 attacks). Moreover, we also report results when the injected features vyy) follow random directions (as opposed to low-frequency DCT components). Overall, we find similar findings to those reported the main text.
D.1 OTHER DATASETS
Similarly to Section 3 we modify the SVHN, CIFAR-100 and high resolution Imagenet-100 100 and TinyImagenet datasets to inject highly discriminative features vyy) Since SVHN also has 10 classes, we use the exact same settings as in CIFAR- 10 and we train and evaluate with = 4 where training on the original data does not lead to CO (recall B = 0 corresponds to the unmodified dataset). On the other hand, for CIFAR-100 and Imagenet- 100 we select v(v) to be the 100 DCT components with lowest frequency and we present results with 6 5 and 6 4 respectively. Similarly, for Tinylmagenet (which has 200 classes) use the first 200 DCT components and present results with
e == 6. Moreover, for ImageNet-100 we evaluate robustness with AutoAttack Croce & Hein (2020). Regarding the training settings, for CIFAR 10/100 and SVHN datasets
the same settings as Andriushchenko & Flammarion 2020 for ImageNet- 100 we follow Kireev et al. (022) and for
TinylmageNet Li al. 20200)
In all datasets we can observe similar trends as with CIFAR- 10: For small values of 3 the injected features are not very discriminative due their interaction with the dataset images and the model largely ignores them. As we increase 3, there is range in which they become more discriminative but not yet robust and we observe CO. Finally for large values of 3 the injected features become robust and the models can achieve very good performance focusing only on those.
D.2 OTHER NORMS
Catastrophic overfitting has been mainly studied for C. perturbations and thus we presented experi- ments with foo attacks following related work. However, in this section also present results where
17

---

Under review as a conference paper at ICLR 2023
Clean eecuracy
Robust accuracy
007
00%
80%
40%
00%
0%
10
8 B (/255)
10
(/255)
5/255
Figure 8: Clean and robust performance after FGSM-AT on injected datasets D3 constructed from CIFAR-100. FGSM-AT already suffers CO on CIFAR- 100 c 6/255 we use 5/255 this experiment where FGSM-AT does not suffer from CO as seen for 3 O. In this setting, we observe CO happening when slightly smaller than e. Results are averaged over seeds and shaded areas report minimum and maximum values.
Clean arruracy
Rohust ammuaay
100%
100%
80%
90%
60%
80%
00%
70%
20%
2
4
10
12
14
6
12
14
9 (/25))))
(/285)
= 4/266
Figure 9: Clean and robust performance after FGSM-AT injected datasets D3 constructed from SVHN. As FGSM-AT already suffers CO on SVHN at = 6/255 we use = 4/255 this experiment where FGSM-AT does not suffer from CO seen for B 0. In this setting, observe CO happening when 3 x e. Results are averaged over seeds and shaded areas report minimum and maximum values.
Clean accuracy
Robust accuracy
100%
60%
80%
40%
60%
20%
B (/255)
(/256)
6/266
Figure 10: Clean and robust performance after FGSM-AT on injected datasets D3 constructed from TinylmageNet. We use = 9/255 in this experiment where FGSM-AT does not suffer from CO as seen for 0. this setting, we observe CO happening for 3 175/255, 8//255).
we induce CO with l2 perturbation which are also widely used adversarial robustness. In Fig. we show the clean (left) and robust (right) accuracy after FGM-A1 on our injected dataset from CIFAR-10 (Dz). Similarly
results with &oo attacks, also observe CO as the injected features
'FGM the & version of FGSM where do not take the sign of the gradient.
18

---

Under review as a conference paper at ICLR 2023
Table 2: Test performance of FGSM-AT trained on different injected versions of ImageNet- 100 with 4/255. We observe that for / > FGSM-AT clearly suffers from CO.
Clean accuracy
FGSM accuracy
Robust accuracy
3 /255
71.0%
49.1%
45.6%
255
70.9%
51.1%
45.1%
/255
74.2%
52.6%
42.1%
6/255
100%
99.8%
0.0%
6/255
100%
98.0%
1.9%
become more discriminative (increased 3). is worth mentioning that the & norm we use (c 1.5) is noticeably larger than typically used in the literature, however, it would roughly match the magnitude of an too perturbation with 7/255. Interestingly, we did not observe CO for this range of / with
Clean accuracy
Robust aceuracy
100%
20%
96%
90%
15%
85%
10%
8%
75%
0.3
11.3
1.0
1.5
2.0
2.3
3.0
0.0
0.3
1.0
1.5
2.0
2.5
3.0
(/255)
(/255)
1.5/255
Figure 11: Clean and &g robust performance after FGSM-AT on injected datasets Dg constructed from CIFAR-10. FGM-AT suffers CO on CIFAR-10 around e = 2, so we use e = 1.5 in this experiment where FGM-AT does not suffer from CO as seen for 3 = 0. In this setting, we observe CO happening when e. Results are averaged over 3 seeds and shaded areas report minimum and maximum values.
D.3 OTHER INJECTED FEATURES
We selected the injected features for our injected dataset from the low frequency components of the DCT to ensure an interaction with the features present on natural images Ahmed et al. (1974). However, this does not mean that other types features could not induce CO. In order to understand how unique
also created another family of injected datasets but this time using set of 10 randomly generated vectors as features. As in the main text, we take the sign of each random vector ensure they take values in {-1, +1} and assign one vector per class. In Fig. 12 we observe that using random vectors as injected features we can also induce CO. Note that since our results are averaged over random seeds, each seed uses different of random vectors.
choice of features
D.4 OTHER ARCHITECTURES
In all our previous experiments trainde a PreActResNet18 (He et al. 2016 it is the standard architecture used in the literature. However, our observations our also robust to the choice of architec- also induce CO when training WideResNet28x Zagoruyko & Komodakis 2016) on an injected version of CIFAR-10.
ture. As
in Fig. [3]
E LEARNED FEATURES AT DIFFERENT B
In Section 3 we discussed how, based on the strength of the injected features 3, our injected datasets seem to have 3 distinct regimes: (i) When 3 is small we argued that the network would the
19

---

Under review as a conference paper at ICLR 2023
Clean accuracy
Robust aceuracy
100%
100%
80%
60%
60%
10%
20%
40%
0%
12
(/255))
(/255)
0/256
Figure 12:
Clean and robust performance after FGSM-AT on injected datasets D3 constructed from CIFAR-10 using random signals in V. We perform this experiments at 6/255 where injected the dataset with the DCT basis vectors did induce CO. In the random V setting, we observe the same behaviour, with CO happening when B = c. Results averaged over seeds and shaded areas report minimum and maximum values.
that
Clean accuracy
Robust ameuaac
100%
80%
60%
98%
10%
92%
90%
20%
88%
0%
2
12
12
(/2255))
(/255)
4/255
Figure 13: Clean and robust performance after FGSM-AT on injected datasets D3 constructed from CIFAR-10 when training WideResNNe12x100 We perform this experiments at 4/255. Results averaged over 3 seeds and shaded areas report minimum and maximum values.
injected features as these would not be very helpful. (ii) When 3 would have a very large value then the network would only look these features since they would be easy-to-learn and provide enough margin to classify robustly. (iii) Finally, there was a middle range of 3 usually when 8 ~ where the injected features would be strongly discriminative but enough provide robustness on their own. This regime is where we observe CO.
In this section we present an extension Fig, 3 where we take FGSM trained models on the injected datasets (D3) and evaluate them on three test sets: (i) The injected test set (D₂) with the same features as the training set. (ii) The original dataset (D) where the images are unmodified. (iii) The shuffled dataset (D (D)) where the injected features are permuted. That is, the set of injected features is the same but the class assignments are shuffled. Therefore, the injected features will provide conflicting information with the features present the original image.
In Fig. we show the performance on the aforementioned datasets for three different values of B. For 2/25 we in regime (i) : observe that the datasets have the same performance, i.e. the information of the injected features does not seem to alter the result. Therefore, we can conclude the network is mainly using the features from the original dataset D. When / 20/255 we are in regime (ii) : the clean and robust performance of the network is almost perfect on the injected test set D3 while it is close to 0% (note this is worse than random classifier) for the shuffled dataset. So when the injected and original features present conflicting information the network aligns with the injected features. Moreover, the performance the original dataset is also very low. Therefore, the network mainly using the injected features. Lastly. 8/255 corresponds to regime (iii) as discussed in Section 4.1 in this regime the network initially learns combine information from both the original
20

---

Under review as a conference paper at ICLR 2023
and injected features. However, after CO, the network seems focus only the injected features and discards the information from the original features.
PGSM Inining (8 22/256)
FGSM Lraining (3 8/255)
FGSM Imiinng (9 20/255)
6006
Clou
070
17%
10
20
1007
Robozt noeroey
409
0055
30%
2006
20%
09%
1030
Epoch
Epoch
Injected (Pg)
CIPARIO
Sleiffld (7)ag)
Figure 14: Clean (top) and robust (bottom) accuracy of FGSM-AT on D3 at different 3 values on
different test sets: (i) the original CIFAR-10 (D), (ii) the dataset with injected features DD and (iii) the dataset with shuffled injected features D Da(s) All training runs use = 6/255. Left:: 3 = 7/265
Center: 3 = 8/255 Right: B 20/255.
F ANALYSIS OF CURVATURE IN DIFFERENT SETTINGS
In Fig. 1 (left) we track the curvature of the loss surface while training on different injected datasets with either PGD-AT or FGSM-AT. We observe that (i) Curvature rapidly increases both for PGD-AT and FGSM-AT during the initial epochs of training. (ii) In those runs that presented CO, the curvature explodes around the 8th epoch along with the training accuracy. (iii) When training with the dataset with orthogonally injected features (D) the curvature does not increase. This is aligned with our proposed mechanisms to induce CO whereby the network increases the curvature in order to combine different features to learn better representations. this section we extend this analysis to the original CIFAR-10 dataset (as opposed to our injected datasets) and to different values of feature strength B on the injected dataset (Dz). For details on how estimate the curvature refer Appendix C
In Fig. 15 we show the curvature when training on the original CIFAR-10 10 dataset with < = $/255 (where CO happens for FGSM-AT). Similarly to observations the injected datasets, the curvature during FGSM-AT explodes along with the training accuracy while for PGD-AT the curvature increases very similar rate than FGSM-AT during the first epochs and later stabilizes. This indicates that our described mechanisms may as well apply to induce CO on natural image datasets.
On the other hand, Fig. presents the curvature for different values of feature strength on the injected dataset (Dg). We show three different values of representative of the three regimes discussed in Appendix Recall that when 3 is small (B 2/255) we observe that the model seems focus only on CIFAR-I 10 features. Thus, we observe curvature increase aligned with (CIFAR-10) feature combination. However, since for the chosen robustness radii 6/255 there is no CO, observe that the curvature increase remains stable. When 3 is quite large (3 = 20/235) then the model largely ignores CIFAR-10 information and focuses the easy-to-learn injected features. Since these features are already robust, there is no need to combine them and the curvature does not need to increase. In the middle range when CO happens (3 8/255) observe again the initial curvature increase and then curvature explosion.
G ADVERSARIAL PERTURBATION BEFORE AND AFTER CO
Qualitative analysis order to further understand the change in behaviour after CO we presented visualizations of the FGSM perturbations before and after CO in Fig. 2 We observed that while prior
21

---

Under review as a conference paper at ICLR 2023
Curvature
Training attack accurary
1075
10
8058
10
609
10'3
4096
20%
10
20
26
30
6
10
16
20
20
30
Epoch
Egoch
FGSM training
PGD-10 training
Figure 15: Evolution of curvature and training attack accuracy of FGSM-AT and PGD-AT trained on the original CIFAR-10 with 8/255. When CO happens the curvature explodes.
Curvature
Training attack accuracy
10038
10
80%
100
6156
10-3
0156
10-3
0
20
25
303
5
10
25
30
Epocl
Epoch
B 2/255
8/255
20/255
Figure 16: Evolution of curvature and training attack accuracy of FGSM-AT and PGD-AT trained on D3 different and for e = 6/2ss. Only when CO happens (for 8/25s) the curvature explodes. For the other two interventions the curvature does not increase so much. We argue this is because the network does not need disentangle 'D from V, ignores either one of them.
CO, the injected feature components v(y) clearly identifiable, after CO the perturbations do not seem to point in those directions although the network is strongly relying on them to classify. In Fig. [7 and Fig. [8 we show further visualizations of the perturbations obtained both with FGSM or PGD attacks on networks trained with either PGD-AT or FGSM-AT respectively.
We observe that when training with PGD-AT, i.e. the training does suffer from CO, both PGD and FGSM attacks produce qualitatively similar results. In particular, all attacks seem to target the injected features with some noise due the interaction with the features from CIFAR-10. For FGSM-AT, we observe that at the initial epochs (prior to CO) the pertubations are similar to those of PGD-AT, however, after CO perturbations change dramatically both for FGSM and PGD attacks. This aligns with the fact that the loss landscape of the network has dramatically changed, becoming strongly non-linear. This change yields single-step FGSM ineffective, however, the network remains vulnerable and multi-step attacks such PGD are still able to find adversarial examples, which in this case do not point in the direction of discriminative features Jettley et al. (2018) Ilyas et al. (2019).
Quantitative analysis Finally, to quantify the radical change of direction of the adversarial pertur- bations after CO, we compute the evolution of the average alignment (i.e., cosine angle) between the FGSM perturbations 8 and the injected features, such that point æ is associated with class we com- (8.v(y)) Figure 19 (left) shows the results of this evaluation, where we can see that before CO pute |||())|la||@|
there non-trivial alignment between the FGSM perturbations and their corresponding injected fea- tures, that after CO quickly converges to the same alignment as the one between two random vectors.
To complement this view, we also perform analysis of the frequency spectrum of the FGSM perturbations. Fig. 19 (right), we plot the average magnitude of the DCT transform of the FGSM perturbations computed on the test set of an intervened version of CIFAR- 10 during training. As can see, prior to CO, most of the energy the perturbations is concentrated around the low frequencies (remember that the injected features are low frequency), but after CO happens, around epoch 8, the
22

---

Under review as a conference paper at ICLR 2023
Inrooed
Invenee supple
suups
qpodd
Aran spoodda 20
upachh
(a) FGSM adversarial perturbations
(b) PGD adversarial perturbations
Figure 17: Different samples of the injected dataset Ds. and adversarial perturbations at epoch 4 and 22 of PGD-AT on Di at c = 6/255 and 3 $/255 (where FGSM-AT suffers CO). The adversarial perturbations remain qualitatively similar throughout training and align significantly with V.
suupla ie oi Dy
cumpler in 9)
frcms et spodit
fren spedis
(a) FGSM adversarial perturbations
(b) PGD adversarial perturbations
Figure 18: Different samples of the injected dataset Dg, and adversarial perturbations at epoch
(before CO) and 22 (after CO) of FGSM-AT on D) at e = 6/255 and B = 8/255 (where FGSM-AT suffers CO). The adversarial perturbations change completely before and after CO. Prior to CO, they align significantly with V, but after CO they point to meaningless directions.
energy of the perturbations quickly gets concentrated towards higher frequencies. These two plots corroborate, quantitatively, our previous observations, that before CO, FGSM perturbations are pointing towards meaningful predictive features, while after CO, although we know the network still uses the injected features (see Fig. the FGSM perturbations suddenly point in different direction.
H FURTHER RESULTS WITH N-FGSM, GRADALIGN AND PGD
In Section 5 we studied different SOTA methods that have been shown to prevent CO. Interestingly. we observed that order to avoid CO on the injected dataset stronger level of regularization needed. Thus, indicating that the intervention is strongly favouring the mechanisms that lead to CO. For completeness, Fig. 20 we also present results of the clean accuracy (again with the robust accuracy). As expected, for those runs in which we observe CO, clean accuracy quickly saturates. Note that for stronger levels of regularization the clean accuracy is lower. An ablation of the regularization strength might help improve results further, however the purpose of this analysis not to improve the performance on the injected dataset but rather to show it is indeed possible to prevent CO with the same methods that work for unmodified datasets.
I FURTHER DETAILS OF LOW-PASS EXPERIMENT
We expand here over the results in Section [ and provide further details on the experimental settings of Table [T. Specifically, replicate the same experiment, i.e., training low-pass version of CIFAR-10
23

---

Under review as a conference paper at ICLR 2023
Aligament of ó and injected lealures
Evolution of perturisttion spectrum
Co
FGSM attacks
0.00
Raudom busline
Moguirude (dB)
DCT frem:
20
J5-
10
6.02
6-
0
30
15
30
Epoch
Epoch
Figure 19: Quantitative analysis of the directionality of the FGSM perturbations in the test set during FGSM-AT before and after CO when training the injected CIFAR-IO with / = 8/255 and 6/255. (Left) Evolution of alignment FGSM perturbations with their corresponding injected features during training. The red dotted line shows the expected value of alignment between two random vectors the same dimensionality as CIFAR-10 images. (Right) Evolution of the average magnitude of the DCT spectrum of the same FGSM perturbations during training. The plot shows only the diagonal components of the DCT at every epoch.
Clean accuracy
Robust necuracy
1.0
80%
0.9
00%
0.8
20%
0.7
0%
B (2265)
1/200)
PGD-10
Grad 0.112)
GradlAlgg (A
X-FGSM (R
10c)
N-FGSM (2 2c)
Figure 20: Clean (deft) and robust (right) accuracy after AT with PGD-10, GradAlign and N-FGSM on D at e = 6/255. Results averaged over three random seeds and shaded areas report minimum and maximum values.
using FGSM-AT at different filtering bandwidths. As indicated in Section 5 we use the low-pass filter introduced in Ortizz-Jmeeezz et al. (2020b) which only retains the frequency components in the upper left quadrant of the DCT transform of an image. That is, a low- cow-pass filter of bandwidth W would retain the W x W upper quadrant of DCT coefficients of all images, setting the rest of the coefficients to
Figure 21 shows the robust accuracy obtained by FGSM-AT CIFAR-10 versions that have been pre-filtered using such a low-pass filter. Interestingly, while training on the unfiltered images does induce CO FGSM-AT, just removing few high-frequency components is enough prevent CO e= 8255. However, as described before, it seems that at 16/255 no frequency transformation can avoid CO. Clearly, this transformation cannot be used technique to prevent CO, but highlights once again that the structure of the data plays significant role in inducing CO.
Relation with anti-aliasing pooling layers As mentioned Section 5 our proposed low-passing technique is very similar in spirit to works which propose using anti aati-aiiasin low-pass filters at all pooling layers (Grabinski et al. 2022 Zhang 2019). Indeed, as shown by Orttiz-lmmenz et al. (2020b), CIFAR10 contains a significant amount of non-robust features on the high-frequency end of the spectrum due to aliasing produced in their down-sampling process. In this regard, it is no surprise that methods like the one proposed in Grabinski et al. (2022) can prevent CO at = $/255 using the same training protocol as in our work (robust accuracy is 45.9%). Interestingly, though, repeating the experiments in Grabinski et al. (2022) work using accuracy is 0.0%) in Section
16/255 does lead to CO (robust 5 This result was not reported in the original paper, but we see it as a corroboration of our observations. Indeed, features play role in CO, but the problematic features
24

---

Under review as a conference paper at ICLR 2023
Robust accuracy
50%
2055
1098
055
10
15
20
25
30
Bandwieth
PGSM-AT (c= 2250
PGD-AT 8/255
PGSM-AT (e- 16/255)
PGD-AT (e- 16/255)
Figure 21: Robust accuracy of FGSM-AT and PGD-AT on different low-passed versions of CIFAR-10 using the DCTT-assed low pass filter introduced in Ortiz-Jimenez al. (2020b). Bandwidth 32 corresponds to the original CIFAR-IO, while smaller bandwidths remove more and more high- frequency components. At e = 8/255 just removing a few high-frequency components is enough to prevent CO, while at 16 255 no frequency transformation avoids CO.
do not always come from excessively high-frequencier or aliasing. However, we still consider that preventing aliasing in the downsampling layers is promising avenue for future work in adversarial robustness.
25


Reviews:
[{"review_id": "qqSGXaqSXa", "reviewer": "Reviewer_F9cZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors investigate the phenomenon of catastrophic overfitting (CO) which occurs during FGSM adversarial training (AT). CO describes the “overfitting” of the model to simple adversaries (FGSM) during a certain amount of training epochs while losing robustness against more complex adversaries like PGD.  \n\nThe authors show empirically:\n\n1. that it is possible to induce catastrophic overfitting on smaller \\epsilon values (where \\epsilon is the hyperparameter to determine the strength of the adversarial attack) than before by adding simple discriminative features to the dataset. These injected features are added onto the original image scaled with a hyperparameter $\\beta$\n2. if $\\beta << \\epsilon$ the model behaves normal \n3. if $\\beta >> \\epsilon$ the robust accuracy is high as the model can rely on the simple features\n4. if $\\beta \\approx \\epsilon$ they can induce CO at \\epsilon values at which the model on the clean data would not suffer from CO\n\nThe part where $\\beta \\geq \\epsilon$ is further exploited to analyse the network's training behavior, especially at the point of CO. Further, the authors empirically show:\n\n1. that standard training and FGSM training after CO highly rely on the simple features while PGD training forces the model to learn more robust features\n2. that the curvature of the model’s loss landscape explodes after CO  \n\nThe authors show empirically that models favour easy-to-learn features when no robustness constraints are considered. Further, they claim that a network that needs to learn different features needs to increase its non-linearity and thus opens a shortcut to breaking FGSM with CO.\n\nFurther, the authors analyse two methods which can prevent CO (GradAlign and N-FGSM). They empirically show that these methods can keep low curvature of the loss landscape thus preventing CO. Additionally, the authors try to use a lowpass filter on top of the data but could not prevent CO for large $\\epsilon$ values.\n", "summary_of_the_review": "The contribution of this paper is a simple, yet interesting empirical analysis of CO. However, the authors missed a new perspective from previous work. Thus, I would like to encourage the authors to include this perspective in their analysis. Further, it would be interesting to see if the presented results hold for high-resolution datasets and bigger networks. \nAdditionally, the authors should provide their code for reproducibility and enablement of future research on the analysis of the explainability of CNNs.\n\nGenerally, the suggested idea and findings are interesting and worse publishing. However, I can not accept the paper for now due to my concerns mentioned above. I highly encourage the authors to address these concerns such that I can increase my score. \n", "strength_and_weaknesses": "**Strength:**\n- The paper is well-written and supported by adequate graphics to demonstrate the author's findings.\n- The authors provide an empirical analysis of the learning preferences of CNNs under different training settings (standard training, FGSM AT and PGD AT)\n- The approach of injecting simple features for the analysis of CO is interesting and can be extended for future explainability studies on CNNs\n\n**Weakness:**\n\n- Prior work/extensiveness: the authors investigated prior work that prevents CO, however, missed prior work which includes additional insights into why CO happens:                                   \n[0] https://arxiv.org/abs/2204.00491 (ECCV 2022): showing that CO is correlated with a vast increase in aliasing and preventing CO by using an aliasing-free downsampling layer\n\n- The authors conducted their empirical study only on low-resolution datasets and for small networks. It would be interesting to see if their approach also applies to high-resolution data and bigger networks.\n- The authors did not provide any code for reproducibility. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q-SKnHh1Ad", "reviewer": "Reviewer_2rv5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "the paper investigates possible causes of catastrophic overfitting (CO), a well known state in adversarial training (AT) of robust model which leads to the collapse of model robustness during later stages of AT. The authors propose to augment the data with easy to classify DCT patterns during analisis, in order to gain insights on the properties of the feature space leading to OT.  ", "summary_of_the_review": "the initial question of how the feature space representation affects the CO is indeed very interesting and requires further investigation. However, I think that presented approach is not suitable to achieve this goal because the chosen augmentation of training data will trigger clean overfitting and thus prevents an unbiased investigation.\n\nI would suggest that the authors investigate the features at CO without external manipulation. [1+2] provided a large DB of models which could be used to conduct these studies and have already shown that robust features are only different in specific layers of a network...\n\n[1] Gavrikov et. all Adversarial Robustness through the Lens of Convolutional Filters, CVPR-W 2022\n\n[2] Gavrikov et. all. CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters, CVPR 2022", "strength_and_weaknesses": "Strengths:\n* the paper addresses an important and interesting question \n* the paper introduces the OT problem well and gives a quite good overview of the related work (missing only a view important works [1-3])\n\n[3] Grabinski et. all: FrequencyLowCut Pooling--Plug & Play against Catastrophic Overfitting, ECCV 2022\n\nWeaknesses:\n\n1) the title is overclaiming: overfitting is NOT a bug and the paper fails to show that OT is indeed caused by features\n2) novelty: the paper fails to show new insights: OT is overfitting - this is in line with all related work, all known counter measures (augmentation, regularization, input smoothing ...) and the presented experiments only confirm this\n3) experimental setup is not suited to answer the questions defined at the beginning of the paper. Adding easy to learn patterns to the data, leading 100% clean ACC is definitely provoking ofervitting. The described case of beta=eps does not allow strong conclusions other than OT is overfitting. Additional results on the nonlinearity and non-smoothness of the optimization space only further confirm this (and have been shown before)   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cm65VkUqsbT", "reviewer": "Reviewer_GG1w", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tried to explain the cause of catastrophic overfitting (CO) during FGSM adversarial training. The authors first showed that CO can be deliberately induced by dataset intervention. Then, using this intervened dataset as a starting point, the authors conducted three well-designed experiments to study the chain of mechanisms behind CO. They conclude that in FGSM-AT:\n1. The network attempts to learn both non-robust and robust features.\n2. The network increases its non-linearity for better learning these two types of features.\n3. The explosion of non-linearity breaks FGSM, which provides a shortcut for networks to only learn from non-robust features, which eventually leads to CO.\n\n", "summary_of_the_review": "Overall it’s a solid and well written paper. The way the authors design experiments is novel. Both empirical results and theoretical justifications are provided. And they successfully fill the knowledge gap on the onset of CO. The only weakness is that they seem to over-claim their contribution in dataset construction. The authors stated that it’s hard to separate robust features from non-robust features in real datasets, so they used their way of dataset construction. But (Ilyas et al., 2019) seems to have successfully disentangled these two features from neural networks’ perspective. In (Ilyas et al., 2019), a method is provided to extract non-robust features and robust features. So why don't the authors use their methodology?\n", "strength_and_weaknesses": "S1. The experiments are well designed and convincing.\nS2. The theory for onset of CO is both empirically and theoretically justified.\nS3. The related work is well cited and discussed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SGitGul_aq", "reviewer": "Reviewer_27L3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors attempt to understand catastrophic overfitting in adversarial training with an FGSM adversary. To analyze catastrophic overfitting, they proposed to induce it using controlled manipulations of the dataset technique. Depending on the magnitude of the data perturbations, they observed that it is possible to induce catastrophic overfitting to any model given sufficiently large dataset perturbation. The authors discussed the implications of this finding and provided further insights into catastrophic overfitting.", "summary_of_the_review": "This paper provides novel insights into catastrophic overfitting with fast adversarial training. Understanding catastrophic overfitting is a very important problem in adversarial machine learning and addressing it is extremely important. To this end, the authors proposed a novel method to induce catastrophic overfitting in controlled settings, provided possible explanations of this phenomenon, and suggested a few directions to reduce catastrophic overfitting in practice. Overall, the paper is interesting and enjoyable to read and makes few contributions.", "strength_and_weaknesses": "### Strengths\n\n- A novel approach to understanding catastrophic overfitting based on controlled manipulations of the dataset, which provides interesting insights into the problem.\n- A novel explanation of catastrophic overfitting that can be used to guide the development of new defense methods.\n- Detailed extensive empirical analysis of catastrophic overfitting under different dataset manipulations.\n- The paper is really easy and enjoyable to read.\n\n### Weaknesses\n\n- Technical novelty is limited, all results are very intuitive.\n- At the time of the review, the author’s code is not available, which will be hopefully addressed in the revision phase.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject