Catastrophic overfitting can be induced with discriminative non-robust features
Authors: Guillermo Ortiz-Jimenez
Pau de Jorge
Amartya Sanyal
Adel Bibi
Puneet K. Dokania
Pascal Frossard
Grégory Rogez
Philip Torr
Venue: Accepted by TMLR ()

Abstract: Adversarial training (AT) is the de facto method for building robust neural networks, but it can be computationally expensive. To mitigate this, fast single-step attacks can be used, but this may lead to catastrophic overfitting (CO). This phenomenon appears when networks gain non-trivial robustness during the first stages of AT, but then reach a breaking point where they become vulnerable in just a few iterations. The mechanisms that lead to this failure mode are still poorly understood. In this work, we study the onset of CO in single-step AT methods through controlled modifications of typical datasets of natural images. In particular, we show that CO can be induced at much smaller $\epsilon$ values than it was observed before just by injecting images with seemingly innocuous features. These features aid non-robust classification but are not enough to achieve robustness on their own. Through extensive experiments we analyze this novel phenomenon and discover that the presence of these easy features induces a learning shortcut that leads to CO. Our findings provide new insights into the mechanisms of CO and improve our understanding of the dynamics of AT.

Published in Transactions on Machine Learning Research (07/2023)
Catastrophic overfitting can be induced with discriminative non-robust features
Guillermo Ortiz-Jimenez"
gullermmo.orteihneeeedeeepp.od
Ecole Polytechnique Fédérale de Lausonne
Pau de Jorge'
pouskrobots.or.cco
University of Orford Naver Labs Europe
A Amartya Sanyal
amarttp..ongulituubinnnn.....
ETH Zürich
Mar Planck Institute for Intelligent Systems, Tuebingen
Adel Bibi
Bielbbbbi@egggooo or..nggorrer uk
University of Oxford
Puneet K. Dokania
puneel@robols.or www.:trroootsoo...cc.. oz.ac. uk
University of Orford Five AT Ltd.
Pascal Frossard
frosalffrossarrdeepp..c
Ecole Polytechnique Fédérale de Lausonne
Grégory Rogez Nover Labs Europe
grggorg.agecrhhaeerddb..om
Philip H.S. Torr University of Oxford
Whillp.torm@eng.r..co
Reviewed on OpenReview: https: // openreview. net/ forum? id= 10hCbu70Sr
Abstract
Adversarial training (AT) is the de facto method for building robust neural networks, but can be computationally expensive. To mitigate this, fast single-step attacks can be used, but this may lead to catastrophic overfitting (CO). This phenomenon appears when networks gain non-trivial robustness during the first stages of AT, but then reach breaking point where they become vulnerable in just few iterations. The mechanisms that lead to this failure mode are still poorly understood. In this work, we study the onset of CO in single-step AT methods through controlled modifications of typical datasets of natural images. particular, we show that CO can be induced at much smaller e values than was observed before just by injecting images with seemingly innocuous features. These features aid non-robust classification but not enough to achieve robustness their own. Through extensive experiments
analyze this novel phenomenon and discover that the presence of These easy feat ures induces a learning shorteut that leads to CO. Our findings provide new insights into the mechanisms of CO and improve our understanding of the dynamics of AT.
"The first. authors contributed equally. Guillerma Ortiz-Jimenes did this work while visiting the University of Oxford.

---

Published in Transactions on Machine Learning Research (07 2203))
1 Introduction
Deep neural networks are sensitive to imperceptible worst-case perturbations, also known as adversarial per- turbations (Szegedy et al. 2014). As a result, training neural networks that are robust to such perturbations has been an active area, of study in recent years (see Ortiz-Jiménez et al. (2021) for a review). The most prominent line of research, referred to as adversarial training (AT), focuses on online data augmentation with adversarial samples during training. However, finding these samples for deep neural networks is an NP-hard problem (Weng al., 2018). In practice, this is usually overcome with various methods, referred to adversarial attacks, which find approximate solutions to this hard problem. The most popular attacks are based projected gradient descent (PGD) (Madry et al. 2018), which is still a computationally expensive algorithm that requires multiple steps of gradient ascent. This hinders its use in many large-scale applications motivating the use of alternative efficient single-step attacks (Goodfellow et al. 2015
The use of single-step attacks within AT, however, raises concerns regarding stability. Although single-step AT leads to an initial increase in robustness, the networks often reach a breaking point beyond which they lose all gained robustness in just few additional training iterations (Wong et al. 2020). This phenomenon is known catastrophic overfitting (CO) (Wong al. 2020; Andriushchenko & Flammarion 2020). Given the clear computational advaniage of single-step attacks during AT, significant. body of work has been dedicated to finding ways to circumvent CO via regularization and data augmentation.
However, despite recent methodological advances, the mechanisms that lead to CO remain poorly understood. Nevertheless, due to the complexity of this problem, we argue that identifying such mechanisms cannot. be done through observations alone and requires active interventions [lyas al. (2019). That is, we want to synthetically induce CO in settings where it would not naturally happen in order to explain its causes.
The key contribution of this work is exploring the root causes of CO, leading to the core message:
Catastrophic overfitting is a learning shortcut used by the network to avoid learning complex robust features while achieving high accuracy using easy non-robust ones.
In more detail, our main contributions arc:
(i) We demonstrate that CO can be induced by injecting features that are strong for standard classification but insufficient for robust classification.
(ii) Through extensive empirical analysis, find that CO is connected to the network's preference for certain fealures in datasel.
(iii) We describe and analyse a mochanistic explanation of CO and its different prevention methods.
Our findings improve our understanding of CO by focusing on how data influences AT. They provide insights into the dynamics of AT, where the interaction between robust and non-robust features plays a key role, and open the door to promising future work that manipulates the data to avoid CO.
Outline The rest of the paper is structured as follows: In Section we review related work on CO. Section 3 presents our main observation: CO can be induced by manipulating the data distribution by injecting simple features. In Section 4 we perform an in-depth analysis of this phenomenon by studying the features used by networks trained with different procedures. In Section 5 we link these observations
to the evolution of the curvature of the networks and find that CO is a learning shortcut. Section 6 pieces
all these evidence together and provides a mochanistic explanation of CO. Finally, in Section 7 we use our new perspective to provide new insights on different ways to prevent CO.
2

---

Published in Transactions on Machine Learning Research (07/ (203))
2 Preliminaries and related work
Let fo Rd + + denote neural network paramcterized by a sct of weights O E R which maps input samples # E Rd to y E (l,...,cc.. The objective of adversarial training (AT) is to find the network parameters 0 E Rn that minimize the loss for the worst-case perturbations, that is:
min B 0
max C(fo(æ + +0)) |1
(1)
where D is a data distribution, d E Rd represents an adversarial perturbation, and (p,c) characterize the adversary. This is typically solved by alternately minimizing the outer objective and maximizing the inner one via first-order optimization procedurcs. The outer minimization is tackled via a standard optimizer, c.g., SGD, while the inner maximization problem is approximated with adversarial attacks with one or more steps of gradient ascent. Single-step AT methods are built. on top of FGSM (Goodfellow et al. 2015). In particular, FGSM solves the linearized version of the inner maximization objective (when p = 00), i.e.
SFGSM argmax ((fo(z)) v) 8 VzL(fo(x),)) esign (V ((f(((())))))
FGSM is computationally efficient as it requires only a single forward-backwar step. However, FGSM adversarial training (FGSM-AT) often yields networks that are vulnerable to multi-slep atlacks such as PGD (Madry et al. 2018). In particular, Wong et al. (2020) observed that FGSM-AT presents a failure mode where the robustness of the model increases during the initial training epochs, but then loses all robustness within a few iterations. This known catastrophic overfitting (CO). They further observed that augmenting Aaaeever,,Adruushhhhheee AAndrishchennoo & Flammarion (2020) showed that this method still leads to CO at larger Therefore, they proposed combining FGSM-AT with smoothness regularizer (GradAlign) that encourages the cross-cntropy loss to be locally linear. Although GradAlign successfully avoids CO, the smoot hness regularizer adds a significant computational overhead.
the FGSM attack with random noise seemed to mitigate CO. However,
Even more expensive, multi-step attacks approximate the inner maximization in Equation (1) with several gradient ascent steps (Kurakin et al. 2017 Madry et al.) 20188 Zhang et al.)
2019). af sufficient number of steps are used, these methods do not suffer from CO and achieve better robust ness. Due to their superior performance
and extensively validated robustness (Madry et al.] 2018 Tramer et al. 2018 Zhang et al. 2019: Rice et al. 2020), multi-step methods such as PGD are considered the reference in AT. However, using multi-step attacks in AT linearly increases the cost of training with the number of steps. Other methods have been proposed to avoid CO while reducing the cost of AT. However, these methods either only move CO to larger c rradii (Golgooni) et al. 2021 are more computationally expensive (Shafahi et al. 2019 Li et al. 2020 or achieve sub-optimal robustriss (Kang & (bosavi-Dezfoolij 2021 Kim et al.] 2021
Recently, /de Jorge el. al. (2022) proposed N-FGSM, which avoids CO without extra cost, however, the phenomena of CO remains poorly understood.
Other works have also analyzed the training dynamics when CO occurs. Initially, it suggested that CO was a result of the networks overfitting to attacks limited to the corners of the Exo ball (Wong et al. 2020). However, this conjecture was later dismissed by Andriushchennoo & Flammarion (2020) who showed PGD-AT projected to the corners of the Ex ball docs not suffer from CO. Alternatively, they suggested that the reason Wong et al. (2200) avoids CO is the reduced radií of their perturbations on expectation, but de Jorge et al. (2022) showed that they could still prevent CO while using larger perturbations. Mcanwhile, it has been consistently reported (Andriushchenkoo & Flammarion 2020 Kim et al., (021) that networks suffering from CO exhibit highly lonnlinnar loss landscape. As FGSM relies the local linearity of the loss landscape, this sudden increase in non-linearity of the loss renders FGSM ineffective (Kim et al. 2021 2021). However, none of these works have managed to identify what causes the
Kang & network to become strongly non-linear. In this work, we address this knowledge gap, and explore a plausible mechanism that can cause single-step AT to develop CO.
3

---

Published in Transactions on Machine Learning Research (07/2023)
Data intervention
Clean accuracy
Robust accuracy
100%
100%
+
95%
80%
90%
60%
v (1)
85%
Injected features
20%
80%
0%
5
10
15
20
25
5
10
15
20
25
[1(255]
(1/255)
E 2/255
6
4/255
C 6,255
C 8/255
Figure 1: Left: Modified dataset with injected simple and discriminative features. Right: Clean and robust performance after FGSM-AT on injected datasets D3. We vary the strength of the synthetic features B (3 0 corresponds to the original CIFAR-10) and the robustness budget e. For e E {1/255, 9/2ss), our intervention induce CO when the synthetic features have strength 6, even when the original data does not suffer CO. Results averaged over 3 seeds. Shades show min-max values.
3 Inducing catastrophic overfitting
Our starting point is the observation that robust solutions are not the default outcome of standard training and are avoided unless explicitly enforced through AT. That is, robust solutions are hard to learn. We also know that robust classification requires leveraging alternative robust features that are not learned in standard training (lyys et al., 2019; Sanyal al.] (202). Finally, know that when CO occurs, the robust accuracy drops while the clean and FGSM accuracy increases (Wong et al. 2020 Andriushchenko Flammarion 2020 With that in mind, we pose the question: Can CO be mechanism to avoid learning complex robust features? If so, the network may be using CO as a way to favor casy non-robust features over complex robust ones.
Directly testing this hypothesis is difficult as it requires identifying and characterizing robust and non-robust features in a real dataset. This challenging task, equivalent to actually solving the problem of adversarial robustness. To overcome this, we follow the standard practice in the field and use controlled modifications of the data (Arpi al. 2017 Myas et al.) 2019; Shah et. al., 2020; Ortix-Jimenez al. 2020a). By conducting experiments the manipulated data, can make claims about CO and its causes. Specifically, that by injecting a simple discriminative feature on standard vision datasets, we can induce CO at lower values of e (i.e., with smaller perturbations) than those at which it naturally occurs. This suggests that the structure of the data plays a significant role the onset of CO.
find
Our injected dataset Let (z,y) ~ D be an image-label pair sampled from distribution D. We modify the data by adding a label-dependent feature vly), scaled by a parameter B >> O, and construct a family of injected datasets Dj:
(G. v) D0)-Da3
Dz:
H == = (y)) with (x, v) D.
2
We design vyy) to be linearly separable with ||w(~)] for all y E y. We mainly use = oo in the paper as CO has been observed more often in this setting (Wong et al. 2020 Andriushchenko Flammarion 2020), but we also present some results with p = 2 in Appendix D The set all injected features is denotod as V (o(v) y E 2). The scale parameter 3 > 0 is fixed for all classes and controls the relative strength of the original and injected features, i.e., and v(y), respectively (see Figure I (left)).
Injecting features that are lincarly separable and perfectly correlated with the labels induces some interesting properties. In short, although v(y) is easy-to-learn, the discriminative power and robusiness of a classifier that solely relies on the injected features V depends on the scale parameter 3. Indeed, a linear classifier relying only on V can separate Dy for large enough B. However, if x has some components in span(V), the interaction between æ and vg)) may decrease the robustness of the classifier for small B. We rigorously illustrate this behavior in Appendix A
4

---

Published in Transactions on Machine Learning Research (07/2023)
To control the interaction between and v(y), we design V by selecting the first vectors from the low-frequency components of the 2D Discrete Cosine Transform (DCT) (Ahmed et al.) 1974). depicted in Figure ! (left), which have large alignment with the space of natural images in our experiments (e.g., CIFAR-10). To ensure the norm constraint, we binarize these vectors to have only #1 values, resulting in a maximal per-pixel perturbation that satislies [[ = 1. These design constraints also make it easy to visually identify the alignment of adversarial perturbations with e(y) as they have distinctive patterns.
Injection strength (3) drives CO We train Nes1 (He et al. 2016 on different intervened versions of CIFAR-10 10 (Krizhevsky & Hinton 2009) using FGSM-AT for different robustness budgets e and scales B. Figure 1 (right) shows the results of these experiments in terms of clean accuracy and robustness D In terms of clean accuracy, Figure [
(right) shows two regimes. First,, when / < e, the network achieves roughly the same accuracy by training and testing on D3 as by training and testing on D (corresponding
to 8 = 0), i.e., the network ignores the added features v(v). Mcanwhile, when 3 > e, the clean accuracy reaches 100% indicating that the network relies on the injected features and ignores D. This is further shown empirically in Section and Appendix F
The behavior with respect to robust accuracy is even more interesting. For small e (e 2/255), the robust accuracy shows the same trend as the clean accuracy, albeit with lower values. For large (e= 8/25s), the model incurs CO for most values of B. This is not surprising as CO has alrcady been reported for this value of e on the original CIFAR-10 datasel (Wong et al.) 2020). However, the interesting setting is for intermediate values of (e E {4/25,, 5/25s}). For these settings, Figure | (right) distinguishes three regimes:
(i) When the injected features are weak (B < <), the robust accuracy is similar with the original data.
(ii) When they are strong (3 >> c), robustness is high as using v(y) is enough to classify ã robustly.
(iii) When they are mildly robust (3 R <), training suffers from CO and the robust accuracy drops to zero.
This last regime is very significant, as training on the original dataset D (B = 0) does not suffer from CO for this value of e. In the following section, we delve deeper into these results to better understand the canse of CO.
Inducing CO in other scttings To ensure generality, we replicate our previous experiment using different settings. Specifically, we show in Appendix D that our intervention can induce CO in other datasets such CIFAR-100 and SVHN (Netzer et al. 2011 and higher-resolution ones such as Tiny NetgeNet (Li et al. 2020) and ImageNet-100 100 (Kireev et al. 2022). We also reproduce the effect using larger networks like a WideRes-
Net28x10 (Zagoruyko & Komodakis 2016) and for & perturbations. We, therefore, confidently conclude that there is a link between the structure of the data and CO, independently of actual datasets, and models.
4 Networks prefer injected features
Since we now have a method to intervene in the data using Equation 2 and induce CO, we can use it to better characterize the mechanisms that lead to CO. The previous experiments showed that when 3 E E or B > e, our data intervention does not induce CO. However, when / =z c, FGSM-AT consistently experiences CO. This raises the following question: what makes 8 x e special? We now show that for B e, network trained using AT uses information from both the original dataset D and the injected features in V to achieve high robust accuracy on the injected dataset Dz. However, when trained without any adversarial constraints i.e., for standard training, the network only uses the features in V and achieves close to perfect clean accuracy.
In order to understand what features are learned when training on the injected dataset D3, we evaluate the performance of different models on three distinct test sets: (i) CIFAR-10 test set with injected features (Dp). (ii) original CIFAR-10 test set (D), and (iii) CIFAR-10 test set with shuffled injected features (Dx(s)) where the additive signals are correlated with a permuted set of labels, i.e.,
(T()), y) (mp
-(>)= = + (T))),
(3)
RRbustness measured using PGD with 50 steps and 10 restarts.
5

---

Published in Transactions on Machine Learning Research (07/2023)
PGD-10 10 trainnng
Standard training
FGSM training
100%
100%
100%
Clean arstriyy
50%
50%
50%
0%
086
0%
10
20
10
20
10
20
30
60%
009
60%
CO
Robast acruary
40%
40%
40%
20%
20%
20%
0%
0%
10
Eparh
Epoch
Epoch
Injected (D,)
CIEARIO (D)
Shuffled (Datn)
Figure 2: Clean (top) and robust (bottom) accuracy on 3 different test sets: (i) original CIFAR-10 (D), (ii) dataset with injected features D3 and (iii) dataset with shuffled injected features D Da(0)- All runs B = 8/255 and 6/253. Blue shading denotes the use of both D and V and yellow only V.
with yz D and E V. Here, : - y is fixed permutation to shuffle the labels. Evaluating these models on D=(3) exposes them contradictory information, since æ and v(z(y)) are correlated with different l Thus, if the classifier only relies on V, the performance should be high on D3 and low D and Dx(0)) while if it only relies on D, the performance should remain constant for all injected datasets. Figure 2 shows the results of this evaluation during standard, FGSM-AT and PGD-AT on Dj with with 3 = 8/255 and E = 6/285.
PGD training: We can conclude that the PGD-AT network achieves robust solution using both and V from Figure (left). It uses D as it achieves better than trivial accuracy on D and Dz(9)- But it also uses V as it achieves higher accuracy on D3 than D, and suffers from drop in performance on Da(s)- This implies that this network elfectively combines information from both the original and injected features for classification.
Standard training: Standard training shows a different behaviour than PGD-AT (sce Figure (center)). In this case, the network achieves excellent clean accuracy on D3, but trivial accuracy on DD indicating that it ignores the information present in D and only uses non-robust features from for classification. This is further supported by its near-zero accuracy on Dz(s)- We conclude that the injected features are easy learn i.e., they are preferred by standard training.
FGSM training: The behaviour of FGSM-AT in Figure 2 (right) highlights further the preference for the injected features. FGSM-AT undergoes CO around epoch 8 and its robust accuracy on Da drops zero despite its high clean accuracy on Dr. It presents two distinct phases:
(i) Prior to CO the robust accuracy on D3 is non-zero and the network exploits both D and V, similar to PGD-AT.
(ii) After CO both the clean and robust accuracy on D and Dr(s) drop, exhibiting behavior similar to standard training. This indicates that, after CO, the network forgets D and solely relies on the features in V.
Why does FGSM change the learned features after CO? From the behaviour of standard training, we concluded that. the network has preference for the injected features V, which are easy to learn.
We also evaluate on alone Appendix F

---

Published in Transactions on Machine Learning Research (07 2022)
Curvature (Amasx)
CIFAR-10 feature accuracy
C0
10
Plasse
10
10-1
Phose
50%
10-2
10
15
20
25
30
0
5
10
15
20
25
30
Epoch
Epoch
FGSM-AT (Da)
PCD-AT (Da)
PCSM-AT (D))
PGD-AT (Dt)
Figure 3: Evolution of metrics for FGSM-AT and PGD-AT on dataset with injected features (Dz) and with orthogonally projected features (Dt), where there is no interaction between features. AT is performed with B = 8/255 and = 6/255.
Meanwhile, the behaviour of PGD training suggests that when the easy features are not sufficient for robust classification, the model combines them with other (harder-to-learn) features, such those in D. FGSM initially learns robust solution using both D and V, similar to PGD. However, if the FGSM allacks are rendered ineffective, the robustness constraints are removed, allowing the network to revert back to the simple features and the performance on the original dataset D drops. This is exactly what occurs with the onset of CO around epoch 8, which we will further discuss in Section 5.1 As we will see, the key to understanding why CO happens lies in how learning each type of feature influences the local geometry of the classifier.
5 Geometry of CO
Recent works have shown that after the onset of CO, the local geometry around the input becomes highly non-linear (Andriushchenko & Flammarion 2020 Kim et al. 2021 de Jorge et al. 2022). Motivated by these, and to better understand induced CO, we investigate the evolution of the curvature of the loss landscape when this happens.
5.1 Curvature explosion drives CO
To measure curvature, we use the average maximum eigenvalue of the Hessian on N 100 training points Ámax = N Σw Amax training. Figure 3 (left) shows the result of this experiment for FGSM-AT (orange line) and PGD-AT (green line) training on D3 where CO with FGSM-AT happens around epoch 8.
# (VLC(J( vn) in Moosavi-Dexfooli et al. (2019)), and record it throughout
Two-phase curvature increase Interestingly, we observe a steep increase in curvature for both FGSM-AT and PGD-AT (the y-axis is in logarithmic scale) before CO. However, while there is a large increase curvature for PGD-AT right before the 8th epoch, stabilizes soon after PGD-AT acts as a regularizer on the curvature (Moosavi-Dezfooli et al.] 2019 Qin et al. 2019) which explains how it controls the curvature explosion. However, curvature second-order property of the loss and unlike PGD, FGSM is based on coarse linear (first order) approximation of the loss. In this regard,
that FGSM-AT cannot contain the curvature increase, which eventually explodes around the 8th epoch and saturates at very large value. The final curvature of the FGSM-AT model is 100 times that of the PGD-AT model.
High curvature leads to meaningless perturbations The fact that the curvature increases rapidly alongside the FGSM accuracy during CO agrecs with the findings of Andrinshchenko Flammarion (020 The loss becomes highly non-linear and thus reduces the success rate of FGSM, which assumnes a linear loss.

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Curvature
Training attack accuracy
100%
80%
109
60%
10-1
40%
10-2
20%
20
25
15
20
30
Epoch
Epoch
FGSM Imaning
PGD-10 tmining
Figure 5: Evolution of curvature and training attack accuracy of FGSM-AT and PGD-AT trained on the
original CIFAR-10 with c = 8/255. When CO happens the curvature explodes.
To show that CO indced occurs due the increased curvature breaking FGSM, visualise the adversar- ial perturbations before and after CO. As observed in Figure 4 before CO, the adversarial perturbations are non-triven coniirivially aligned with sVan(V). albeit with some corruptions originating from æ. Nonetheless, after CO, the new adversarial perturbations point towards meaningless directions; they do not align with even though the network is heavily reliant this information for classifying the data (cf. Sec- tion 4. This reinforces the idea that the increase in curvature causes a breaking point after which FGSM is no longer effective. This behavior of adversarial perturbations after CO is different from the one on standard and PGD-AT networks which the pertur- bations align with discriminative directions (Fazi
Intervened
samgle (x. ~ Dg
ÁPOa after CO
Figure 4: Samples of the injected dataset D3 (top) and FGSM perturbations before (middle) and after CO (bottom). Before CO, perturbations focus on synthetic features, but after CO they become noisy.
et al.] 2018 Jetley et al., 2018 Ilyas al. 2019 Ortiz-Jimener et al. 2020b
Two-phase curvature increase also occurs in naturally ocurring CO In Figure 5 show the curvature when training on the original CIFAR-10 datasct with = 8/255 (where CO happens for FGSM-AT). Similarly to our observations on the injected datasets, the curvature during FGSM-AT explodes along with the training accuracy while for PGD-AT the curvature increases at a very similar rate than FGSM-AT during the first epochs and later stabilizes. This indicates that our described mechanisms may as well apply induce CO on natural image datasets.
5.2 Curvature increase is a result of feature interaction
But why does the network increase the curvature in the first place? In Section we observed that this is a shared behavior of PGD-AT and FGSM-AT, at least during the initial stage before CO. Therefore, it should not be a mere "bug". We conjecture that the curvature increase a result of the interaction between features the dataset which forces the network to increase its non-lincarity in order to combine them effectively for robustness.
Curvature does not increase without interaction To demonstrate this, we perform a new experiment. in which we modify D again (as Scction 3). However, this time, we ensure that there no interaction
We also quantify this alignment in Figure
8

---

Published in Transactions on Machine Learning Research (07/2023)
between the synthetic features vty) and the features from D creating DJ such that:
@ .)) ~ D3 x = Pvi (a)) + Bo(l))
(4)
with (x, :) ~ D and v(y) E V, where Pyt denotes the projection operator onto the orthogonal complement of V. Since the synthetic features vly) are orthogonal to D, a simple linear classifier relying only on v(y) can robustly separate the data up to radius that depends solely on B (see the theoretical construction in Appendix A
Interestingly, we find that, in this dataset, none of the (3,e) configurations used in ?? induce CO. Here, we observe only two regimes: one that ignores V (when 8 < e) and one that ignores D (when 3 > e). This supports our conjecture that the interaction between the features of and v(y) causes CO in Dz. Morcover, Figure 3 (left) shows that, when performing either FGSM-AT (light blue) or PGD-AT (dark blue) on D' the curvature consistently low. This agrees with the fact that in this case there is no need for the network to combine the injected and original features for robustness and hence the network does not need to increase its non-linearity to separate the data.
Non-linear feature extraction Finally, we study the relationship between the features extracted by the network and its curvature. We train multiple logistic classifiers to classify D based on the feature representations (output of the penultimate layer) of networks trained on D.y as in Shi et al. (2022). The feature accuracy of these classifiers will depend on how well the network has learned to extract information from D.
Figure 3 (right) shows that for the PGD-AT network (green), the feature accuracy on D progressively increases over time, indicaling that the network has learned lo extract meaningful information from D even though it was trained on Dg. The feature accuracy also closely matches the curvature trajectory in Figure 3 (left). In contrast, the FGSM-AT network (red) exhibits two phascs on its feature accuracy: Initially, it grows at a similar rate as the PGD-AT network, but after CO occurs, il starts to decrease. This decrease feature accuracy does not correspond with decrease in curvature, which we attribute to the network taking a shortcut and ignoring D in favor of easy, non-robust features.
When we use features from networks trained D we find that the accuracy D is consistently low, suggesting that the network is increasing its curvature to improve its feature representation as it must combine information from V with D to become robust in this case.
6 A mechanistic explanation of CO
Based on the previous insights, can finally summarize the chain of events that leads to CO in our injected datasets:
(i) To learn a robust solution, the network combine easy non-robust features with complex robust ones. However, without robustness constraints, the network favors learning only the non-robust features (see Section
(ii) When learning both kinds features simultaneously, the network increases its non-linearity to improve its feature extraction ability (sce Section 5.2
(iii) This increase in non-lincarity provides a shorteut to break FGSM, triggering CO and allowing the network to avoid learning the complex robust features while still achieving a high accuracy using only the easy non-robust. ones (see Section 5.1
How general is this mechanism? So far, all our experiments have dealt with datasets that have been synthetically intervened by injecting artificial signals. However, we advocate that our work provides sufficient and compelling evidence that features are the most likely cause of CO in general. Deep learning is an empirical discipline, where many insights are gained from experimental studies, and as shown by prior work (Arpit et al,) 2017 Myas et al. 2019 Shah et al. 2020; Ortiz-Jimenez et al. (220aa), manipulating data in controlled ways is a powerful tool to infer the structure of its features. In this light, we are confident our work provides robust and comprehensive evidence that features are likely the primary catalyst of CO across different. datasets.
9

---

Published in Transactions on Machine Learning Research (07/ (002))
Aligmment of ò and injected leatures
Evolution of perturbation spectrum
0.08
30-
CO
FGSM attacks
(x)) vfg]]
0.06
Random baseline
Magnitudo (BB)
DCT fren.
20-
0.04
0.02
0.00
15
20
25
30
10
15
200
30
Epoch
Epoch
Figure 6: Quantitative analysis of the directionality of FGSM perturbations before and after CO in CIFAR-10 with B = 8/255 and 6/255. Left: Aligument of FGSM perturbations with injected features during training. Right: Magnitude of the DCT spectrum of FGSM perturbations during training. The plot shows values for the diagonal components at every epoch.
Our study includes extensive ablations with various types of injected features, such as random directions, and different norms and we have shown these settings also lead to CO (see Appendix [D. Furthermore, we have observed that the increase in curvature within our manipulated datasets mirrors the patterns found in naturally occurring CO instances, as demonstrated in Figure 5
Beyond our empirical investigations, we also provide theoretical support for our conjecture in Appendix B The concept that robust classifier may need combine different features can be formally proven in certain scenarios. Specifically,
Appendix B we offer a mathematical proof showing that numerous learning scenarios exist in which learning a robust classifier requires leveraging additional non-linear features on top of the simple ones used for the clean solution.
Finally, in what follows, we will present further evidence that CO is generally induced by features. Specifically, we will show that other feat manipulations besides the injection of features can also influence CO. Additionally, we will show that the primary strategies effective at preventing CO in natural datasets can also mitigate CO in our manipulated datasets. This suggests a commonality between these types of CO, further underscoring the widespread relevance of our mechanism
7 Analysis of CO prevention methods
Our proposed dataset intervention, defined in Sec- tion 3. has allowed us to better understand the chain of events that lead to CO. In this section, we will focus on methods to prevent CO and analyze them in our framework for further insights.
Table 1: Clean and robust accuracies of FGSM-AT and PGD-AT trained networks CIFAR-10 and the low pass version described in Ortiz-Jimenee Jimenez et al. 2020b at different 6.
Original
Low pass
High-freyuency features and CO Recently, Grabinski et al. (2022) suggested that CO could be caused by aliasing effects from downsampling layers and proposed using low-pass filter before pooling. We take this theory step further and show that just removing the high-frequency components from 8/255
Method (c)
Clean Robust
Clean
Robust
FGSM ($/255)
85.6
0.0
81.1
47.0
PGD ($/255)
80.9
50.6
80.3
49.7
FGSM (16/25))
76.3
0.0
78.6
0.0
CIFAR-10 consistently prevents CO at (where FGSM-AT fails). Furthermore, in Figure (right) we observe that after CO on D3, FGSM attacks ignore low-frequencies and become increasingly high-frequency.
PGD (16/255)
68.0
29.2
66.9
28.4
Surprisingly, though, applying the same low-pass technique at c 16/255 does not work and neither does Grabin ski et al. (2022) (see Appendix]]] Jy. We conjecture this is because the features robust at e 5/255 in the low pass version of CIFAR-10 might not be robust. at 16/255 and although high-frequency features might contribute to
10

---

Published in Transactions on Machine Learning Research (07 2222)
Robust accuracy
Curvature (Amax)
101
80%
10
00%
40%
10-1_
20%
10-2
0%
10
14
0
15
30
(/255)
Epoch
PGD-10
GradAlign (A 0.112)
CrudAlign (A - 2]
N-rGSM (k
10c)
N-FCSM (k 2c)
Figure 7: Left: Clean and robust accuracy of GradAlign, N-FGSM and PGD-AT. Right: Curvature evolution
of all methods. Training performed on D3 with 6/255 and B = 8/235 and averaged over 3 random seeds.
CO, the overall set of CO inducing features is more complex. Additionally, we observe that CO can be induced even when the injected features are randomly sampled (see Figure I4 Although removing low-frequency features does not work settings, we see this as proof of concept that manipulating the data itself may prevent CO some settings. Generalizing this idea to other datasets promising avenue for future work.
Preventing CO in injected dataset In Figure 2 (left), have shown that using PGD-AT prevents CO our injected dataset, but results indicate that these methods can also be successful in preventing CO, but need stronger regularization for certain values of 3. As 3 grows, the injected feature v(v) becomes more discriminative, creating a stronger incentive for the network to use it. This increases the chances CO as the network increases its curvature to combine the injected features with others order to become robust. Figure 7 shows that the curvature of N-FGSM and Align decreases with stronger regularizations, similar PGD-AT. This supports the idea that preventing the curvature from exploding can indeed prevent CO.
also evaluate the effectiveness of GradAlign and N-FGSM. Our
Stronger CO high-resolution datasets As discussed in Section 3 we observe CO when injecting discriminative features in several datasets. Interest- ingly, for ImageNet-100, we have observed that even multi-step attacks like PGD are not able to find adversarial perturbations after CO (see Figure S However, when using AutoAttack (AA) (Croce &
Accuracy
100
80
60
40
Hein 2020), find that models not robust" To the best of our knowledge, this had not been observed before with "vanilla' CO, where after CO, models would be vulnerable to very weak PGD attacks, like PGD with 10 steps (Wong et al.) 2020 We argue that in this case, the curvature increase that breaks FGSM also hinders PGD optimization. This aligus with our findings with N-FGSM and GradAlign, where stronger regularization is needed to prevent CO with the injected datasets, and suggests that CO is stronger in Dz.
20
8
10
12
14
(/255)
Clean
FGSM
PGD
AA
Figure 8: Clean, FGSM, PGD and AutoAttack accu- racy on injected ImageNet- 100 with varying B. FGSM- AT performed wtih = 4/255.
40n other datasets, AA and PGD work equally.
11

---

Published in Transactions on Machine Learning Research (07 (02)))
8 Concluding remarks
In this work, we have presented thorough empirical study to establish a link between the features of the data and the onset of CO in FGSM-AT. Using controlled data interventions, we have shown that CO is a learning shortcut used by the network to avoid learning complex robust features while achieving high accuracy using easy non-robust ones. This new perspective sheds new light on the mechanisms that trigger CO2 as shifts our focus towards studying the way the data structure influences the learning algorithm. We believe this opens the door to promising future work towards understanding the intricacies of these mechanisms,, deriving methods for inspecting data and identifying feature interactions, and finding faster ways to make neural networks robust.
Broader impact
The broader social implications of this research extend beyond its academic scope, touching upon the spheres of the entire field of adversarial machine learning. At its core, our work aims to enhance the robustness and reliability of AT systems, thereby addressing critical challenges in various sectors such as healthcare and autonomous driving. In these domains, mitigating the vulnerability to adversarial examples can lead to more accurate and reliable diagnoses and treatments, as well as safer navigational decisions during unforeseen events like storms.
Moreover, having a better understanding of CO has the potential to democratize access to robust machine learning models. By enabling faster adversarial training fraction of the cost, could make advanced AI tools more accessible to smaller organizations and independent researchers. This democratization could spur innovation across various sectors, broadening the positive impacts of AI.
However, while these benefits are significant, it is crucial to acknowledge the potential drawbacks. An increase in the robustness Al systems, while generally beneficial, can inadvertently strengthen systems employed for detrimental purposes in exploitative circumstances, as illustrated by Albort al.) (2020) 2021). These paradoxical situations, such those occurring under warfare conditions within authoritarian regimes, underscore the fact that enhancing Al system robustness sometimes lead to negative societal impacts.
Our work in this paper is purely academic, focusing on the technical understanding and prevention of CO.
While we aim to improve Al systems' robustness and democratize access to them, it is the responsibility of policymakers, technologists, and society at large to ensure these advancements are employed ethically and for the greater good.
Acknowledgements
We thank Maksym Andriushchenko, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli and Ricardo Volpi for the fruitful discussions and feedback. This work supported by the UKRI grant: Turing AT Fellowship EP SEP/W002981/ and EPSRC/MURI grant: EP/N019474/I. We would also like thank the Royal Academy of Engineering and FiveAL. Guillermo Ortiz-Jimenez acknowledges travel support from ELISE (GA no 951847) in the context of the ELLIS PhD Program. Amartya Sanyal acknowledges partial support from the ETH AT Center.
12

---

Published in Transactions on Machine Learning Research (07 2022)
References
N. Ahmod, T. Natarajan, and K. R. Rao. Discrete Cosine Transform. IEEE Transactions Computers, 1974.
Kendra Albert, Jonathon Penney, Bruce Schneier, and Ram Shankar Siva Kumar. Politics of adversarial machine learning. 006424002.00448, 2020.
Kendra Albert, Maggie K. Delano, Bogdan Kulynych, and Ram Shankar Siva Kumar. Adversarial for good? how the adversarial ML community's values impede socially beneficial uses of attacks, arXic:2107.10302, 2021.
Maksym Andrinshchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In Newral Information Processing Systems (NewrIPS), 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emnanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look memorization in deep networks. In International Conference on Machine Learning (ICML), 2017.
James Aspues, Richard Beigel, Merrick Furst, and Steven Rudich. The expressive power of voling polynomials. Combinatorice, 1994.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206 2216. PMLR, 2020.
Pau de Jorge, Adel Bibi, Riccardo Volpi, Amartya Sanyal, Philip H. S. Torr, Grégory Rogez, and Puneet K. Dokania, Make some noise: Reliable and efficient single-step adversarial training. 011188 2022.
Alhussein Fawzi, Mohsen Moosavi-Dezfooli, Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zeinab Golgooni, Mchrdad Saberi, Masih Eskandar, and Mohammad Hossein Rohban. Zerograd: Mitigating and explaining catastrophic overfitting in FGSM adversarial training. arXiv:2103.154 2021.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR), 2015.
Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. FrequencyLowwCtt Pooling Plug & Play against Catastrophic Overfitting. European Conference on Computer Vision (ECCV), 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016.
Andrew Nlyss, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Neural Information Processing Systems (NeurIPS), 2019.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversarics? Neural Information Processing Systems (NeurIPS), 2018
Peilin Kang and Geyed-MMhsen Dezoovi- Understanding catastrophic overfitting in adversarial training. arXim:210.. 22944 2021.
Hoki Kim, Woojin Lee, and Jaewook Loc. Understanding catastrophic overfitting in single-step adversarial training, In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Klim Kireev, Maksym Andriushchenko, and Nicolas Flammarion. On the effectiveness of adversarial training against common corruptions. Tn Conference on Uncertainly in Artificial Intelligence (UAT), 2022.
13

---

Published in Transactions on Machine Learning Research (07/2023)
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Compuler Science, University of Toronto, 2009.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2017.
Bai Li, Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial training. arXiv:2006.0900 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Seyed-Mohsen Moosari-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Newral Information Processing Systems (NewrIPS), Workshops, 2011.
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi, and Pascal Frossard. Neural anisotropy directions. Neural Information Processing Systems (NeurIPS), 2020a.
Guillemmo Ortiz-Jimencz, Apostolos Modas, Scyed-Mohsen Moosavi, and Pascal Frossard. Hold me tight! influence of discriminative features on deep network boundaries. Neural Information Processing Systems (NewrIPS), 2020b.
Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsen Dezeooi,, and Pascal Frossard. Optimism in the face of adversity: Understanding and improving deep learning through adversarial robustness. Proceedings of the IEEE, 2021.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhusscin Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. Tn Neural Information Processing Systems (NeurIPS), 2019.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning (ICML), 2020.
Amartya Sanyal, Puncet K. Dokania, Varun Kanade, and Philip Torr. How benign is benign overfitting' In International Conference on Learning Representalions (ICLR), 2021.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Neural Information Processing Systems (NewrIPS), 2019.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Neural Information Processing Systems (NeurIPS), 2020.
Yuge Shi, Imant Daunhawer, Julia E Vogt, Philip HS Torr, and Amartya Sanyal. How robust are pre-trainod models to distribution shift? arXio preprinl ur arXi::2206.0885 2022.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Florian Tramèr, Alexcy Kurakin, Nicolas Papernot, Tan Goodfellow, Dan Bonch, and Patrick McDaniel. En- semble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsich, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference
on Machine Learning (ICML), 2018.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. Tn International Conference on Learning Representations (ICLR), 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In brvee, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), 2019.
Richard Zhang. Making convolutionall neural networks shift-invariant again. In International Conference Machine Learning (ICML), 2019.
15

---

Published in Transactions on Machine Learning Research (07/2023)
A Analysis of the separability of the injected datasets
With the aim to illustrate how the interaction between D and V can influence the robustness of a classifier trained on D we now provide a loy theoretical example in which we discuss this interaction. Specifically, without loss of generality, consider the binary classification setting on the dataset (ær, v) ~ 'D where E 1, for case. Let's now consider the injected datasct D3 and further assume that v(+1)= = u and v(-1) = -u with E Id and ||u||2 1, such that I x + Syu. Moreover, let E [0, 1] denote the interaction coefficient between 'D and V, such that -7 S x u S.
{-1,+1} and |2||
We are interested in characterizing the robust of a classifier that only uses information in V when classifying Dg depending on the strength of the interaction coefficient. In particular, as we are dealing with the binary setting, will characterize the robustness of linear classifier h Rd [-1,+1) that discriminates the data based only on V, i.e., h(m) = sign(u Φ). In our setting, we have
x u Bu' u xxx + if y +1
u u x Bu" JuTu u u if =-
Proposition A.1 (Clean performance). If 3 Y, then h achicves perfect classification accuracy Dg.
Proof. Observe that if y 0, i.c. the features from original dataset D do not interact with the injected features V, the dataset is perfectly linearly separable. However, if the data from D interacts with the injected signal u, ie. non zero projection, then the dataset is still perfectly separable but for a sufficiently
larger B, such that u x + B > 0 when = +1 and u x + B < O when y = -1 Because -7 x u S this is achieved for 3 > 7.
Proposition A.2 (Robustness). 1 > 7, the linear classifier h is perfectly accurate and robust to adversarial perturbations in an balball of radius e S 3 - 7, Or, equivalently, for to be c-robust, the injected features must have a strength B 2 e + 7.
Proof. Given T, we seek lo find the minimum distance to the decision boundary of such a classifier. A minimum distance problem can be cast as solving the following optimization problem:
(
min rerd
|| - subject T Tu 0,
which can be solved in closed form
(x)
|u TT x + yy
The robustness radius of the classifier will therefore be = inf bounded by
REsupp(5g) (*)),, which our case can be
inf
(())) min IuT (
JuT + y/8l + B-7-
(spp)
Based on these propositions, can clearly see that the interaction coefficient y reduces the robustness of the additive features V. In this regard, if E 2 3 - robust classification at a radius e can only be achieved by also leveraging information within D.
16

---

Published in Transactions on Machine Learning Research (07/2023)
B Robust classification can require non-linear features
We now provide rigorous theoretical example of a learning problem that provably requires additional complex information for robust classification, even though it can achieve good clean performance using only simple features.
Given some E N, let RR+1 be the input domain. A concept class, defined over R is a set of functions from Re+1 to {0,1}. 1]. A hypothesis h is s-non-linear if the polynomial with the smallest degree that can represent has a degree (largest order polynomial term) of
Using these concepts we now slate the main result.
Theorem 1. For any p, k E N, <
0.5 such that k < p, there exits a family of distributions Dr over RP+1 and a concept class H defined over 18+1 such that
H. PAC leornable (with respect to the clean error) with linear (degree 1) classifier. However, not robuslly learnable with any linear classifier.
There erists an efficient learning algorithm, that given a dataset sampled it.dd. from a distribution D E DE robustly learns H.
In particular, the algorillom relurns k-non-linear classifier and in addition, the relurned classifier also exploits the linear features used by the linear non-robust classifier.
Proof. We now define the construction of the distributions in DA. Every distribution D in the family of distribution D is uniquely defined by three parameters: a threshold parameter p E {4te: : E {0.... ,}} (one can think of this as the non-robust, casy-to-lcarn fcature), p dimensional bit vector c E {0, 1)2 such that ||l]| l (this is the non-linear but robust feature) and Therefore, given p and e (and which discuss when necessary and ignore from the notation for simplicity), can define the distribution Do.p. We provide illustration of this distribution for p 2 in Figure
Sampling the robust non-linear feature To sample a point (æ. y) RR+1 from the distribution D first, sample a random bit vector & E IR from the uniform distribution over the boolean hypercube (0, 1) Let i 0= E & -eli] (mod 2) be the label of the parity function with respect to evaluated on &. The marginal distribution over V, if sampled this way, equivalent to the Bernoulli distribution with parameter 4. To see why, fix all bits in the input except. one (chosen arbitrarily from the variables of the parity function), which is distributed uniformly over {0, 1}. It is easy to see that this forces the output of the parity function to be dis- tributed uniformly over f0, 1} as well. Repcating this process for all dichotomics of p- variables of the parity function proves the desired result. Intuitively, & constitutes the robust. non-linear feature of this distribution.
Sampling the non-robust linear feature To ensure that & is not perfectly correlated with the true label, we sample the true label y from a Bernoulli distribution with parameter 1. Then we sample the non-robust feature as follows
Unif (X,
y = 0 9 = 0
Unif (X+)
y 1 1A 7
21
Unif (X₂)
y = 0 A
Unif (xxt)
y = 1A A =
where
X = =T=ppppe and
X; = [(p + 2c, 3c)], Xi = -pp- pl and
X₂ = [(p - 3c, p 2c)]
Finally, return (ac. y) where
=xx1; .t) is the concatenation of æ and &.
17

---

Published in Transactions on Machine Learning Research (07/202) 2222)
[1]
30 [1]
[1]
(0, 1)
eCX 200
(0.
(0. 1
(0.,
(1.0)
(0..0)
(1,0)
(0.0)
F-
For [ee(0
S
For
[a(0]
Figure 9: Illustration of one possible distribution De in three dimensions. The data is linearly separable in the direction æ²0] but has a very smalll margin in that direction. Leveraging a[1] and x[2) additionally, we see how the data can indeed be separated robustly, albeit non con-linearly.
Linear non-robust solution First, we show that there is a linear, accurate, but non-robust solution to this problem. To obtain this solution, sample an m-sized dataset Sn = (I) (#.. Mm)) E g++1 x { from the distribution De.P Ignore all, except the first coordinate, of the covariates of the dataset to create 5 {(x1 [0] 11).... (am ymm) where S, [i] indexes the jth coordinate of the 7th element of the dataset. Then, sort 5o on the basis of the covariates (É.e the first coordinate). Lct á be the largest element whose label is
Define finn,p as the linear threshold function on the first coordinate i.e. Jinnp construction, fumm. accurate classifies all points in Sm. The VC dimension of linear threshold function in 1 dimension is 2. Then, using standard VC sample complexity upper bounds [ for consistent learners, if m 2
I (ax 2 Å)- By
1 + _/ log (t)). where Kg some universal constant, we have that with probability at least - B, 1 log
Ko
Err pf DC.P) S.
Non-linear robust solution Next, we propose an algorithm to find a robust solution and show that this solution has a non-linearity of degree k. First, sample the m-sized dataset Sm and use the method described above to find p. Then, create modified dataset 3 by first removing all points æ from Sn such that lee [0] - A 2 and then removing the first coordinate of the remaining points. Thus, each element. in S belongs to IR x 1) dimensional dataset.
Note, that by construction, there is consistent (i.e., an accurate) parity classifier on S. Let the parity bit vector consistent with Š be è. This can be learned using Gaussian elimination. Consequently, construct. the parity classifier fpar. EE 2 [i] C (mod 2).
Finally, the algorithm returns the classifier gp.e, which acts follows:
T H (0) 2 p + 6 8 }
ga.e (x) =
[( < 0
(5)
o.w.
where क = round (æc [1,...,p) is obtained by rounding off x starting from the second index till the last. For example, æ = [0.99, 0.4, 0.9, 0.8], e = 0.2, and c = [0, 0, 1, then = [o, 1, 0, 1] and 90.s, [o] = 1. Finally, it is easy to verify that the classifier ga.e is accurate on all training points and as the number of total parity classifiers less than 2a (hence finite VC dimension), as long as m 2 K1 (a log (2) a where K₁ is some universal constant, we have that with probability least 1 - B,
log (a))
Err (gp.e: DC.P) < a.
Robustness of gae As [0] is distributed uniformly in the intervals [p c, U (p, p +c, have that p A < 4e Err (fim De.P)= < 4ea. Therefore, when is large enough (m poly (1) such that a S z we have that 1 - pl < 5. Intuitively, this guarantees that Ja.e uses the linear threshold function on x [0] for classi- fication in the interval [p.p c]U(p,p - c] and /par, the (p 2c, + 3cJUpp 2e, 3c]. A crucial property
https://wwut http://www......o.a....a..aaaaa.
18

---

Published in Transactions on Machine Learning Research (07/2023)
of Ip.6 is that for all Supp (Dar), the classifier gp.e does not alter its prediction in an ba- of radius We show this by studying four separate cases. First, we prove robustness along all coordinates except the first.
1. When |0=[0] - أ 2 c+ 3, as shown above, Ia.e is invariant to all x [i] for all i >> 0 and is thus robust against all Loo perturbations against those coordinates.
2. When |æ=[0] A < + g. due to Equation (5), we have that ga.. p
(0) where ã round (æ [1. .I.....p)) pl) obtained by rounding off all indices of æ except the first. As the rounding oper- ation on the boolean hypercube is robust to any fx perturbation of radius less than 0.5, we have that I is robust to all &o perturbations of radius less than 0.5 on the support of the distribution De.,
Next, we prove the robustness along the first coordinate. Let 0 < 8 < represent an adversarial perturbation. Without loss of generality, assume that a [0] > á similar arguments apply for the other case.
1. Consider the case az S
g- - 6l S e + & and hence, by construction, I (x) = g (Ix [0] 6: [= [I.lll.....ll. On the other hand, for all 8, we have that
+ 8. Then, lor [0] 8 ---t P s l +
gp.e ([x [0] + 6; [x] [1,... [l.....p]]] pll) = if Ia.e 1.
2. For the case =0 2 X + + g. the distribution is supported only on the interval [p + 2c, p + 3c]. When a positive 5 is added to the first coordinate, the classifier's 's prediction does not change and it remains 1. For all 8 S 5. when the perturbation is subtracted from the first coordinate, its first coordinate still larger than X + + 8 and hence, the prediction still 1.
This completes the proof of robustness of ga.e along all dimensions to &o perturbations of radius less than e, Combining this with its error bound, we have that Adv,,oo (99,6; D=.)) S.
To show that the parity function is non-linear, we use classical result from Aspnes et al. (1994). Theorem 2.2 in Asspes et al. (1944) shows that approxinating the parity function in k bits using polynomial of
degree l innurss at least Zizn (t) where ke- = |--=-| mistakes. Therefore, the lowest degree polynomial that can do the approximation accurately is at least k.
This completes our proof of showing that the robust classifier is of non-lincar degree h while the accurate classifier is linear. Next, we prove that no linear classifier can be robust. We show this by contradiction.
No linear classifier can be robust Construct a set Z of s (to be defined later) points in gp+1 by sampling the first coordinate from the interval [p, + e] and the remaining p coordinates uniformly from the boolean hypercube. Then, augment the set by subtracting e from the first coordinate while retaining the rest of the coordinates. Note that this set can be generated, along with its labels, by sampling enough points from the original distribution and discarding points that do not fall in this interval. Now construct adversarial examples of each point in the augmented set by either adding or subtracting e from the negatively and the positively labelled examples respectively and augment the original set with these adversarial points. For a large enough this augmented set of points can be decomposed into a multisct of points, where all points in any one set has the same value in the first coordinate but nearly half of their label zero and the other half one.
s C
Now, assume that there is a linear classifier that has a low error on the distribution DeP. Therefore the classifier is also accurate on these sets of points the classifier is robust, by assumption, and the union of these sets occupy significant under the distribution DE.P. However, as the first coordinate of every point within a set is constant despite half the points having label one and the other half zero, the coefficient of the linear classifier can be set to zero without altering the behavior of the classifier. Then, effectively the linear classifier is representing a parity function on the rest of the coordinates. However, we have just seen that this is not possible as a linear threshold function cannot represent a parity function on k bits where k 1. This contradicts our initial assumption that there is a robust linear classifier for this problem.
This completes the proof.
oThher a slight technicality as we might not obtain points that are exact reflections of each other around p but that can be overcome by discretising upto a certain precision
19

---

Published in Transactions on Machine Learning Research (07 2203))
C Experimental details
In this scction provide the experimental details for all results presented in the paper. Adversarial training for all methods and datasets follows the fast training schedules with a cyclic learning rate introduced in Wong et al. (020). We train for 30 epochs on CTFAR Krizhevsky & Hinton (2009) and 15 epochs for SVHN Netzer et al. (2011) following Andriushchenko & Flammarion (2020). When we perform PGD-AT we use 10 steps and a step size a = 3/255: FGSM uses a step size of a = c. Regularization parameters for GradAlign Andriushchenko & Flammarion (2020) and N-FGSM de Jorge et al.
(2022) will vary and are stated when relevant in the paper. The architecture employed is a PreactResNNet.S He al. (2016). Robust accuracy
is evaluated by attacking the trained models with PGD-50-10, i.e. PGD with 50 iterations and 10 restarts.
In this case we also employ step size a 2/255 as in Wong et al. 2020 All accuracies are averaged after training and evaluating with 3 random seeds.
The curvature computation is performed following the procedure described in Moosavi-Dezzooli et al. (2019). As they propose, we use finite differences to estimate the directional second derivative of the loss with respect to the input, i.e.,
((fo(())) x + tw) - VaL(fe(x tw).y)
21
with >> 0 and use the Lanczos algorithm to perform a partial eigendecomposition of the Hessian without the need to compute the full matrix. In particular, we pick t = 0.1.
All our experiments were performed using cluster equipped with GPUs of various archilectures. The estimated compute budget required to produce all results in this work is around 2, 000 GPU hours (in terms of NVIDIA V100 GPUs).
D Inducing catastrophic overfitting with other settings
In Section 3 we have shown that CO can be induced with data interventions for CIFAR-10 and fo perturbations. Here we present similar results when using other datasets (i.e. CIFAR-100 and SVHN) and other types of perturbations (i.. & attacks). Morcover, also report results when the injected features v(y) follow random directions (as opposed to low-frequency DCT components). Overall, we find similar findings to those reported the main text.
D.1 Other datasets
Similarly to Section 3 we modily the SVHN CIFAR-100 and higher resolution Imagenet-100 and Tiny Imagenet. datasets inject highly discriminative features v(y). Since SVHN also has 10 classes, we use the exact same settings as in CIFAR-10 and we train and evaluate with e where training on the original data does not lead to CO (recall B = 0 corresponds to the unmodified dataset). On the other hand, for CIFAR-100 and Imagenet-100 we select v(y) to be the 100 DCT components with the lowest frequency and we present results with e = 5 and = 4 respectively. Similarly, for Tiny Imagenett (which has 200 classes) we use the first 200 DCT components and present results with = 6. On Image.Net-100 we evaluate robustness with AutoAttack Croce & Hein (020), as PGD-50 seems to be broken when CO occurs in this dataset. On the other datasets, we evaluated with AttoAttack at the extrema of our plots and found that the estimated robustness agroes with the one of PGD-50. To save computational complexity, we only used PGD-50 to estimate the robustness the other datasets.
Regarding the training settings, for CIFAR10/100 and SVHN datasets we use the same settings as An- driushchenko & Flammarion 2020 For ImageNet- we follow Kireev et al. 2022 and for Tiny Imt
Li et al. 22200).
In all datasots we can obscrve similar trends as with CIFAR-10: For small values of 3 the injected features are not very discriminative due to their interaction with the dataset images and the model largely ignores them. As we increase 3, there is range in which they become more discriminative but not yet robust and
20

---

Published in Transactions on Machine Learning Research (07 (07/2023)
observe CO. Finally for large values of B the injected features become robust and the models achieve very good performance locusing only on those.
Clean accuracy
Robust accuracy
80%
00%
60%
50%
40%
70%
20%
60%
0%
0
2
6
10
12
14
4
10
12
14
(/255)
3 (/255)
5/255
Figure 10: Clean and robust performance after FGSM-AT on injected datasets D3 constructed from CIFAR- 100. As FGSM-AT already suffers CO on CIFAR-100 at e 6/255 we use e == 5/255 in this experiment where FGSM-AT does not suffer from CO seen for B In this setting, we observe CO happening when B is slightly smaller than Results averaged 3 seeds and shaded areas report minimum and maximum values.
Clean accuracy
Robust accuracy
100%
809
90%
60%
80%
40%
70%
20%
2
6
10
14
10
14
(/255)
(/255)
4/255
Figure 11: Clean and robust performance after FGSM-AT on injected datasets Dg constructed from SVHN. As FGSM-AT already suffers CO on SVHN at = 6/256 we use 4/255 in this experiment where FGSM-AT does not suffer from CO as seen for 0. In this setting, we observe CO happening when = e. Results are averaged over seeds and shaded areas report minimum and maximum values.
D.2 Other norms
Catastrophic overfitting has been mainly studied for loo perturbations and thus we presented experiments with lo attacks following related work. However, in this section we also present results where we induce CO with & perturbation which are also widely used in adversarial robustness. In Figure 13 we show the clean (left) and robust (right) accuracy after FGM-AT on injected dataset from CIFAR-10 (Dz). Similarly to our results with loo attacks, we also observe CO as the injected features become more discriminative (increased 3). It is worth mentioning that the &s norm we use (e = 1.5) is noticeably larger than typically used in the literature, however, it would roughly match the magnitude of an loo perturbation with e = 7/255. Interestingly, we did not obscrve CO for this range of 3 with =
7FGM is the &g version of FGSM where take sign of the gradient.
21

---

Published in Transactions on Machine Learning Research (07 (023)
Clean accuracy
Robust accuracy
100%
60%
80%
40%
60%
20%
10
12
2
8
10
12
(/255)
(/255)
6/255
Figure 12: Clean and robust performance after FGSM-AT on injected datasets D3 constructed from TinyImageNet. We use c == 8/255 this experiment where FGSM-AT does not suffer from CO as seen for B = In this setting, we observe CO happening for [7/255, 8/255].
Clean accuracy
Robust accuracy
100%
20%
93%
90%
15%
85%
10%
80%
5%
75%
0%
0.0
0.5
1.0
1.5
2.0
2.3
3.0
0.0
0.5
1.0
1.3
2.0
2.5
3.0
8 (/255)
(/255)
1.5/255
Figure 13: Clean and & robust performance after FGSM-AT on injected datasets Dg constructed from 1.5 in this experiment where
CIFAR-10. FGM-AT suffers CO CIFAR-10 around 2, FGM-AT does not suffer from CO seen for 8 = 0. In this setting, observe CO happening when 8 x Results are averaged over 3 seeds and shaded areas report minimum and maximum values.
D.3 Other injected features
We selected the injected features for our injected dataset from the low frequency components of the DCT ensure am interaction with the features present on natural images Ahmed et al. (1974). However, this does not mean that other types of features could not induce CO. In order to understand how unique choice of features also created another family of injected datasets but this time using set of 10 randomly generated vectors as features. As in the main text, we take the sign of each random vector to ensure they take values in {11 and assign one vector per class. In Figure 14 we observe that using random vectors as injected features we can also induce CO. Note that since our results are averaged over random seeds, each seed uses a different set of random vectors.
D.4 Other architectures
In all our previous experiments we trainde a PreActResNeet18 (He et al. 2016) as the standard architecture used in the literature. However, our observations our also robust to the choice of architecture. As we can see in Figure 15 we can also induce CO when training a WideResNNer28110 Zagoruyko & Komodakis (2016) on injected version of CIFAR-10.
22

---

Published in Transactions on Machine Learning Research (07 (07/202) /023)
Clean accuracy
Robust accuracy
100%
100%
80%
80%
60%
60%
40%
20%
40%
0%
10
12
2
6
8
10
12
(/255)
(/255)
6/255
Figure 14: Clean and robust performance after FGSM-AT on injected datasets D3 constructed from CIFAR-10 using random signals V. We perform this experiments at c= 6/255 where we saw that injected the dataset with the DCT basis vectors did induce CO. In the random V setting, we observe the same behaviour, with CO happening when B e. Results are averaged over seeds and shaded areas report minimum and maximum values.
Clean accuracy
Robust accuracy
100%
80%
98%
60%
95%
40%
92%
90%
20%
88%
0%
10
12
2
8
10
12
(/255)
(/255)
4/255
Figure 15: Clean and robust performance after FGSM-AT on injected datasets Di constructed from CIFAR-10 when training a WideResNet28x110, We perform this experiments at 4/255. Results are averaged over soods and shaded areas report minimum and maximum values.
E Performance on V
further understand the behavior of the models described Section 4 we conduct an additional experiment. ovaluating the performance of the networks on the set of fcatures This experiment corroborates the findings shown in Figure 2 and reveal more insights into how the networks utilize the features from V and D.
In Figure 16 see that networks trained using standard training demonstrate near-perfect accuracy on V from the start, indicating that they heavily rely on these features for classification and can correctly classify the data based solely on these. In contrast, networks trained with PGD-AT show a gradual improvement in accuracy on V, as they learn to effectively combine the information from V with that D to achieve robustness. anticipated, networks trained with FGSM-AT exhibit similar behavior those trained with PGD-AT initially. However, following the occurrence of catastrophic overfitting (CO), they experience sudden increase in accuracy V.
23

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Accuracy
100%
80%
60%
40%
20%
15 Epoch
25
30
Standard
FGSM
PGD
Figure 16: Accuracy on of networks trained on D3 using standard training, FGSM-AT or PGD-AT FGSM-AT. We use = 4/255 and 3 6/255, Results averaged over random seeds. Shades show min-max values.
F Learned features at different B
In Section 3 we discussed how, based on the strength of the injected features 8, our injected datasets seem have 3 distinct regimes: (i) When 3 is small argued that the network would not use the injected features as these would not be very helpful. (ii) When 3 would have very large value then the network would only look at these features since they would be easy-to-learn and provide enough margin to classify robustly. (iii) Finally, there was a middle range of 3 usually when / ~ c where the injected features would be strongly discriminative but not enough to provide robustness on their own. This regime is where we observe CO.
In this section we present an extension of Figure where we take FGSM trained models on the injected datasets (Dz) and evaluate them on three test sets: (i) The injected test (Dz) with the same features the training set. (ii) The original dataset (D) where the images are unmodified. (iii) The shuffled dataset (Dx(z)) where the injected features are permuted. That is, the set of injected features the same but the class assignments are shuffled. Therefore, the injected features will provide conflieting information with the features present on the original image.
In Figure 17 we show the performance on the aforementioned datascts for three different values of B. For 3 = 2/255 we are in regime (i) we observe that the tree datasets have the same performance, i.e. the information of the injected features does not seem to alter the result. Therefore, we can conclude the network is mainly using the features from the original dataset D. When 3 = 20/255 we are regime (ii) the clean and robust performance of the network is almost perfect on the injected lest set Dj while it is close to 0% (note this is worse than random classifier) for the shuffled dataset. So when the injected and original features present conflicting information the network aligns with the injected features. Moreover, the performance on the original dataset is also very low. Therefore, the network mainly using the injected features. Lastly, β = 8/255 corresponds to regime (iii) as discussed in Section 4 in this regime the network initially learns to combine information from both the original and injected features. However, after CO, the network seems to focus only on the injected features and discards the information from the original features.
G Analysis of curvature in different settings
In Figure 3 (left) we track the curvature of the loss surface while training on different injected datasets with either PGD-AT FGSM-AT. We observe that (i) Curvature rapidly increases both for PGD-AT and FGSM-AT during the initial epochs of training.
(ii) In those runs that presented CO, the curvature explodes around the 8th epoch along with the training accuracy. (iii) When training with the dataset with orthogonally injected features (D.) the curvature does not increase. This is aligned with our proposed mechanisms induce CO whereby the network increases the curvature in order to combine different features to learn better
24

---

Published in Transactions on Machine Learning Research (07/2023)
FGSM training 1,0 2/255)
FGSM training (8 8/255)
FGSM training (B 20/255)
100%
100%
100%
Coean uccuzsey
50%
50%6
5096
096
09%
076
10
20
10
20
30
600%
60%
100%
Rolass ucturay
00%
40%
500
20%
20%
0%%
10
20
30
30
30
20
30
Egoch
Epoch
Epoth
Injected (Ds)
CIFARIO (D)
Shuffled (Deta)
Figure 17: Clean (top) and robust (bottom) accuracy of FGSM-AT on D3 at different B values 3 different test sets: (1) the original CTFAR-10 (D), (ii) the dataset with injected features Dg and (iii) the dataset with shuffled injected features D Da()). All training runs use c c= 6/255. Left: B = =/225 Center: B = 8/253 Right: B = 20/255.
representations. In this section we exlend this analysis to different values of feature strength 3 on the injected dataset (Da). For details on how we estimate the curvature refer Appendix C
Figure 18 presents the curvature for different values of feature strength 3 on the injected dataset (Dz). We show three different values of B representative of the three regimes discussed in Appendix []] Recall that when B is small (3 = 2/255) we observe that the model seems focus only on CIFAR-10 features. Thus, we observe curvature increase aligned with (CIFAR-10) feature combination. However, since for the chosen robustness radii 6/255 there is no CO, observe that the curvature increase remains stable. When B is quite large (3- 20/255) then the model largely ignores CIFAR-10 information and focuses on the casy-to-learn injected features. Since these features are already robust, there is no need to combine them and the curvature does not need to increase. In the middle range when CO happens (B = 8/25s) we observe again the initial curvature increase and then curvature explosion.
Curvature
Training attack accuracy
100%
80%
10a
60%
10-1
40%
10-2
20%
20
15 Epoch
30
Epoch
8 2/255
8 8/255
3 20/255
Figure 18: Evolution of curvature and training attack accuracy of FGSM-AT and PGD-AT trained on D3 at different 3 and for c= 9/255. Only when CO happens (for 3 = 8/255) the curvature explodes. For the other two interventions the curvature does not increase much. We argue this is because the network does not need to disentangle D from V, as ignores either of them.
25

---

Published in Transactions on Machine Learning Research (07 (07/2023)
H Adversarial perturbations before and after CO
Qualitative analysis In order to further understand the change in behaviour after CO presented visualizations of the FGSM perturbations before and after CO in Figure | We observed that while prior to CO, the injected fealure components v(y) were clearly identifiable, after CO the perturbations do not seem to point in those directions although the network is strongly relying on them to classify, In Figure [9 and Figure 20 we show further visualizations of the perturbations obtained both with FGSM or PGD attacks on networks trained with either PGD-AT or FGSM-AT respectively.
We observe that when training with PGD-AT, ie. the training does not suffer from CO, both PGD and FGSM attacks produce qualitatively similar results. In particular, all attacks scem to target the injected features with some noise due to the interaction with the features from CIFAR-10. For FGSM-AT, we observe that at the initial epochs (prior to CO) the pertubations are similar to those of PGD-AT, however, after CO perturbations change dramatically both for FGSM and PGD attacks. This aligns with the fact that the loss landscape of the network has dramatically changed, becoming strongly non-linear. This change yields single-step FGSM ineffective, however, the network remains vulnerable and multi-step attacks such as PGD are still able to find adversarial examples, which this case do not point in the direction of discriminative features Jetley al. )2018); []yas al. (2019).
Intersmod sacaples (8.9) > Ds
Intevened samples (x. - Dy
frsss epodh
fyon epoch
Sypss at spodi 25
Syen at spoch
(a) FGSM adversarial perturbations
(b) PGD adversarial perturbations
Figure 19: Different samples of the injected dataset DR, and adversarial perturbations at epoch 4 and 22 of PGD-AT on D, <= 9/255 and B 8/255 (where FGSM-AT suffers CO). The adversarial perturbations remain qualitatively similar throughout training and align significantly with V.
Quantitative analysis Finally, to quantify the radical change of direction of the adversarial perturbations after CO, we compute the evolution of the average alignment (i.e., cosine angle) between the FGSM perturbations 8 and the injected features, such that if point æ is associated with class y compute |
(8.u(y)) Figure (left) shows the results of this evaluation, where we can see that before CO there is
a non-trivial alignment between the FGSM perturbations and their corresponding injected features, that after CO quickly converges to the same alignment as the one between two random vectors.
To complement this view, we also perform an analysis of the frequency spectrum of the FGSM perturbations. In Figure 6 (right), we plot the average magnitude of the DCT transform of the FGSM perturbations computed on the test set of an intervened version of CIFAR-10 during training, As CO, most of the energy of the perturbations is concentrated around the low frequencies (remember that the injected features are low frequency), but after CO happens, around epoch 8, the energy of the perturbations quickly gets concentrated towards higher frequencies. These two plots corroborate, quantitatively, our previous observations, that before CO, FGSM perturbations are pointing towards mcaningful prodictive features, while after CO, although we know the network still uses the injected features (see Figure 2) the FGSM perturbations suddenly point in different direction.
see, prior to
26

---

Published in Transactions on Machine Learning Research (07 (272))
Interveed sacupães
Intevened sumple (r.g) e Dy
(5.9) : Da
dyess epodh
epoch
degas epodi 25
drem at spoch
(a) FGSM adversarial perturbations
(b) PGD adversarial perturbations
Figure 20: Different samples of the injected dataset Da, and adversarial perturbations at epoch 4 (before CO) and 22 (after CO) of FGSM-AT on D3 at c= 9/255 and B 8/255 (where FGSM-AT suffers CO). The adversarial perturbations change completely before and after CO. Prior to CO, they align significantly with V, but after CO they point meaningless directions.
Further results with N-FGSM, GradAlign and PGD
In Section we studied different SOTA methods that have been shown to prevent CO. Interestingly, we observed that in order to avoid CO on the injected dataset stronger level of regularization is needed. Thus, indicating that the intervention strongly favouring the mechanisms that lead to CO. For completeness, in Figure 21 we also present results of the clean accuracy (again with the robust accuracy). As expected, for those runs in which we observe CO, clean accuracy quickly saturates. Note that for stronger levels of regularization the clean accuracy lower. An ablation of the regularization strength might help improve results further, however the purpose of this analysis is not to improve the performance the injected dataset but rather to show it is indeed possible to prevent CO with the same methods that work for unmodified datasets.
Clean accuracy
Robust accuracy
1.0
80%
60%
40%
20%
0%
10
14
1/235)
B (/255)
PGD-10
GradAlign (A 0.112)
GradAlign (A 2)
N-FGSM 10c)
N-FGSM (k = 2c)
Figure 21: Clean (left) and robust (right) accuracy after AT with PGD-10, GradAlign and N-FGSM on D3
6/255. Results averaged over three random seeds and shaded areas report minimum and maximum values.
J Further details of low-pass experiment
We expand here over the results in Scction 7 and provide further details on the experimental settings of Table Specifically, we replicate the same experiment, i.e., training low-pass version of CIFAR-10 using FGSM-AT at different filtering bandwidths. As indicated in Section 7 the low-pass filter introduced
27

---

Published in Transactions on Machine Learning Research (07/2023)
in Ortiz-Jimenez et al. (2020b) which only retains the frequency components in the upper left quadrant of the DCT transform of an image. That is, a low-pass filter of bandwidth W would retain the W x W upper quadrant of DCT coefficients of all images, setting the rest of the coefficients to O.
Figure 22 shows the robust accuracy obtained by FGSM-AT on CIFAR-10 versions that have been pre-filterdd using such low-pass filter. Interestingly, while training on the unfiltered images does induce CO on FGSM-AT, just removing few high-frequency components is enough to prevent CO 8255. However, described before, it seems that at 10/255 frequency transformation can avoid CO. Clearly, this transformation cannot be used technique to prevent CO, but it highlights once again that the structure of the data plays significant role in inducing CO.
Robust accuracy
50%
10%
30%
20%
10%
0%
10
15
20
25
30
Bandwidth
FGSM-AT 8/25)
PGD-AT (e=8/255)
FGSM-AT (e=16/25))
PGD-AT (e- (6/2555
Figure 22: Robust accuracy of FGSM-AT and PGD-AT on different low-passed versions of CIFAR-10 using the DCT-based low pass filter introduced in Ortiz-Jimenez et al. (2020b). Bandwidth = 32 corresponds to the original CIFAR-10, while smaller bandwidths remove more and more high-frequency, components. At 8/255 just removing few high-froquency components is enough to prevent CO, while at c 16/255 no frequency transformation avoids CO.
Relation with anti-aliasing pooling layers As mentioned in Section 7 our proposed low- low-passing technique very similar in spirit works which propose using anti-aliasing low-pass filters at all pooling layers Grabiinkk et al., 2022 Zhang 2019). Indced, as shown by Ortiz-Jimenez et al. (2020b), CIFAR10 contains a siguilicant amount of non-robust features the high-frequency end of the spectrum due to aliasing produced in their down-sampling process. In this regard, it is no surprise that methods like the one proposed in Grabinski et al. (2022) can prevent CO at = $/255 using the same training protocol as in our work (robust accuracy is 45.9%). Interestingly, though, repeating the experiments in Grabinski et al. 2022 work using 6 = 16/255 does lead to CO (robust accuracy 0.0%) in Section 7 This result was not reported the original paper, but we see it a corroboration of our observations. Indeed, features play role in CO, but the problematic features do not always come from excessively high-froquenncie aliasing. However, we still consider that preventing aliasing in the downsampling layers is a promising avenue for future work in adversarial robustness.
28


Reviews:
[{"review_id": "wh1faL5G4B", "reviewer": "Reviewer_KwEe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper thoroughly investigates CO in FGSM-AT. The design of the data intervention through linearly separable perturbations is innovative and provides flexibility to verify CO in different settings empirically. Furthermore, the connections made to shortcut learning and heavy regularisation through GradAlign put the paper well within the literature.\n\n- The paper is well-structured and easy to read. It takes the reader through each step of the empirical investigation and provides sufficient reasoning for the observations made.\n\n- The experiments are conducted over multiple datasets (CIFAR-10, CIFAR-100, SVHN, TinyImageNet, and ImageNet-100) and models (ResNet and WideResNet). This would make the results likely to be reproducible.\n\n### Weaknesses:\n\n- Currently, the paper only provides interesting observations into CO. The paper needs to delve deeper into the consequences of these observations and how we can exploit them to mitigate CO. Currently, it only investigates existing options, such as GradAlign, or presents a low-pass filtration technique that would only work for small perturbation magnitudes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically studies the causes of catastrophic overfitting (CO) in FGSM adversarial training (AT). In particular, this paper proposes a novel data intervention technique to induce CO. To this end, a set of linearly separable data points are added to the original data. By controlling the magnitude of these easy-to-learn features, the paper aims to force CO. Conducting extensive exploratory experiments on various datasets and models, the paper concludes that CO happens when the network prefers easy non-robust features over hard-to-learn robust ones. Using these observations, the paper makes interesting connections between the observed behaviour and well-known phenomena such as shortcut learning and curvature increase in deep neural networks. A preliminary approach to using these observations in mitigating CO is also proposed. Motivated by the reliance of DNNs on high-frequency features, the authors suggest using a simple low-pass version of CIFAR-10 to prevent CO. However, this method is shown to be effective in $\\epsilon=8/255$ perturbation norm. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned above, I believe that even if the authors cannot provide a viable working solution based on the observations made, it would be beneficial to expand on Sec. 7. In other words, I encourage them to discuss the possible ways that these empirical observations could be incorporated to mitigating CO in practice. This would make the paper more interesting.", "broader_impact_concerns": "The current submission needs to provide a broader impact section. It would be nice if the authors could provide such a statement for the general audience.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qyn35JuKHf", "reviewer": "Reviewer_z6NM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n1. The authors provided ample experiments covering both breadth and depth of research questions regarding CO.\n2. The experimental results are non-trivial enough to intrigue some other readers’ interests.\n    1. In particular, the experiments covering the reason for the curvature increase (Section 5) contains some non-trivial finding regarding CO.\n\n# Weakness\n1. I’m not sure if the findings reflect some realistic scenarios and if the results explain the reason for CO in reality.\n    1. In Section 4, The authors point out the networks’ preference for the injected features $\\mathcal{V}$. Also, in Section 5, the authors point out that the interaction between the feature in the input $x$ and the injected feature $\\mathbf{v}(y)$ is the cause of CO. However, where are those label-dependent features in the real world?\n    2. I’m worried that the findings could be limited to the authors’ experiment setting. To explain, the existence of the injected features is crucial, and the findings might not generalize in real-world CO that does not involve such injected features.\n    3. If the authors have some good way to justify this difference (between the experiment setting and the real-world scenario), such justification should appear in the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper experimentally explores the reason for catastrophic overfitting (CO), a problematic behavior of adversarial training utilizing single-step attacks.\n\nThe main experiment starts from injecting synthesized perturbations $\\mathbf{v}(y)\\in\\mathcal{V}$ (of size $\\beta$) that depend on the label $y$ to the training set (following the distribution \\mathcal{D}). The authors first observe that for mildly strong perturbations of size $\\beta\\approx\\epsilon$, they can induce CO for adversarial training using FGSM of attack size $\\epsilon$.\n\nThen, to understand why FGSM-based adversarial training (FGSM-AT) suffers CO for such $\\beta\\approx\\epsilon$, the authors compare the clean/robust accuracies of standard training, FGSM-AT, and PGD-AT on three different datasets: the clean dataset, the dataset after adding label-dependent perturbations $\\mathbf{v}(y)$, and the dataset after adding perturbations $\\mathbf{v}(\\pi(y))$ that depends on perturbed label $\\pi(y)$. From this comparison, the authors observe that the FGSM-AT’s accuracies behave similarly to those of PGD-AT before CO, but they behave similarly to those of standard training after CO. Because standard training resorts to the label-dependent features $\\mathcal{V}$ whereas PGD-AT utilizes both $\\mathcal{D}$ and $\\mathcal{V}$ to achieve high robust accuracies, they conclude that CO changes the learned feature of FGSM-AT from \"both $\\mathcal{D}$ and $\\mathcal{V}$\" to \"$\\mathcal{V}$ only\".\n\nNext, following the previous observations that the local curvature of the loss landscape explodes after CO, the authors tracked the evolutions of the loss curvatures for both FGSM-AT and PGD-AT. From the experiment, they found out that the FGSM-AT experience the explosion of loss curvature, whereas PGD-AT successfully suppresses the curvature increase. After the curvature increase, the local loss landscape becomes highly nonlinear. However, because FGSM assumes a linear loss landscape, FGSM starts generating meaningless perturbations for adversarial training, so FGSM-AT starts suffering CO. With an additional experiment with a dataset by projecting $\\mathcal{D}$ onto the orthogonal complement of $\\mathcal{V}$ (to remove features in $\\mathcal{D}$ that interacts with $\\mathcal{V}$), the authors also demonstrated that the curvature increase is due to the interactions between the features from $\\mathcal{D}$ and those from $\\mathcal{V}$.\n\nFinally, the authors summarize their findings about the reason for CO, then present some additional analysis of existing CO prevention methods.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. As pointed out in the weakness, the experiment design may reflect something other than what really happens in real-world CO.\n    1. I’m unsure whether it is possible to provide a connection between the settings of the experiments and the real-world scenario.\n    2. Please justify that the findings can generalize to the realistic CO cases.\n2. Figure 2 successfully visualizes the effect of CO on FGSM perturbations. \n    1. Similarly, some people would be interested in looking at the effect of CO on the classification. To explain, after the model learns with the meaningless FGSM perturbations, does the model just lose robustness, or it starts focusing on the meaningless FGSM perturbation? (Is the FGSM perturbation just meaningless, or it has some negative impact on training?)\n    2. To answer this question, I suggest the authors use pixel attribution methods such as Grad-CAM to highlight the feature that the model used for classifications.\n3. The finding in the last paragraph of Section 7 may require more attention and raises other research questions.\n    1. Do the authors have some experimental support that the curvature increase (by AutoAttack) actually hindered PGD optimization, or is there any other reason for this?\n    2. If AutoAttack (AA) introduced such a bad curvature increase, why does it happen specifically for AA? Do the authors have a reasonable justification?\n4. Some minor changes would make the paper easier to read.\n    1. For example, I suggest the authors move Figure 1 to Section 3 rather than Section 1. (I don’t see the reason to put Figure 1 in Section 1, whereas it is not mentioned in Section 1.)\n    2. Similarly, Figure 2 can be moved to Section 5. While it is mentioned in Section 3, the readers would be more interested in the findings in Section 5 rather than the advantage of the authors’ design choice.\n", "broader_impact_concerns": "I don’t see a particular broader impact concern regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "up436oZZF6", "reviewer": "Reviewer_VSzM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea and method proposed by the author are innovative. Through experimental validation, it is demonstrated that CO can be induced, thereby inferring that the occurrence of CO in the original dataset is due to the same reasons. That is, the network exhibits a preference for certain features in the dataset, resulting in shortcuts.\n2. The experiments are well-designed and comprehensive. \n3. The writing is clear and easy to follow.\n\nWeakness\n1. As stated in Section 5.1, “before CO, the adversarial perturbations point in the direction of V.” But should it be in the opposite direction of V？ Since adversarial attack aims to obfuscate useful features and V is exactly the useful feature here. I believe some clarification is needed here.\n2. There is a writing error in equation (3), the comma should be a multiplication symbol.\n\nPost rebuttal:\nThank the authors for the clarification. I personally think the proposed finding is new and the authors have adequately verified the finding. I thus recommend acceptance of this paper. Also, I would like to suggest the authors add the keywords \"adversarial\" or \"adversarial robustness\" in the title to clarify the scope of the paper (as the phenomenon may not exist for other settings, like self-supervised training).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on catastrophic overfitting (CO) in single-step AT methods. The author proposes a new approach that introduces class-wise linearly separable noise into the natural dataset to investigate the relationship between CO, the magnitude of the noise added to the dataset, and the size of the attack budget used during training. Through a series of experiments, the author discovers that injecting noise into the dataset can induce CO. The authors have shown that CO is a kind of shortcut used by the network to avoid learning complex robust features while achieving high accuracy using easy non-robust ones. Furthermore, the authors investigate the mechanism of CO from the perspectives of curvature and feature interactions.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I wonder whether an experiment with ViT can be performed with the CO.", "broader_impact_concerns": "There are no obvious concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers suggest accepting the paper (1 x accept, 2 x leaning accept), thus, there is a clear consensus.
The authors put a lot of effort into addressing comments from reviewers. The updated version of the paper reads very well and, in my opinion, it does not require further corrections. As a result, I propose to accept the paper as is.
Decision: Accept (poster)