Perturbation Deterioration: The Other Side of Catastrophic Overfitting
Authors: Zichao Li
Liyuan Liu
Chengyu Dong
Jingbo Shang
Venue: ICLR 2022 Submitted (2022)

Abstract: Our goal is to understand why the robustness accuracy would abruptly drop to zero, after conducting FGSM-style adversarial training for too long.  While this phenomenon is commonly explained as overfitting, we observe that it is a twin process: not only does the model catastrophic overfits to one type of perturbation, but also the perturbation deteriorates into random noise. For example, at the same epoch when the FGSM-trained model catastrophically overfits, its generated perturbations deteriorate into random noise. Intuitively, once the generated perturbations become weak and inadequate, models would be misguided to overfit those weak attacks and fail to defend strong ones. In the light of our analyses, we propose APART, an adaptive adversarial training method, which parameterizes perturbation generation and progressively strengthens them.  In our experiments, APART successfully prevents perturbation deterioration and catastrophic overfitting.  Also, APART significantly improves the model robustness while maintaining the same efficiency as FGSM-style methods, e.g., on the CIFAR-10 dataset, APART achieves 53.89%accuracy under the PGD-20 attack and 49.05% accuracy under the AutoAttack.

Under review as a conference paper at ICLR 2022
PERTURBATION DETERIORATION: OF CATASTROPHIC OVERFITTING
THE OTHER SIDE
An Anonymous authors Paper under double-bling review
ABSTRACT
Our goal is to understand why the robustness accuracy would abruptly drop to zero, after conducting FGSM-style adversarial training for too long. While this phe- nomenon is commonly explained as overfitting, we observe that it is a twin process: not only does the model catastrophic overfits to one type of perturbation, but also the perturbation deteriorates into random noise. For example, at the same epoch when the FGSM-ttriined model catastrophically overfits, its generated perturbations deteriorate into random noise. Intuitively, once the generated perturbations become weak and inadequate, models would be misguided to overfit those weak attacks and fail defend strong ones. In the light of our analyses, we propose APART, adaptive adversarial training method, which parameterizes perturbation generation and progressively strengthens them. In our experiments, APART successfully prevents perturbation deterioration and catastrophic overfitting, Also, APART significantly improves the model robustness while maintaining the same efficiency as FGSM-style methods, e.g., on the CIFAR-10 dataset, APART achieves 53.89% accuracy under the PGD-20 attack and 49.05% % accuracy under the AutoAttack]]
INTRODUCTION
While neural networks keep advancing the state of the arts, their vulnerability to adversarial at- tacks (Szegedy al. (013) casts shadow over their applications--suulee human-imperceptible input shifts can fool these models and alter their predictions. Adversarial training adds perturbations to model inputs during training, and is one of the most successful approaches establish model
robustness (Goodfellow et al. 2014 Madry et al. 2017 Kurakin et al. 2017 Tramer et al. 2017
Zhang et al. [019b Liu al., 2018).
One crucial and common challenge in adversarial training is the significant computation overhead, e.g., may take 3-30 times longer to conduct adversarial training than the vanilla training. In response to this challenge, there has been recent surge in work aiming to reduce the computation overhead (Goodfellow et al., 2014 Shafahi et al.) 2019a Zhang et al., 2019a Wong et al. 2020). Although these methods successfully accelerates adversarial training, they lead to an unexpected phenomenon-wher conducting FGSM-style adversarial training, the model robustness would abruptly drop zero (Rice et al. 2020) at certain epoch. This phenomenon is referred catastrophic overfitting, and the robustness drop is usually viewed as model overfitting the model overfits to one specific type of perturbation (Rice al. 2020 Wong et al. 2020
Kim al. 2021).
Here, we show that the robustness drop is twin process. Besides the model overfitting to one observe perturbations becoming weak establish model robustness.
type of perturbation, For example. as visualized in Figure [ both the model and the perturbation change dramatically between Epoch 15 and Epoch (6-the robust accuracy of the model drops from 45 almost zero, the perturbation strength deteriorates dramatically. As implied by this phenomenon, we suggest that there exists strong correlations between the catastrophic overfitting and perturbation deterioration. Intuitively, without perturbation deterioration, even if the model overfits to one type of perturbation with reasonable strength, it would lead to sub-optimal robustness instead of entirely diminished robust accuracy. Meanwhile, once the perturbation deteriorates into random noise, overfitting to that random noise-like perturbation could cause the model robustness drop to zero.
Code will he released under the Apache-2.0 license for future studies.

---

Under review as a conference paper at ICLR 2022
Table 1: Notation Table (Elaborated in Section 2
x is input
y is label
a step size
is objective
Ax = 0C/0x
A6 = 0C/00
model parameter trained for epochs by method A
fA x, v)
perturbation generated by method A attack model @ x, y
14'x
perturbation initialization as parameterized by FGSM+ and APART
(A,
perturbation strength ( calculated as its gap to fa( ( on model 0
Acc(O (ss(0(
accuracy of 0((2) when attacked perturbations generated by fs(()) for o
Original Image
10
0.4
15th spach
Accuracy
0.3
Petuttation
228
bedggann
16 spoch
Pertubationn
FESM
an rendam
APART-Stmple
manner
APART
Truck
rd
Airslane
Ship
Index of epochs
(a) FGSM-Generated Perturbations.
(b) Test Accuracy under PGD-10.
Figure 1: Analyses of FGSM-generated perturbations (for Pre-ResNett 8 on the CIFAR-10 dataset).
From this perspective, the key to prevent catastropic overfitting falls upon shielding models from per- turbation deterioration. Correspondingly, we design an adaptive adversarial training method, APART, which parameterizes perturbation generation and updates its parameters to progressively strengthen the perturbation generator with gradient ascent, Specifically, we first treat perturbation initialization parameters-aae instead of starting from scratch every time, APART improve the initialization with gradient ascent. Then, APART factorize the input perturbation as a series of perturbations, which integrated with learnable step sizes and can self-adapt different scenarios. In experi- ments, APART leads to consistent performance improvements over FGSM-style algorithms while maintaining roughly the same efficiency.
PRELIMINARIES AND NOTATIONS
Given neural network with convolution blocks, we denote the input of the i-th block the input and output of the entire network as x and y. Note that, x and X₃ are the same in conventional residual networks, such ResNet (He et al. 2016a), Wide ResNet (Zagoruyko & Komodakis 2016)), and Pre-Act ResNet (He et al.) 20166b). Adversarial training aims to establish the model robustness by solving the optimization problem below (0 is the network parameter, 5 is the perturbation, (x, y) is a data-label pair, and f(.) is the perturbation generation function).
min C(O, + d, s.t. 8 f(0, x, v)
Different adversarial training methods generate perturbations differently. Ideally, adversarial training should use the most effective (i.e., strongest) perturbation within the same (*(() = arginax|ssjss C(O, x x y). In practice, as an approximation, f(-) ) is typically implemented gradient ascent with fixed iteration and step size. For example, the FGSM algorithm directly
constraint, i.e.,
calculates perturbations as (f x, y) = c sign(Ax). Also, we use method to refer to model parameters that trained by specific adversarial training algorithm for epochs. We use ( fE(()?), .)) to indicate the performance of model gfill) under fe(O( ..)). i.e., the
g())
perturbation generated by method fs(.) attack o(:2) These notations summarized in Table
2

---

Under review as a conference paper at ICLR 2022
16
155H 30A0
0.82
FOSM APART
Accuracy
0.76
Gro0-10
15
5-
0.70
FREM
10
15
20
25
20
Index epochs
Index of epochs
(a) Accuracy of @paD 10 on transferred attacks (smaller value indicates stronger attacks). Line named method PGD-10: (ress((fuctta 1)), while Random (()))
(b) Strength gap between method and PGD-10, i... L(B
methoei X + Smeetua ( X, 3). - L( meeeood X +
refers to Acc(Oppp refers to Aco pen.. fandum (-))-
frein- Aean-1(0m (otthaa X, 4). Smaller value indicates
stronger attacks.
Figure 2: Perturbation strength in different epochs. In part (a), perturbation strength estimated by transfer the adversary image to attack model trained by PGD-10 separately for 30 epochs. In part (b), perturbation strength estimated as the gap to a stronger attack (PGD-10 here)
3 PERTURBATION DETERIORATION AND CATASTROPHIC OVERFITTING
Typically, the robustness drop is viewed as model overfitting (Rice et al. 2020 Wong et al. 2020 Kim et al. 202T) and this phenomenon is referred as catastrophic overfitting. Meanwhile, it has been observed that catastrophic overfitting only happens to simple methods like FGSM (Goodfellow et al. 2014 D indicating this phenomenon is not only about model overfitting, but also the strength of perturbation. Inspired by this observation, aim to explore the other side of catastophic overfitting, ie., perturbatioon deterioration.
3.1 PERTURBATION STRENGTH
To verify our intuition, we try to empirically estimate the perturbation strength and analyze its dynamics during the adversarial training.
First, we try to estimate the perturbation strength of fA as Acc(O2ar.n fa -), i... using fa ( attack model trained separately with PGD-10 for 30 epochs. For comparisons, we also list the performance of ( under random noise. As visualized in Figure 2(a) in the first 15 epochs, the perturbation strength of FGSM keeps increasing. while the 16 epoch, its strength dramatically drops to the random noise level. This shows that, after the perturbation deterioration, the perturbations only look like random noise, but also behaves like random noise.
Alternatiell,, we estimate the perturbation strength of fA as its gap to more powerful method fs. Specifically, calculate the strength for (((()) , .) (A,) in Equation 2 Intuitively, the weaker the perturbation fA is, the larger (A, would be.
(A. ((5x) - L(dg) where L(Smetod) = C(8 :x fmenood (09 x, v).)))
(2)
We conduct experiments with Pre-Act ResNet18 on the CIFAR-I0 dataset and visualize (FGSMM Figure 2 It shows that the strength gap between PGD-10 and FGSM is small in the early stage, dramatically explodes at the 16th epoch, and keeps a large value since then (i.e., the perturbation strength mostly vanishes at the 16th epoch).
Perturbation Deterioration. In both cases, the timing of perturbation strength deterioration co- incides with the timing of the robustness drop, thus supporting our intuition that the other side of
For FGSM trained Pre-ResNet13 (on CIFAR-10), its accuracy under PGD-20 attacks drops from 45% (epoch 15) 0% (epoch 16); For PGD-10 trained model, drops from 49% (epach 30) to 38% (epoch 200).
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1: APART (the first and the second round propagations are marked with red and blue; c the perturbation bound; pa. Pas and Pa are learning rates for 6, w, and o; Table summarizes others notations).
while not converged do
for x, in the training set do
81 t Ow W/ / ///innitaaiizee the perturbation for the model input. 81 < max(min(0i + ai signn @2(6x++61.) -)), +e) / calculate d.
8x1
for each residual block with index i
do
dg f sign( @0(0x+6110) // //caalllllae perturbations for block i.
8x;
0 = 0 - pa hi (i
/update parameters.
Wx = max max((in( (min(wx + Mu sign( 0L(0;(x:] 1
+ - -1), / //uddtte we.
Ous
a4 as + Ma (x(0 + (6 A
Mo.B. // //update step sizes.
da.
8au
10
return 0
catastrophic overfitting is perturbation deterioration. Intuitively, adversarial training cannot establish satisfying model robustness without strong enough perturbations, and strong perturbations require small gap 9(-). However, one local optima of Equation [ is the parameter 0 that deteriorates ((0,-) into random noise, which advances the optimization of Equation at the cost of deteriorated perturbations. Also, since all parameter updates are made decrease L, most existing methods have no regularization to keep the perturbation strength.
3.2 ADAPTIVITY HELPS PREVENT CATASTROPHIC OVERFITTING
Here, we further verify our intuition by showing that it prevents the catastrophic overfitting by alleviating the perturbation deterioration. Intuitively, straightforward way strengthen generators to parameterize them and update them together with model parameters. Specifically, we treat the perturbation initialization for the input (denoted as wx) and the step size (referred as a)) parameters of FGSM, and change the objective from Equation to:
max min L(B: x fras Wx: 8, x). a,2
(3)
During model training, we update 0 with gradient descent and update and wi with gradient ascent. We refer this variant as APART-Simple Note that its only difference to FGSM that APART- Simple is parameterized and can be enhanced during training, thus suffering less from perturbation deterioration.
We conduct experiments with Pre-Act ResNet18 on the CIFAR-10 dataset, and visualize Acc(o(30)
(APART-Simpll in Figure It shows that APART-Simple does not suffer from the catastrophic overfitting. This not only supports our intuition that the perturbation deterioration is one cause of the catastrophic overfitting, but motivates
peccin fress (e(1) Simpee 1) in Figure 2(a) and
us to add more adaptivity to the perturbation generator.
4 ADAPTIVE ADVERSARIAL TRAINING
Guided by our analyses, we propose to improve FGSM by further improving the perturbation generator during the training. Since the algorithm features the ability to adapt itself, we refer our method adaptive adversarial training (APART). Specifically, it first factorizes the perturbation for the input image into a series of perturbations, one for each residual block. Moreover, it employ different step sizes for different perturbations, treat them learnable parameters, update them to integrate perturbations adaptively, and strengthen the generator during the model training.
Factorize the Input Perturbation. For multi- network, the perturbation generated at the input attacks not only the first layer, but also all following layers. Intuitively, existing methods like PGD-N implicitly blender these attacks with N additional forward- and backward- propagations
backward-propagatioon
4

---

Under review as a conference paper at ICLR 2022
Table 2: Model Performance of WideResNNe34-10 on the CIFAR-I0 dataset.
Efficient
Methods
PGD-20
AA
C&W
Gaussian
Clean Data
Time/Epoch
ATTA-10
54.33%
49.10%
59.11 %
77.05%
83.80%
706 secs
PGD-10
55.41 %
52.08 %
58.77%
77.70%
86.43 %
680 secs
Free-8
47.68%
46.21%
56.31%
75.98%
85.54 %
252 secs
S+FGSM
36.71%
33.15%
44.50%
81.25% %
85.15%
232 secs
F+FGSM
46.37%
44.27%
56.21%
75.10%
85.10%
122 secs
V
APART
53.89%
49.05%
58.50%
77.31%
84.65%
162 secs
the input perturbation, which significantly inflates the computation cost. Here, we factorize the input perturbation series of perturbations and explicitly learn combine them. Specifically, we refer to the input of i-th residual block as X; and the output of i-th residual block as x; (X,ssx;)). Then, add the perturbation Ax; to the input of CNNs(.) to establish its robustness, i.e. x; CNNs(x; + 5;) where 5; = o; Ax,, Similar to PGD-N, this approach also involves multiple perturbations; different from PGD-N, these perturbations can be calculated with the same forward- and backwarr-propaagatio propagations.
Initialization Parameterization. Similar APART- -imple, we also treat perturbation initializations as learnable parameters to better defense the deterioration. Since it consumes additional memory, we only parameterize the perturbation initialization the input, and keep all other perturbations zero-initialized. In this way, the additional storage has roughly the same size with the dataset.
APART Algorithm. We summarize APART in Algorithm I Same with FGSM, contains two rounds of forward- and and calculates gradients for both the input of the first layer and the input of all the following blocks. In the second round, it applies the generated perturbations to the input of the corresponding blocks, Le., change x; + CNNs(x,) to x, + CNNs(x; + 5,). Then, besides updating model parameters with gradient descent, we enhance the generator with gradient ascent (i.e., updating step sizes as and the perturbation initialization wx). Note that, to control the magnitude of step sizes ag, we add a L2 regularization its updates and
In the first round, it initializes the input perturbation
control it (as line 9 in Algorithm D
Note that, calculating the exact gradients of w'x or Ow requires a second order derivation (Awx and Aaw based 6,, and the calculation of 6, includes some first order derivations involving Wx and o.). Due to the success of the First-order MAML (FOMAML) (Finn et al.) 2017), we simplifies the calculation by omitting higher order derivations. Specifically, FOMAML demonstrates the effectiveness to ignore higher order derivations and approximate the exact gradient with only first order derivations. Here, we have similar objective with FOMAML-FFMAMM aims to find good model initialization, and try to find a good perturbation initialization. Thus, we also restrict gradient calculations to first-order derivations. In this way, APART has roughly the same computation complexity with FGSM and it is significantly faster than PGD-N.
EXPERIMENTS
As in Figures ((a) and Figure 2(b) APART shields adversarial training from catastrophic overfitting and largely alleviates the robustness drop. Systematic evaluations are further conducted as below.
5.1 EXPERIMENTAL SETTINGS
Datasets. We conduct experiments on the CIFAR- 10 and CIFAR-100 datasets (Krizhessky 2009 as well as the ImageNet dataset (Krizhevsky et al. 2012).
Neural Architectures. We conduct experiments with ResNet He et al. (2016a). Pre-ResNet (He al. 2016b) and WideResNet (Zagoruyko & Komodakis 2016). Specifically. we use Pre-ResNet18 and WideResNee33-11 10 on the CIFAR-10 dataset, Pre-ResNet18 on the CIFAR-100 dataset, and ResNet50 (He et al. 2016a) the ImageNet dataset.
Data Augmentation. Following the previous work, we apply the standard data augmentation. For the CIFAR datasets, we apply random flipping data augmentation procedure and take a random
5

---

Under review as a conference paper at ICLR 2022
Table 3: Model Performance of Pre-ResNet18 the CIFAR-10 dataset.
Efficient
Methods
PGD-20
AA
C&W
Gaussian
Clean Data
Time/Epoch
ATTA-10
49.03%
45. 45.70%
58.30%
73.10%
82.10%
140 secs
x
PGD-10
52.16%
48.50 %
58.97%
72.33%
82.05%
133 secs
Free-8
47.37%
44.53%
56.10%
72.56%
81.64%
62 secs
S+FGSM
34.47%
32.15%
42.21%
73.45%
89.28 %
61 secs
F+FGSM
46.06%
42.37%
55.34%
72.25%
83.81% %%
20
APART
51.30%
45.92%
58.73%
73.65%
82.80%
29 secs
crop with 32 32 from images padded by 4 pixels each side (Lee et al. 2015). For the ImageNet dataset, we divide the training into three phases, where phases and use images resized to 160 and 352 pixels and the third phase uses the original images (Wong et al. 2020).
Optimizer. For experiments, we use SGD with momentum as the optimizer. The momentum factor is set to 0.9 and the training conducted for 60 epochs with eyclic learning rate (Smith 2017), where the maximum learning rate is set to 0.2 and minimum learning rate is set to O. For the ImageNet dataset, adopt setting similar to WWng al. (2020) and train the model for 15 epochs; The maximum learning rate of cyclic learning rate schedule is set to 0.4 and the learning rate is set to O.
Other Syper-parameters For all experiments, apply the cyclic learning rate scheduler for Pa- On the CIFAR datasets, the maximum learning rate and À are set as 5 x 10-8 and 200, respectively; on the ImageNet dataset, they between line 4 and line 7 in Algorithm [ we set Pw as 21, which makes the update on w has similar impact with the update in line of [
4 10-9 and 5000, respectively. Due the similarity
Robustness Evaluation. We adopt PGD-20, AutoAttack (Croce & Hein 2020). Gaussian random noise, and C&W (Carlini & Wagner 2017) as the attack methods for evaluation. For both adversarial training and evaluation, restrict perturbations 0 8/255 on the CIFAR datasets, Jō oc S 2/255 on the ImageNet dataset.
Infrastructure. Our experiments conducted with NVIDIA Quadro RTX 8000 GPUs; mixed- precision arithmetic (Micikevicius et al. (2018) is adopted to accelcrate model training; the training speed of APART or baselines is evaluated on idle GPU.
5.2 COMPARED METHODS
For comparison, we select three state-of-the-art adversarial training methods, which features efficient training. Also, we list two other adversarial training methods that are significant slower. On the CIFAR datasets, report accuracy and training time based our experiments. As to ImageNet, we directly refer to the number reported the original papers.
PGD-N (adryy et al. 2017) is a classical, sophisticated adversarial training method. PGD is an iterative version of FGSM with uniform random noise as initialization and the number of
iterations.
ATTA-K (Zheng et al. 2019) uses the adversarial examples from neighboring epochs. K is the denotes the strength attack.
number of iterations
Free-m (Shafahi et al. 2019a) uses the same backward propagation to update both the model and row. Here we set m
trains on the same minibatch m times
F+FGSM (Wong et al. 2020 uses a large step size and random initialization to improve FGSM. It achieves comparable performance with PGD-10, but still suffers from the robustness drop.
S+FGSM (Kim et al. 2021) uses several checkpoints to validate the inner interval of perturbation
direction to determine appropriate magnitude of the perturbation of each image.
5.3 PERFORMANCE COMPARISON
Generally, observe that PGD and ATTA achieves better performance than other methods, at the cost of significant training overheads. Meanwhile, APART achieves consistent performance improvements over FGSM-style methods, while maintaining roughly the same training speed.

---

Under review as a conference paper at ICLR 2022
Table 4: Model Performance of Pre-ResNer18 on the CIFAR-100 dataset.
Efficient
Methods
PGD-20
AA
C&W
Gaussian
Clean Data
Time/Epoch
ATTA-10
25.60%
22.90%
30.75% 33.35%
42.10%
56.20%
140 secs
x
PGD-10
28.10%
25.11 %
42.41 %
57.23%
133 secs
Free-8
25.88%
22.15%
30.55%
42.15%
55.13%
62 secs
S+FGSM
10.15%
8.91%
15.11%
50.00 %
71.10% %
61 secs
F+FGSM
25.31%
21.32%
30.06%
42.03%
57.95%
20
APART
27.56%
23.38%
32.36%
44.37%
58.40%
29 secs
Table 5: Model Performance of ResNet50 on the ImageNet dataset. + indicates single precision training.
Efficient
Methods
Clean Data
PGD-10 Attack
Time/Fpoch
ATTA-2 (Zheng et al.
2019)
60.70%
44.57%
4.85 hrs
Free-4 (Shafahi et al.
2019a)
64.44%
43.52%
3.46 hrs
F+FGSM (Wong et al.
2020
60.90%
43.46%
0.8 hrs
APART
60.52%
44.30%
hrs
Table Ablation study of APART on the CIFAR-10 dataset with Pre-ResNet20.
Training Methods
Clean Data
PGD-20 Attack
AA
APART
82.80%
51.30%
45.92%
APART (w/o layer-wise perturbation) 83.64%
47.28%
43.10%
APART (w/o perturbation initialization) 82.42%
50.10%
45.10%
Specifically, we summarize results on the CIFAR-10 in Table Table Table CIFAR-100 Table 4 and ImageNet in Table Comparing Free-8, S+FGSM, and F+FGSM, APART achieves consistent performance improvements against PGD-20, Auto/tttck,, and C&W attack for both Pre-ResNet18 and WidcResNNe33--10. As ATTA-IO and PGD-10, APART achieves slightly worse performance with 4+ times speedup. Among all methods, F+FGSM is the fastest method, and APART significantly improves the model robustness without significant computation overheads. For example, F+FGSM takes 122 secs/epoch for training WideResNer34-10 on the CIFAR-10 10 dataset, and APART takes 162 secs/epoch to achieve 7.52 absolute aceuracy improvement under the PGD-20 attack and a 4.78 absolute accuracy improvement under the AutoAttack.
is worth mentioning that, on CIFAR-10 and 100, S-FGSM has the best performance under Gaussian noise attack due to its high clean accuracy while the performances under PGD-20 and C&W attack are much poorer than other methods. This trade-off is further discussed in Section 5.4
5.4 BALANCING CLEAN ACCURACY AND ROBUST ACCURACY
As shown in Tables and 4 the robustness improvement usually comes the cost of accuracy on clean images. Meanwhile, the performances on corrupted images have consistent trends, e.g. if method A outperforms B under PGD-20 attack, A likely also outperforms B under Auto or C&W attack. To better understand the trade-off off between the clean accuracy and the model robustness, we employ different values during training (i.e., I 255
10 D.. train multiple models for each method, and visualize their performance of Pre-ResNetDl on the CIFAR-10 dataset in Figure 4 Points of APART locate in the right top corner of the figure and significantly outperform other methods. This further verifies the effectiveness of APART.
5.5 ABLATION STUDIES
APART employs two techniques to parameterize the perturbation generator. The first is to learn an initialization for the perturbation, and the second is to factorize input perturbations into series of
7

---

Under review as a conference paper at ICLR 2022
0,1755
00150
00125
Value
0.1000
30
Freatm=5)
0.0025
S+RGSM
0.0000
70
75
80
85
90
epochs
Tlean Arcurrry
Figure 3: Step sizes in different epochs (with 8 CIFAR-10)
Figure 4: Pre-ResNet18 performance on
CIFAR-10 (Ctrain 235
255
24-
W
Accureyy
value
32-
RC0-30
32
325
Indexx opocts
spochss
(a) Training Accuracy
(b) Test Accuracy w. PGD-20
(c) Perturbation Gap to PGD-20.
Figure 5: APART can alleviate the robust overfitting (note that APART is ~ 4x faster than PGD-10).
perturbations. To understand the effectiveness of them, we conduct an ablation study and summarized the results in Table 6 Removing layer-wise perturbations leads to a 4.02% drop and a 2.48% drop on accuracy under PGID-20 attack and AutoAttack respectively; Removing perturbation initialization leads to 1.20% drop and 0.82% drop on accuracy under PGD-20 attack and AutoAAtack respectively, Therefore, both techniques are helpful and necessary achieve better model robustness.
5.6 EVOLUTION OF STEP SIZES
After factorizing the input perturbation as a series of perturbations, we employ learnable step sizes to compose perturbations more effectively, To better understand these step sizes, visualize their values during the training of Pre-ResNer20 on the CIFAR-10 dataset in Figure It shows that the first-layer perturbation is important than others. Also, the step sizes of perturbation at the first and the second layers decrease after 10 epochs, while other step sizes keep increasing across the training. This phenomenon verifies that APART is able adapt the generator setting to different training stages.
5.7 ALLEVIATION OF ROBUST OVERFITTING
Besides shielding the model from catastrophic overfitting, we observe that APART can also alleviate robust overfitting, Specifically, as visualize the training curve APART and PGD-10 Figure 5 both APART and PGD-10 consistently get better training accuracy in the first 200 epochs. Meanwhile, after the first epochs, both methods suffer from robustness drop against the PGD-20 attaek. This phenomenon is referred robust overfitting, and the robust overlitting. To better understand this phenomenon, we also calculated their perturbation
observe that APART suffers less from
(0()) strength gap to PGD-20, i.e., Gram-os, (methood = C()(3) (( x x m methods v), ((( nethod x
20 (e()) ( X, v), y)- We can find that both methods get larger perturbation strength gap in the later stage of training, and APART consistently gets smaller gap. Intuitively, adaptive perturbation generator can prevents the model overfitting to a single type of perturbation, thus alleviating the robust overfitting.
8

---

Under review as a conference paper at ICLR 2022
6 RELATED WORK
6.1 ADVERSARIAL TRAINING
Goodfellow et al. (2014) first recognize the cause of the adversarial vulnerability to be the extreme nonlinearity of deep neural networks and introduced the fast gradient sign method (FGSM) to generate adversarial examples with single gradient step. Madry et al. (2017) propose iterative method based on FGSM with random starts, Projected Gradient Descent (PGD). PGD adversarial training effective but time-consuming, and thus some recent work also pays attention the efficiency of adversarial training. For example. (Saaahii et al. (2019b) propose to update both the model parameters and image perturbations using one simultaneous backward pass. Zhang et al. (2019a) show that the first layer of the neural network is more important than other layers and make the adversary computation focus more on the first layer. Zheng al. (2019) also improve the utilization of gradients to reuse perturbations across epochs. WWong et al. (2020) use uniform random initialization to improve the performance of FGSM adversarial training. APART improves the efficiency and effectiveness of adversarial training by factorizing the input perturbation as series of perturbations. Previous methods only added the perturbation to input images, while APART adds perturbation to the input of residual blocks. Perturbations added intermediate variables help improve the robustness, as discussed in Section[5.5
6.2 ROBUSTNESS DROP
Wong et al. 2020) mention the robustness drop as overfitting and first identify failure mode named as "catastrophic overfitting" which caused FGSM adversarial training to fail against PGD attacks. Rice al. (2020) further explore the overfitting in other adversarial training methods, such as PGD adversarial training and TRADES. They observe that the hest test set performance was achieved after a certain epochs and further training would lead a consistent decrease in the robust test accuracy, and therefore explain it as "robust overfitting". Rice et al. (2020) show that robustness drop is general phenomenon but they did not analyze its cause. Kim et al. (202I) asserts that catastrophic overfitting is caused by the fixed perturbation step size in single-step adversarial training, while found PGD-2 may also suffer from catastrophic overfitting, even it does not fix the perturbation step ea
size. In this work, explore the nature of robustness drop in adversarial training and further propose APART to address the perturbation deterioration issue.
CONCLUSIONS AND FUTURE WORK
In this paper, we attempt explore the mechanism behind catastrophic overfitting. As the common wisdom views the robustness drop as model overfitting, our analyses in Section present a novel perspective and suggest that the other side of catastrophic overfitting is perturbation deterioration. Guided by our analyses, we propose APART, an adaptive adversarial training framework. APART parameterizes the perturbation initialization, factorizes the input perturbation into a series of per- turbations (one for each layer in the neural networks), and progressively strengthens them during the training. our experiments, APART not only successfully shields the model from catastrophic overfitting, but also achieves consistently performance improvements while maintaining roughly the same training efficiency with FGSM-style methods.
The major limitations of our method is that it can only be applied to residual networks, and it achieves faster training at the cost of some model robustness. There are several interesting directions to pursue in future work, including applying APART to general neural models, further improve the performance of APART by further strengthing the perturbation generator. Besides, we plan to explore the underlying mechanism of other phenomenons, like the trade-off between clean accuracy and robust accuracy.
Reproducibility. In this study, we conduct experiments three public datasets, i.e., CIFAR- 10. CIFAR- 100, and ImageNet. We will release implementations for all methods and scripts for all experiments GitHub, under the Apache 2.0 license.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICMI., 2020.
Chelsea Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ArXiv, abs/1703.03400, 2017.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 201
Hoki Kim, Woojin Lee, and J. Lee. Understanding catastrophic overfitting in single-step adversarial training. In AAAI, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. ArXiv, abs/1607.02533. 2017.
Chen- Yu Lee, Saining Xie, P. W. Gallagher, Z. Zhang, and Zhuowen Tu. Deeply-supervised nets. ArXiv, abs/1409.5185, 2015.
Xuanging Liu. Minhao Cheng, Huan Zhang, and C. Hsieh. Towards robust neural networks via random self-ensemble. ECCV, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ArXiv,
2017.
P. Micikevicius, Sharan Narang, J. Alben, G. Diamos, E. Elsen, D. Gareía, B. Ginsburg, Michael Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. ArXiv, abo/1710.037400 2018.
Leslie Rice, Eric Wong, and J. Z. Kolter. Overfitting in adversarially robust deep learning. ArXiv, abs/2002.115699 2020.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! ArXiv, abs/ boss/194.11 12843, 2019a.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pp. 3353-3364, 2019b.
Leslie N. Smith. Cyclical learning rates for training neural networks. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networkstheoratical. arXiv preprint 1312v 2013.
10

---

Under review as a conference paper at ICLR 2022
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, lan Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 2017.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. ArXiv, abs/2001.0390) 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/ 0bbs/1605.07146, 2016.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu. and Bin Dong. You only propagate once: Accelcrating adversarial training via maximal principle. In NeurIPS, 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICMI., 2019b.
Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. ArXiv, abs/1912.1 1969, 2019.


Reviews:
[{"review_id": "YeJ8BI3Gts", "reviewer": "Reviewer_amVQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. It is difficult to understand that catastrophic overfitting is related to the type of perturbation. I learned that Figure 1 tries to show the phenomenon that can support the authors' claim. However, it is difficult to determine whether the FGSM-based perturbation is similar to the random noise that leads to overfitting. Or, when overfitting occurs, the model worsens, making the FGSM-based perturbation look random. Besides,  the results in this paper are similar to an existing work [1]. The work [1] also considered changing the perturbation types to overcome the overfitting.\n\n2. In Algorithm 1, why do the authors calculate the perturbation for each residual block? I think this is similar to the adversarial weight perturbation [2]. What's the difference? Furthermore, if we don't adopt the network with residual structure, what modification should be made in this place?\n\n3. If we also use this adaptive attack method during evaluations, what will the experimental results be?\n\n[1] Towards Understanding Fast Adversarial Training, arxiv. \n[2] Adversarial Weight Perturbation Helps Robust Generalization. ", "summary_of_the_paper": "This paper focuses on an emerging topic in adversarial training, which is about catastrophic overfitting. First, the authors observe that such overfitting is related to the limited types of perturbations. So, the authors introduce an adaptive adversarial training method, called APART, which adjusts the parameters of the perturbation generator. The experiments can verify that the APART can achieve competitive results. ", "summary_of_the_review": "The observation results lack reasonable verification. And, the evaluation of adaptive attack is missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V_w0GMX54N", "reviewer": "Reviewer_Z8eC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths (motivation, correctness, clarity):\n\nWriting: The paper is well motivated, its writing is clear and easy to understand.\n\nObservation: The observation that the abrupt drop in robustness may be due to perturbation degrading to random noise (drastic distribution shift) is new, to my best knowledge. I think this is an interesting empirical observation.\n\nMethodology: The proposed APART algorithm is sensible, with reasonable performance while being computationally advantageous. I appreciate that the authors performed an ablation study.\n\nExperiments: The experiments validated the paper's claims. Although their methods don't outperform the stronger methods, they are much faster than them. The paper nicely adds to practitioners' toolbox between the two spectrums, from being fast to accurate.\n\n\nWeakness (in depth analysis, claim rigor):\n\nIn depth analysis:\nWhile the paper may be (among) the first that performs a thorough study of such co-occurrences, the intuition where the generated adversarial samples being too weak, can cause training failure, is not new in differentiable games. In GAN training, discriminator being too strong may lead to vanishing gradient for the generator and other phenomena [1] [2]. I think such related works (maybe ones that are more immediately relevant to adversarial training) can be discussed to better motivate the current work. The similarities and differences may be interesting to compare. The present work can benefit additionally by examining/hypothesizing how the random noise like perturbation suddenly occurred. \n\nClaim rigor/theoretical discussion:\n- As stated in my summary, adversarial training's learning (theoretic) framework differs from those in empirical risk minimization. Since a major contribution of the paper is that the catastrophic overfitting and perturbation deterioration are highly correlated, the readers can benefit from a more formal comparison of the identified learning, to IID learning, and adversarial learning when the perturbation distribution doesn't change too much (degrading to random noise). Note I'm not suggesting the authors to prove theorems, etc., just a more formal framing.\n\n- In section 3.1, \"not only look like random noise, but also behaves like random noise\" - I'd suggest the authors perform some statistical testing to validate this claim.\n\n---\nAfter seeing others' reviews, while I do believe the main writing it clear, the authors may fail to justify the most crucial part of their contribution (how perturbation distribution and robustness are linked).\n\n\n[1] Towards Principled Methods for Training Generative Adversarial Networks\n[2] Unbalanced GANs: Pre-training the Generator of Generative Adversarial Network using Variational Autoencoder", "summary_of_the_paper": "The paper made two contributions: 1. it identified the co-occurrence of abrupt robustness drop and perturbation degrading to (potentially) random noise; 2. it proposed a computationally faster adversarial training algorithm while attaining a good accuracy. Point 1. is interesting: it suggests previously identified \"catastrophic overfitting\" may be due to the perturbation in training time fails to mimic those in test time in distribution, while the adversarial training is being performed. This is different from classical empirical risk optimizations, where the training data do not depend on the training processes. ", "summary_of_the_review": "Overall, I believe this paper merits publication. The only drawbacks I can see are the lack of discussion and some minor claims being not fully rigorous. I'd raise my rating if the author can give a more in depth discussion and analysis on the relation between adversarial robustness drop and changes in perturbation distribution.\n\n---\nAfter seeing other reviews, I think clarifying the learning theoretic framework in the current work is necessary, otherwise, I'd lower my score. It is unclear whether the authors explicitly thought through this point. It can be my biased and overly optimistic interpretation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L3sVn6pvFgk", "reviewer": "Reviewer_996w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. The proposed method is simple yet effective with respect to the balance between generalization performance and computational costs. \n\n2. Author analyzes the behavior of FGSM-style adversarial training and proposes a method from the lens of perturbation deterioration. \n\n**Weaknesses**\n\n1. As clearly described in the abstract, this paper mainly shows that “at the same epoch when the FGSM-trained model catastrophically overfits, it generated perturbations deteriorate into random noise”. However, except the causal direction, this claim is trivial since a poorly performed model returns random prediction and one does not need strong perturbation to change the prediction. \nBesides, it seems the paper assume that the deteriorate perturbation *cause* catastrophic overfitting. For example, in abstract they claim “Intuitively, once the generated perturbations become weak and inadequate, models would be misguided to overfit those weak attacks\nand fail to defend strong ones”. However, if the perturbation becomes simple random noise, it can not cause catastrophic overfitting since it means the adversarial training becomes standard supervised learning. \n\n2. This paper compares APART with several existing computationally efficient methods (free-8, S+FSGM, and F+FSGM), however, connection to these works is not investigated. Especially, it is not clear whether the success of prior works can be seen as the prevention of perturbation deterioration or not as with the APART does. \n\n3. Strengthen the perturbation using parametric adversarial examples is simple and nice. However, this paper lacks several prior works that use different parameterization to generate adversarial examples [1,2]. Connection to these works should be discussed. \n\n[1] Baluja, Shumeet and Ian S. Fischer. “Adversarial Transformation Networks: Learning to Generate Adversarial Examples.” (2017)\n[2] Xiao, Chaowei et al. “Generating Adversarial Examples with Adversarial Networks.” (2018)\n\n4. There is no theoretical analysis, and therefore this is basically an empirical paper and requires strong empirical results to be accepted.  However, the performance improvement over prior computationally effective methods seems to be incremental in many settings (e.g. except PGD-20 in Cifar10). Besides, the discussion on the performance improvement is not enough. \nMinor comments\nI can not understand the meaning of Fig 1-a, as the color bar is grayscale yet the content is RGB. \nEq. 3 should maximize omega_alpha, not omega. \n", "summary_of_the_paper": "This paper focuses on stable and effective adversarial training and improving generalization using it. This paper first investigates the phenomenon called catastrophic overfits in adversarial training, and provides a new hypothesis that the phenomenon occurs since the generated perturbations become too weak. Based on the hypothesis, this paper proposes APART, which parametrizes the adversarial example and trains it using gradient accent to prevent perturbation deterioration. Using several standard benchmark datasets, they show that APART prevents catastrophic overfitting and provides comparable performance with several existing methods (including PGD, Free-8, and variants of FGSM) in terms of generalization performance and computational costs. \n", "summary_of_the_review": "Overall, I found the proposed method potentially interesting for fields in adversarial examples. However, the paper needs relatively large revision to fully support the claim and elaborate the novelty and significance of the work. The experimental results are ok but not strong enough to be accepted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Br5Wv8aD1C", "reviewer": "Reviewer_Bmpq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Weaknesses and Questions** \n\n1. The key argument in the paper for catastrophic overfitting is the deterioration of the augmented adversaries during the training time. However, while this has been supported with several empirical experiments, this was already known in literature. For example, the work by Hoki Kim et al (AAAI21) showed that the loss surface when catastrophic overfitting happens deviates from being linear. Since, FGSM is a closed form solution to the linearized loss, i.e. $\\max_{\\delta \\in \\mathcal{S}} \\mathcal{L}(x,y) + \\delta^\\top \\nabla_x \\mathcal{L}(x,y)$ where $\\mathcal{S}$ is the $\\ell_\\infty$ ball, this implies that the locally the loss function is far from linear. Therefore, the solution of FGSM very crudely approximates the loss surface and thereafter provides a worse approximation to the worst case adversary (original problem $\\max_{\\delta \\in \\mathcal{S}} \\mathcal{L}(x+\\delta,y)$). That is to say, FGSM solutions gets worse equivalently adversaries are deteriorating. To that end, I do not find the empirical experiments conducted in Figures 1 and 2 to provide any new insights that were not known before. It is merely a re-statement to what we already know about catastrophic overfitting. To that end, I do not find the first key component of this paper to be novel or insightful (the part on perturbation deterioration). In fact, this observation was the key element for the Hoki et al AAAI21 work of using stronger adversaries when the linearization as an approximation is worse.\n\n2. The other key contribution to this paper is the algorithmic part. The paper does not report robust accuracy under larger $\\epsilon$. As stated by the paper in section 5.2, F-FGSM, still suffers from catastrophic overfitting. This has also been observed by Gradalign \"Understanding and Improving Fast Adversarial Training\". To see this, one needs to evaluate F-FGSM trained models against a larger $\\epsilon$, e.g. $\\epsilon \\ge 10/255$ (see Figure 1 left in their paper). This demonstrated that adding random noise in F-FGSM only helped against catastrophic overfitting for some choices of evaluated attack $\\epsilon$. To that end, it is essential for this paper to conduct similar experiments as reported in Figure 1 of Andriushchenko et al to confirm whether catastrophic overfitting has been evaded for this paper. \n\n3. I find it hard to understand the motivation of the per layer adversary perturbations. Can the authors comment, or maybe justify empirically on to why do one expect to improve against input adversary by augmenting per layer input adversaries? \n\n4. The key motivation for the algorithmic contribution stems from the fact that one needs to adaptively control the strength of the adversaries during training. To elleviate this difficulty, the paper proposes to learn the step size along with the initialization for the FGSM step. I find the technical contribution here to be minimal. This replaces one problem with another. Instead of selecting the step size $\\alpha$ for FGSM, this procedure is replaced to selecting a learning schedule for the optimzier learning $\\alpha$. Thus, a similar nuisance to FGSM can occur where a choice of $\\alpha$ could lead to catastrophic overfitting, a choice of scheduler for $\\alpha$ could lead to a catastrophic overfitting. Thus, since there are no guarantees and that the key element here is replacing the tuning of a scalar to tuning a scheduler is not considered as a major contribution. I may be wrong or missing something here. I hope authors correct me on this.\n\n\n5. Since the key contribution is the learning procedure (adaptive part) of adversary augmentation during training, I believe most of the experiments should have been conducted where only the adaptive element is presented at the input level. As a plus, one then considers that less motivated variant of per layer adaptive augmentation of adversaries. Currently, even with Table 6, it is very hard to understand whether there is enough improvement with the main claim of the paper. This is specifically that it seems that replacing a constant step size with a scheduler can actually still be worse that simple F-FGSM in complexity with almost similar robust accuracy.\n\n\n\n\n**Algorithmic Typo?**\nIn Algorithm 1, are the updates in Lines 4 and 8 correct. This seems to be a constant function equal to $\\epsilon$ and $1$, respectively, agnostically from the gradient step. I believe the paper meant to switch the order of max and min or the $-\\epsilon$ and $\\epsilon$ in min max operators and equivalently the $-1$ and $1$ for Line 8.\n\n**Minor Comments**\n1. Page 2, Line 5 just below Figure 1: \"APART improve the initialization\" >> \"Improves\".\n2. Section 4, Page 4, Line 4: \"Moreover, it employ\" >> \"employs\".\n3. Section 5.7, Line 2: \"Specifically, as visualize the training\" >> \"we visualize ..\".\n4. Consider making the legend of Figure 5(c) bigger. It is currently hard to read.\n5. Missing citation: \"Understanding and Improving Fast Adversarial Training\".\n\n", "summary_of_the_paper": "**Summary**\n\nThe paper address the issue of catastrophic overfitting in robust training. Robust overfitting is often attributed to the fact that robust training tends at some point during the training to overfit to the underlying threat model augmenting the adversaries. This cases the test robust accuracy against the augmented training adversaries to be high while achieving 0% robust accuracy against stronger threat models. The paper provides new insights to potential other reasons to this phenomena. In particular, the paper argues that that at the same epoch when catastrophic overfitting happens, the quality of the augmented adversaries deteriorates fast and is no stronger than a random noise agumentation. The paper then goes about resolving this issue by strengthening the augmented adversaries during training by learning the hyperparameters of FGSM adversaries, namely the step size and the initialization. In addition, the paper augments the adversaries per Residual block, i.e. finding FGSM attacks per layer input. This approach is titled APART and shows competitive performance compared against existing methods in both final robust accuracy and training time efficiency. Experiments are conducted on CIFAR10, CIFAR100 and ImageNet datasets.", "summary_of_the_review": "See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the Catastrophic Overfitting problem of adversarial training of FGSM. One reviewer gave a score of 6 and the other three reviewers gave negative scores. The authors failed to address or clarify (no rebuttal provided) how perturbation distribution and robustness are linked (four reviewers all agree on this). Other issues include unclear motivation, limited experiments validation, and lack of theoretical analysis. Thus, the current version of the paper cannot be accepted to ICLR.
Decision: Reject