I-PGD-AT: Efficient Adversarial Training via Imitating Iterative PGD Attack 
Authors: Xiaosen Wang
Bhavya Kailkhura
Krishnaram Kenthapadi
Bo Li
Venue: ICLR 2022 Submitted (2022)
Keywords: Single-step Adversarial Training
Catastrophic Overfitting
Adversarial Robustness
Adversarial Example

Abstract: Adversarial training has been widely used in various machine learning paradigms to improve the robustness; while it would increase the training cost due to the perturbation optimization process. To improve the efficiency, recent studies leverage Fast Gradient Sign Method with Random Start (FGSM-RS) for adversarial training. However, such methods would lead to relatively low robustness and catastrophic overfitting, which means the robustness against iterative attacks (e.g. Projected Gradient Descent (PGD)) would suddenly drop to 0%. Different approaches have been proposed to address this problem, while later studies show that catastrophic overfitting still remains. In this paper, motivated by the fact that expensive iterative adversarial training methods achieve high robustness without catastrophic overfitting, we aim to ask: Can we perform iterative adversarial training in an efficient way? To this end, we first analyze the difference of perturbation generated by FGSM-RS and PGD and find that PGD tends to craft diverse discrete values instead of $\pm 1$ in FGSM-RS. Based on this observation, we propose an efficient single-step adversarial training method I-PGD-AT by adopting I-PGD attack for training, in which I-PGD imitates PGD virtually. Unlike FGSM that crafts the perturbation directly using the sign of gradient, I-PGD imitates the perturbation of PGD based on the magnitude of gradient. Extensive empirical evaluations on CIFAR-10 and Tiny ImageNet demonstrate that our I-PGD-AT can improve the robustness compared with the baselines and significantly delay catastrophic overfitting. Moreover, we explore and discuss the factors that affect catastrophic overfitting. Finally, to demonstrate the generality of I-PGD-AT, we integrate it into PGD adversarial training and show that it can even further improve the robustness.

Under review as a conference paper at ICLR 2022
I-PGD-AT: EFFICIENT ADVERSARIAL TRAINING VIA IMITATING ITERATIVE PGD ATTACK
Anonymous authors Paper under double-blind review
ABSTRACT
To improve the efficiency of adversarial training, recent studies leverage Fast Gradient Sign Method with Random Start (FGSM-RS) for adversarial training. Unfortunately, such methods lead to relatively low robustness and catastrophic overfitting, which means that the robustness against iterative attacks (e.g. Projected Gradient Descent (PGD)) would suddenly drop to 0%. Different approaches have been proposed to address this problem, however, recent studies show that catastrophic overfitting still remains. In this paper, motivated by the fact that expensive iterative adversarial training methods achieve high robustness without catastrophic overfitting. we ask: Can we perform iterative adversarial training in an efficient way? To this end, we first analyze the differences in perturbations generated by FGSM-RS and PGD and find that PGD tends craft diverse discrete values instead of +1 in FGSM-RS. Based on this observation, we propose an efficient single-step adversarial training method I-PGD-AT by leveraging I-PGD attack, in which I-PGD imitates PGD virtually. Unlike FGSM that crafts the perturbations directly using the sign of gradient, I-PGD imitates the perturbation of PGD based on the magnitude of gradient. Extensive empirical evaluations on CIFAR-10 and Tiny ImageNet demonstrate that our I-PGD-AT can improve the robustness compared with the baselines and significantly delay catastrophic overfitting. Moreover, we explore and discuss the factors that affect catastrophic overfitting. Finally, demonstrate the generality of I-PGD-AT, we integrate it into PGD adversarial training and show that it can even further improve the robustness.
INTRODUCTION
The breakthroughs in Deep Neural Networks (DNNs) have brought unprecedented boom to various fields, such as Computer Vision (Krizhevsky et al. 2012 He et al., 2016) and Natural Language Processing (Devlin et al. 2019). However, recent studies have revealed that DNNs are vulnerable to adversarial examples, such that imperceptible perturbations with small magnitude added to the input sufficient to mislead the target model (Szegedy et al.) 2014: Goodfellow et al. [001 Xiao et al.) 2018a b Bhattad et al. 2020). Such perturbations are often transferable across different models (Liu al.] 2017 Dong et al.
2018 Wang & He 2021) and can be generated without the knowledge of the target model under black- box setting (Brendel et al. 2018 Li et al. 2020b 2021).
Several works have been proposed mitigate the aforementioned problem and improve the robustness of DNNs, e.g. adversarial training (Goodfellow et al. 2015; Madry et al. 2018). 8), input transforma-
tions (Xie et al. 20188] Nascer et al. 2020), certified defenses (Raghunathan et al. 2018 Gowal et al.) 2019) etc. Among these methods, adversarial training is one of the most effective empirical defense methods (Athalye et al. 2018). Adversarial training can be formulated as a minimax optimization problem (Madry et al. 2018), in which the inner maximization aims to search for an adversarial example that maximizes the classification loss, while the outer minimization aims to train a robust classifier against the worst-case adversarial perturbations. Projected Gradient Descent Adversarial Training (PGD-AT) (Madry et al. (2018) approximately solves the inner maximization problem by multiple steps of projected gradient descent and achieves empirical robustness (Carlini & Wagner 2017: Athalye et al. 20188] Croce & Hein 2020b). However, PGD-AT is much more expensive than standard training, owing to the iterative optimization process of perturbation generation.
To reduce the computational burden, Wong et al. (2020) propose Fast Adversarial Training (Fast-AT), which adopts Fast Gradient Sign Method with Random Start (FGSM-RS) for training and leads to

---

Under review as a conference paper at ICLR 2022
robust models. However, there are two limitations for Fast-AT: a) Fast-AT suffers from catastrophic overfirting, i.e. the robustmess against iterative attack (e.g. PGD) suddenly drops after a few training epochs. b) There is still a gap between the robustness of Fast-AT and PGD-AT. Fast-AT directly adopts early stopping to prevent catastrophic overfitting. Several attempts (Andriushchenko & Flammarion 2020 (S. & Babu 2020 Sriramanan et al. 2020 Kim et al.] )021) are made avoid catastrophic overfitting, but these approaches either are computationally inefficient or hurt the robustness.
In this work, motivated by the benefits of PGD-AT, we propose I-PGD Adversarial Training (I-PGD- AT), to improve the robustness of Fast-AT with the same computational cost and delay catastrophic overfitting. In particular, we first analyze the differences in perturbations generated by FGSM-RS and PGD and find that the perturbation of PGD is more diverse in which the perturbations are different discrete values instead of 11 in FGSM-RS. Based on this observation, we imitate the perturbation of PGD virtually via a single gradient calculation. Specifically, we first calculate all the candidate discrete values for PGD attack (e.g. O and +2 for two-step PGD) and its corresponding probability. Then we generate new gradient vector by filling the cells with the candidate discrete value based on the corresponding absolute gradient while keeping the original sign information. Unlike FGSM-RS which directly crafts adversarial perturbation using the sign of gradient, we leverage this generated vector with more diverse discrete values to generate adversarial perturbation for efficient training.
We conduct extensive empirical evaluations CIFAR-10 and Tiny ImageNet datasets, which demonstrate that I-PGD-AT can significantly improve the robustness against various adversarial attacks. We also show that I-PGD-AT can effectively stabilize the training and delay catastrophic overfitting compared with Fast-AT. Furthermore, we empirically validate our hypothesis that under the same -norm perturbation constraint, training the model using perturbation with a smaller mean absolute value can delay catastrophic overfitting, which could provide new insights to address such issues. Additionally, we extend our imitation strategy for multi-iteration, which achieves substantial improvements on the robustness compared with PGD-AT with different iterations, and delays the robust overfitting (Rice et al. 2020 PGD-AT.
2 RELATED WORK
To mitigate the threat of adversarial examples, large variety of defense method has been proposed.
including adversarial training (Goodfellow et al. 2015 Madry et al. 2018). input transformations (Xie
et al.
2018 Guo et al.) 2018 Liu et al. 2019)) denoising (Liao et al. 2018 Mustafa et al. 2019
Naseer ct al. 2020). certified defense (Raghunathan et al.] 2018 CCohen et al. |20110 Gowal et al.
2019 Li al. [2099). Among them, PGD-AT (Madry al. 2018 the most popular adversarial
training method, brings consistent improvements on robustness against various attacks and inspires
several adversarial training methods (Zhang et al. 2019b; Ding et al. 2020; Wang et al. 2020 Song et al. 2020; Dong et al. 2020). Furthermore, Rice et al. (2020) identify robust overfitting in PGD-AT and find that early stopping or data augmentation (Tack al. 2021) helps PGD-AT achieve comparable robustness with advanced adversarial training methods (Zhang et al. 2019b).
However, PGD-AT is computationally inefficient due to the iterative optimization for adversarial examples. To combat the computational overhead, recent works (Shafahi et al.] 2019 Zhang et al. 2019a) utilize single backpropagation for both training and PGD adversary generation to accelerate the training process. Wong et al. (2020) identify catastrophic overfitting in single-step adversarial training and propose Fast Adversarial Training (Fast-AT) for efficient training. To avoid catastrophic overfitting, [Andriushchenko & Flammarion (2020) propose GradAlign, which maximizes the gradient alignment inside the perturbation set to strengthen the model's ' local linearity. However, it leads to 3x slowdown due to the double backpropagation. Kim et al. (2021) identify decision boundary distortion after catastrophic overfitting, in which a smaller perturbation is sufficient to fool the model, while the model is robust against larger perturbation used in Fast-AT. They further propose to search appropriate step size for each input sample to avoid catastrophic overfitting, but it also hurts the robustness performance. Improved fast AT (FastAdv+) (Li al. 2020al incorporates PGD-AT into Fast-AT when observing the catastrophic overfitting to stabilize the training. Chen et al. (2020) propose backward smoothing which adopts a pack-propogation for initialization to implement single-step training with the objtective of TRADES (Zhang et al. 2019b).
In this work, we identify the differences in perturbations generated by PGD and FGSM-RS and propose single-step adversarial training method I-PGD-AT to boost the robustness of models and delay catastrophic overfitting effectively.
2

---

Under review as a conference paper at ICLR 2022
3 METHODOLOGY
In this section, we first provide preliminaries and revisit FGSM-RS and PGD to identify the differences in perturbations generated by them. Then provide a detailed description of the proposed I-PGD Adversarial Training (I-PGD-AT) and discussion on catastrophic overfitting.
3.1 PRELIMINARIES
Let (z, y, 0) denote the loss function of deep neural net f with parameters 0 on the input (xr, y) drawn from data distribution D. Adversarial training can be formulated as a minimax optimization problem (Madry al. 2018):
min ((r.)) mmx (x [x 8, y) AEA
(1)
where A the perturbation set. In this paper, we focus on foo attack align with previous work, i.e. A = {ō E TRd || < c). where is the maximum magnitude of the perturbation. This minimax problem usually solved by firstly crafting adversarial examples to solve the inner maximization problem and then optimizing the model parameters 0 using the generated adversarial examples.
Fast Gradient Sign Method Adversarial Training (FGSM-AT) (Goodfellow et al. 2015) the first adversarial training method that solves the inner maximization problem using one-step update:
sign (a, y: 0)).
However, FGSM-AT exhibits limited robustness against iterative attacks (e.g. iterative FGSM (Ku- rakin et al. 2017). Later, Madry et al. (2018) propose Project Gradient Descent Adversarial Training (PGD-AT), which iteratively solves the inner maximization problem using PGD:
8,41 1L (6g + a sign(V..J + dz, y, 0))),
where 6, is the adversarial perturbation at the t-th iteration and 6, ~ U(-E, e), (-) is the projection function and is the step size.
Although PGD-AT is empirically robust, compared with FGSM-AT, the inner loop of PGD to craft adversarial examples very expensive. Further, Wong et (2020) identify that FGSM-AT suffers from the catastrophic overfitting phenomenon, and propose Fast Adversarial Training (Fast-AT) using FGSM adversarial example with random start (FGSM-RS), i.e., P u(=e,e):
å 1I (p + a ,. p.v. 0)).
(4)
Fast-AT adopts early stopping to avoid catastrophic overfitting.
3.2 REVISITING FGSM-RS AND PGD
Let 8' IRd denote the perturbation obtained by gradient ascent and g, denote the gradient i-th iteration calculated by the corresponding adversarial attack. Suppose that the input x is normalized into [0, 1). for existing gradient- adversarial attacks, the generated adversarial example Tade can be formulated as:
Kade 11 (x: + (p + +')),
(5)
where p the initialization of perturbation (e.g. P for FGSM and p P~ (d(-=e)) for FGSM-RS and PGD). We could reformulate the final perturbation as 5 = [P 5').
For FGSM-RS, we could easily obtain 8raSM from Eq. For PGDk where is the number of iteration of PGD. we can approximately obtain that:
a sign (Vgn(V_Jee 2J + +. a (ign(
6 lp + 8*)
II (ōx-1 + sign(gh-1))
= II (IIA (6x-2 + a sign(gk-2)) + a sign(gh-1)
II (II (.. Ap + sign(go))) sign(ok-1))
(6)
Ha (p a Σ sign(g:)),
where g = VzJ(r= + 8g. 0). By ignoring the projection operator at each iteration as the step size is expected be small w.r.t. the input, and that 8PGDE 2 Ek-1 sign(g:) for PGDI.
3

---

Under review as a conference paper at ICLR 2022
222
22
T2
22
FGSM-RS
PGD2
PGD3
PGD4
The candidate perturbation 8' (red dots) generated by gradient ascent in various attacks, namely FGSM-RS and PGDK, where is denotes the number of iteration for PGD. For simplicity, normalize the perturbation into [-1, 1] and do not consider the hyper-parameter a.
Figure 1:
Since FGSM-RS and PGD adopt the same random initialization, we only focus on the difference of 8'. For simplicity, we treat the step size a as a scaled factor and do not consider it by normalizing the accumulated sign 8 into [1].
Suppose the input sample side length centered at the origin. Note that the output sign(-) is either or 11 As shown in Fig. 1 for FGSM-RS, PFGSM-RS would be one of the four vertices of the square. In contrast, taking PGD2 for example, there are four cases (+1,+1), (+1,-1) for each pixel I2 E z, which lead to three different accumulated signs AS(2) {+2, 0, -2]. The general accumulated sign set for PGDk can be formulated as:
(æ1, [2) E [0, 1J2, the perturbation set for 8' would be a square with
(-1, +1}, (-1, -1) of the sign
(7) We show the search space for PGD2, PGD3 and PGD4 in Fig. and we can see that the larger is. the more diverse the search space for PGDk is. From the decision boundary distortion perspective (Kim et al. 2021), it supports that PGDk can find more diverse adversarial examples inside the perturbation set so that it does not lead to catastrophic overfitting. Such difference of perturbation generated by FGSM-RS and PGD motivates us to consider the following question:
AS( = (2i + k%2) 0 OSiS [k/23
Could we imitate the diverse discrete values of the perturbation generated by PGDk with single gradient calculation to improve the robustness of single-step adversarial training and mitigate the catastrophic overfitting?
3.3 I-PGD ADVERSARIAL TRAINING
From Sec. 3.2, we find that FGSM-RS directly reaches the boundary of perturbation set, which is one possible reason for catastrophic overfitting (Kim et al. 2021) in Fast-AT that performs adversarial training with perturbation generated by FSGM-RS, while PGD can generate diverse perturbation inside the boundary. To some extent, adopting random initialization which leads to more diverse perturbation, can mitigate such phenomenon. Apart from catastrophic overfitting, the robustness is also weaker for Fast-AT compared with PGD-AT. Considering the properties of perturbation generated by PGD, we aim to explore how to improve the robustness of Fast-AT and mitigate catastrophic overfitting by imitating the perturbation of PGD.
It is noted that both FGSM-RS and PGD adopt the sign of the gradient without considering the relative magnitude of each element the gradient vector E Rd.. Due to the multiple iterations, PGD leads to diverse but still discrete candidate perturbation for 8'. In order to imitate the diverse values of PGD by single gradient calculation, we need to generate a new imitative gradient vector ğ E Rd for crafting adversarial perturbation by additionally utilizing the magnitude of gradient. Generally speaking, to achieve this goal, there are two issues to address.
What discrete values does contain? To imitate specific PGDA attack, the accumulated sign set from Eq, 7 For instance, if A 2, we have three values (-2, 0, +2} which can be assigned each element. For any k the candidate values are symmetric regarding the origin. Thus, we only need to consider the absolute values for simplicity, i.e. we only have {0, 2) for PGD2, and the assigned values keep the same sign with the corresponding gradient dimensions.
effectively calculate
Which value do we assign to the element g After determining what values does ğ contain, we assign the value each element to obtain by utilizing the relative magnitude of the gradient g.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 The general algorithm for I-PGD Adversarial Training
Input: Training data D, loss function (z, y. 0). training epochs T, magnitude of perturbation adversarial
step size learning rate 7), imitative values and corresponding portions I = {u, p.)
I: Initialize model parameters
2: for t- = 1 + 'I' do
3:
for each batch (I. v) C D do
4:
5 8= c)
5:
g g= V.J(( + 6, y, 0), 9 =
Calculate quantile Q on [g] based on the portions ps E I
7:
for i-th E Q do
> p. elements in [g| are smaller than q:
8:
3(9--1 < 19l S 4: end for
9:
10:
0 / sign (g)
> 9 share the same sign of g
11:
5 (8 a 9)
5 [6 + a sign(g)) for FGSM-RS
12:
0=
JoJ (xr 6, 0.
13:
end for
14: end for
We only need consider the absolute value. The smaller [g] is, the smaller value is assigned to [ālit]. Specifically, we first calculate the probability of each possible value for PGDK. Let p denote the probability that specific element obtains the same sign (i.e. +1,+1 in two consecutive iterations, and we assume that > 1/2 due to the neighborhood constraint by adversarial attacks. We can calculate the probability for each value by summing the probability for each possible sequence of sign. For a given gradient g. we can calculate the quantile on [g| according to the accumulated probability P[] S v) for each candidate value v and set all the elements in this quantile range with value e' to obtain for adversarial training. We denote adopting for attack as I-PGD attack and such adversarial training method as I-PGD Adversarial Training (I-PGD-TT) Taking PGD2 for example,
or -I.
could get that the imitative value is O and 2 with the probability P([8' [1] = 0) = p, [[] = 2) P. Le. = {0 p, 2 1). We provide the detailed calculation for I = {v; p.} in Appendix A.1 and overall algorithm of I-PGD-AT in Algorithm
3.4 DISCUSSION ON CATASTROPHIC OVERFITTING
The major difference between FGSM-RS and I-PGD attack is that for each input sample, FGSM-RS generates the adversarial example only using the sign of gradient, while I-PGD adopts the imitative gradient vector of PGD craft adversarial example. Since PGD-AT does not lead to catastrophic overfitting and our I-PGD-AT utilizes I-PGD for adversarial training, we wonder if such imitation could effectively avoid catastrophic overfitting.
Unfortunately, as discussed in Sec. 4.5 we find that catastrophic overfitting still occurs in I-PGD-AT. However, compared with Fast-AT, I-PGD-AT can stabilize the training process and delay catastrophic overfitting effectively. We further explore the relationship between the magnitude of perturbation and catastrophic overfitting and validate a new hypothesis that under the same E noom constraint on the perturbation, the perturbation with a smaller mean absolute value can effectively delay catastrophic overfitting This hypothesis sheds new light on how to avoid catastrophic overfitting in single-step adversarial training effectively.
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets and Models. We conduct empirical evaluations on CIFAR-10 (Krizhevsky 2009) and Tiny ImageNe using PreAct ResNet-18 18 (He et al. 2016). All the images are normalized into [0, 1].
Baselines. To verify the effectiveness of I-PGD-AT, we imitate PGD2 and PGD3 dubbed I-PGD2-AT and I-PGD3-AT with single-step adversarial training, respectively. We compare them with Free- AT Shafahi et al. 2019) and four single-step adversarial training methods. i.e. Fast-AT (Wong
ct al..1 2020), GradAlign (Andriushchenko & Flammarion 2020), Kim et al. (Kim et al. 202 and
5

---

Under review as a conference paper at ICLR 2022
Table 1: Classification accuracy (%) and training time (min) of various single-step adversarial training methods and the proposed methods I-PGDD-AT and I-PGD3-AT on CIFAR-10 and Tiny ImageNet against white-box attacks 8/255). We bold the highest classification accuracy and lowest
training time which outperforms the runner-up by at least 0.2. The proposed methods are in gray
(a) Evaluations on CIFAR-10.
Method
Clean
FGSM
PGD20
PGD50
MIM
CW20
CW50
AA
Time(min)
Free-AT
78.75
71.89
45.57
45.34
45.85
43.60
43.51
41.55
120.8
Fast-AT
83.27
55.31
46.85
46.28
47.82
46.56
46.31
43.28
26.0
GradAlign
81.24
54.55
47.48
47.18
48.00
46.91
46.69
44.09
81.4
Kim et al.
88.91
51.60
37.71
37.25
38.79
39.22
38.90
36.07
29.2
FastAdv+
83.17
76.68
47.10
46.69
47.48
46.40
46.22
43.50
230.1
I-PGD2-AT
81.41
55.31
48.26
47.96
49.32
47.19
46.95
44.47
26.2
I-PGD3-AT
81.42
55.39
48.82
48.34
49.82
47.61
47.25
44.70
26.2
(b) Evaluations on Tiny ImageNet.
Method
Clean
FGSM
PGD20
PGD50
MIM
CW20
CW50
AA
Time(min)
Fast-AT
46.80
21.18
17.82
17.56
17.88
16.16
16.10
14.11
114.9
Kim et al.
56.21
16.62
9.32
9.03
9.70
10.86
10.64
8.10
130.2
I-PGD2-AT
45.53
21.84
18.57
18.37
18.71
16.76
16.64
14.55
115.0
I-PGD3-AT
45.55
21.82
18.56
18.44
18.64
16.84
16.81
14.79
115.0
FastAdv+ Li et al. 2020a). We also extend I-PGD into PGD2-AT and PGD7-AT to imitate more powerful iterative adversarial training and compare them with the corresponding PGD-AT methods.
Training Details. We use SGD optimizer with momentum of 0.9 and decay factor of 5 x 10-1, and adopt the perturbation budget = 8/255 for adversarial training. For single-step adversarial training. we train the model with 30 epochs on CIFAR-10 and 40 epochs on Tiny ImageNet, using cyclic learning rate (Smith) 2017) with the maximum learning rate of 0.2 and increasing until the half of the training epochs. For PGDI-AT, we train the model with 200 epochs using the maximum learning rate of 0.2, which decays to 0.02 50-th epoch. The probability p set to 2/3 and a c for I-PGD-AT.
Evaluations. We evaluate the robustness using various attacks, i.e. FGSM (Goodfellow et al. (015). PGD (Madry et al., 2), Momentum Iterative Method (MIM) (Dong et al. 2018), Carlini-Wagne (CW) (Carlini & Wagner (017) and AutoAttack (AA) (Croce & Hein 2020b). More details about the experimental setting are summarized Appendix A.3
4.2 EVALUATION ON I-PGD-AT
To validate the effectiveness of the proposed I-PGD-AT, we empirically evaluate the robustness of I-PGD2-AT and I-PG which performs the single-step training to imitate PGD with different steps for perturbation optimization CIFAR-10 and Tiny ImageNet.
Results on CIFAR-10. As Fast-AT may lead to catastrophic overfitting after 24-th epoch on CIFAR- 10 (Andriushchenko & Flammarion 2020), train Fast-AT multiple times and provide the best results. As shown Table Taj Kim et al. achieves the lowest robustness against various attacks but higher clean accuracy due to the smaller generated perturbation average. Fast-AT exhibits better robustness than Free-AT while GradAlign and FastAdy+ achieve higher robustness than Fast-AT and do lead to catastrophic overfitting. Our proposed I-PGD2-AT could stabilize the training process and achieve the highest robustness compared with the three baselines, without catastrophic overfitting during the training. It is noted that I-PGD3-AT. which only adopts different imitative values and corresponding portions (i.e. {1 5/9,3 3 4/9} for I-PGD3-AT and {0- 1/3, 2/3} for I-PGD2-AT), exhibits higher performance than I-PGI2-AT with the same computational cost. It further shows the reasonability and effectiveness of the proposed imitation strategy. As for the comparison of training time, although GradAlign improves the robustness, leads to roughly 3x slowdown compared with Fast-AT. Free-AT takes smaller computational cost for each epoch but larger training epochs, leading to longer training time. FastAdy+ takes much longer times due to the larger training epochs and using PGD10 recover catastrophic overfitting. Our proposed methods take about the same training time as Fast-AT since we only process the calculated gradient with constant time complexity, and achieves the highest robustness.

---

Under review as a conference paper at ICLR 2022
Table Classification accuracy (%) and training time (hour) of PGD-AT w/ and w/o the proposed I-PGD on CIFAR-10 against white- box attacks (c 8/255) without early stopping. We bold the highest classification accuracy and lowest training time which outperforms the runner-up by least
0.5 and mark the results of the proposed method in gray
Method
Clean
FGSM
PGD20
PGD50
CW20
CW50
MIM
AA
Time(h)
PGD2-AT
84.78
50.28
39.31
38.64
40.78
39.58
39.22
36.52
4.3
PGD4-AT
84.83
50.32
39.91
39.42
41.29
42.20
39.90
37.12
7.0
I-PGD2-TTpaD2
82.84
52.13
44.06
43.59
45.09
43.14
42.97
40,55
4.4
PGD7-AT
82.27
53.17
45.89
45.41
46.88
44.74
44.55
42.23
11.1
PGDI4-AT
82.02
53.78
47.07
46.76
48.01
45,47
45.29
43.01
20.5
I-GDD-ATPPDD
82.32
53.17
46.67
46.28
47.38
45.26
44.98
42.91
11.3
Results on Tiny ImageNet. To further validate the effectiveness of I-PGD-AT, perform similar experiments on Tiny ImageNet. Due to the high computational cost of GradAlign, here we do not take it baseline. As shown in Table lb the results consistently demonstrate that our I-PGD-AT can improve the robustness over Fast-Ar AT without extra computational cost. We find that the improvement of I-PGD3-AT over I-PGD2-AT is not so obvious as on CIFAR- 10, which might due to the fact that the sign consistency on Tiny ImageNet is higher than that on CIFAR- 10 as discussed in Sec. 4.4
4.3 INTEGRATING I-PGD INTO PGD-AT
Though the primary goal of I-PGD-AT to boost Fast-AT, the imitation strategy is not limited to single-step adversarial training and can be integrated with PGD-AT during each step. Specifically, at each iteration, we incorporate I-PGD2 into PGD2-AT and PGD7-AT, denoted as I-PGD2-ATppan and I-PGID2-TPpana PGD7, to imitate more powerful iterative adversarial training, namely PGD4-AT and PGD14-AT. As shown Table I-PGD2-AT PGD22 and I-PGD2-AT PGD? consistently achieve better robustness than the corresponding baselines PGD2-AT and PGD7-AT, with almost the same training time. Surprisingly, I-PGD2-ATPon exhibits much better robustness than PGD4-AT. 1-PGD2-ATpGDD achieves slightly lower robustness than PGD14-AT, which takes double iterations for optimizing the inner maximization problem of adversarial training and doubles the training time.
4.4 SIGN CONSISTENCY FOR PGD
The proposed method I-PGD is based on the hypothesis that due to the neighborhood constraint by adversarial attacks, the proba- bility p that a specific element in the input obtains the same sign in two consecutive iterations would be high and stable. To val- idate this hypothesis, we define the sign consistency (SC) for i-th iteration of PGD as follows:
6,155
0.90
PGD Tormdoe
The sign consistency for PGD50 Fast-AT various epochs on CIFAR-10 (left) and Tiny ImageNet (right), where Fast-AT-K denotes the model trained k-th epoch. The results are averaged over random seeds used for training and reported with the standard deviation. The high and stable sign consistency at various epochs makes it possible for imitation.
Figure 2:
SC(i) (5)) sign(gets ()),
where 9: denotes the gradient at i-th iteration and 1 1.) is the indicator function. To some extent, SC(i) also indicates the local linearity of the deep neural models. If (C(()) = 1, PGD attack would degenerate to FGSM-RS attack because the direction for perturbation update in Eq. 3 is constant at each iteration so that the iterations do not make any difference. In contrast, the perturbation of PGD differs from that of FGSM-RS significantly if SC()) is small. Besides, the stability of SC()) makes it possible for I-PGD-AT to adopt a fixed probability to imitate PGD-AT effectively.
We calculate SC(i) for PGD50 attack on Fast-AT at various training epochs over 3 random seeds for generally speaking, the sign consistency of PGD50 is high and stable
training. As shown in Fig. 2 at various iterations but training longer degrades the sign consistency and increases the standard deviation. Moreover, on Tiny ImageNet, the sign consistency decreases much slower with smaller standard deviation than that on CIFAR-10. which might be the reason why I-PGD3-AT cannot significantly enhance 1-PGD2-AT and AT is more stable on Tiny ImageNet.
7

---

Under review as a conference paper at ICLR 2022
43
25
1.8
(a) The mean absolute value of perturbation 66.
(b) Robust Accuracy against PGD50 attack.
(c) The epoch when catas- trophic overfitting occurs.
Figure 4: The (a) mean absolute value of perturbation |6| (normalized into [0, <)), (b) robust accuracy, and (c) the epoch when catastrophic overfitting occurs over various step size a for Fast-AT-U and Fast-AT-S. Initially, catastrophic overfitring does not happen and increasing [6] leads to better robustness. However, if catastrophic overfitting occurs, increasing [o] will boost catastrophic overfitting.
4.5 FACTORS RELATED TO CATASTROPHIC OVERFITTING
In this section, we investigate what factors would affect the catastrophic overfitting in single-step adversarial training. First, we explore if our I- PGD-AT can avoid catastrophic over- fitting. We train the model with 60 epochs using two learning rate sched- ulers, i.e. single cyclic learning rate and the eyclic learning rate in the first 30 epochs followed with piecewise learning rate, in which the learning rate is decayed by factor of 0.1 per 10 epochs, dubbed CLR and CPLR re- spectively. As shown in Fig. 3 with CLR, catastrophic overfitting occurs in both Fast-AT and I-PGD2-AT but I-PGD2-AT postpones it for 7 epochs compared with Fast-AT. Besides, with CPLR. Fast-AT still leads to catastrophic overfitting while I-PGD2-AT stably main- tains its high robustness after 30th epochs. This indicates that our I-PGD-AT can effectively delay catastrophic overfitting and stablize the training process.
Relest (o0)
Rebart (P5))
Epocks
Fpach
Figure 3: The robustness against FGSM (dashed line) and PGD1O (solid line) of Fast-AT and I-PGD2-AT over different training epochs with two different learning rate schedulers evaluated on randomly sampled 10% CIFAR-10 10 testset. The learning rate sched- ulers, namely Cyclic Learning Rate (CLR, left) and Cyclic Learning Rate followed by Piccewise Learning Rate (CPLR, right), are shown in the left upper corner in the corresponding figures. The red line and blue line denote the performance of Fast-AT and I-PGD2-AT trained over 60 epachs.
We further explore the factors that affect catastrophic overfitting CIFAR-10. Previous works (An- driushchenko & Flammarion, 2020 Kim et al. 2021) show that small loo -norm constraint c for training prevents catastrophic overfitting. We find that even with the same e, the mean absolute value of the perturbation Jo] generated by I-PGD2 is 6.5, which is a little smaller than the one of FGSM-RS. Furthermore, we find that adopting the sign of uniform perturbation for initialization also prevents catastrophic overfitting, which leads to |8| 5.0. Thus, we speculate that under the same fo -orm constraint on the perturbation 6, smaller [8] can effectivvly delay catastrophic overfitting
To validate this hypothesis, we fix e 8/255, while vary the step size in Eq. 4 from c/2 to 2 to adjust [6] for Fast-AT with uniform perturbation as initialization, dubbed Fast-AT-U. To show the generality of such hypothesis, we further adopt the perturbation sampled from [-1,+1} uniformly as initialization, denoted as Fast-AT-S. As shown in Fig. 4a when we increase the value of a, [8] increases Fast-AT-U while [8] first decreases before a then increases linearly in Fast-AT-S. This is because when = 1, the random perturbation inconsistent with the sign of gradient will cancel each other out, resulting in the minimum perturbation. As shown in Fig. 4bb before the catastrophic overfitting occurring. the larger a which means larger [o] leads to better robustness for both initialization methods. If we continue to increase the value of a, depicted in Fig. [c] the catastrophic overfitting occurs and the larger a leads to the earlier epoch for catastrophic overfitting.
supports the hypothesis and shows the trade-off between high robustness and catastrophic overfitting.
4.6 ABLATION STUDY
To further gain insights on the performance improvement obtained by I-PGD-AT, we conduct ablation study for the probability p for I-PGD2-AT on CIFAR-10. As shown in Sec. A. the higher probability
p indicates the higher portion that the value 2 would occupy in [ğ] at line 7-9 in Algorithm leading
8

---

Under review as a conference paper at ICLR 2022
p 0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Clean Acc.
82.14
82.03
81,55
81.71
81.59
86.09
84.21
85.11
85.71
84.33
84,46
PGD50 Acc.
46,18
46.99
47.40
48.09
48.16
0.00
0.00
0.00
0.00
0.00
0.00
[8]
5.90
6.12
6.31
6.50
6.70
6.90
7.10
7.29
7.48
7.68
7.88
Epoch
23
20
14
10
10
Table 3: Ablation study for the probability used in I-PGD2-AT on CIFAR-10. 10. jöll denotes the mean absolute
value of perturbation and Epoch denotes the epoch when catastrophic overfitting occurs
to larger (o]. We vary p from 0.5 to 1 and summarize the results in Table As can see, when increase the value of p. the clean accuracy decreases a little but the robustness increases significantly when p S 0.7. However, when we continue to increase p. |o] increases and the catastrophic overfitting happens. The larger indicates the larger [ō] and results in the earlier epoch for catastrophic overfitting. This is also consistent with our experiments and analysis Sec. [45 and further support our hypothesis that under the same t-norm constraint on the perturbation 6, smaller {ō] can effectively delay the catastrophic overfitring.
S.c.
4.7 ROBUST OVERHTTING A-PGD-AT
Rice et al. (2020) identify robust overfitting in PGD-AT that the robust accuracy begins to decrease as training progresses after few epochs. Here we briefly explore whether I-PGD- AT is helpful to delay or avoid the robust overfitting. As shown in Fig. S I-PGD2-AT consistently exhibits better robust accuracy than PGD2-AT and PGD4-AT throughout the training process. As for the robust overfitting, I-PGD-AT maintains the peak performance between 50-th and 75-th epochs, while the robust accuracy of PGD2-AT and PGD4- AT begins to decrease around 50-th epoch. Hence, it is clear that I-PGD-AT can delay (not avoid) the robust overfitting in PGD-AT with better final robustness, and the sustainability for peak robustness might be helpful to choose a better epoch for early stopping.
Robut Accorcy (90)
85
10
PLDDED ACC. on PGD2-AT
Arc. PGDEAT
poad
25
50
100
150
175
200
Epochs
Figure 5: The robust accuracy (%) against
PGD10 attack over the training epochs for PGD2-AT, PGD4-AT and PGD2-ATrane
1-PGDK
82
Robust
50
4.8 COMPARING I-PGD WITH PGD ATTACKS
45
Though we have observed that I-PGD-AT can significantly improve the robustness of single-step and multi-step adver- sarial training, we wonder whether I-PGD attack indeed imitates the attack effectiveness of PGD. To verify this, we incorporate the imitation strategy for PGD2 into PGD attack to evaluate the robustness of I-PGD2-AT on CIFAR-10. We denote such attack as I-PGDk where le is the number of iteration. As illustrated in Fig. 6 we find that for various number of iteration k. I-PGDK consistently achieves similar attack performance as PGDk. The imitation for PGD3 also exhibits the same trend and we do not show it in the figure for clarity. The reasons should be two-fold: a) As shown Fig. 6 increasing the number of iteration cannot effectively improve the attack performance for PGDA when & 2 20. b) Our imitation only considers the diverse value of perturbation without effectively predicting the sign of generated perturbation at each iteration, which might limit the attack performance. We believe that it is possible to improve the attack performance through imitating powerful attacks and leave this as our future work.
Eterations
Figure 6: The robust accuracy (%) of I- PGD2-AT on CIFAR- 10 against PGDk and I-PGDk attacks where k denotes the num- ber iteration.
5 CONCLUSION
In this work, we identified the differences in perturbations crafted by FGSM-RS and PGD. Based on this observation, proposed I-PGD adversarial training (1-PGD-AT) by adopting I-PGD attack that imitates PGD virtually through single gradient calculation. Experiments on CIFAR-10 10 and Tiny ImageNet showed that I-PGD-AT could effectively improve the robustness and delay catastrophic overfitting and robust overfitting. We believe that this work would inspire more precise imitation of PGD, which can improve the attack effectiveness and further enhance the robustness of DNNs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In Advances Neural Information Processing Systems, 2020.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In European Conference Computer Vision, 2020.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses adversarial examples. In International Conference on Machine Learning, 2018.
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David A. Forsyth. Unrestricted adversarial examples via semantic manipulation. International Conference on Learning Representations, 2020.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Srivacy. IEEE, 2017.
Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu, and Jingjing Liu. Efficient robust training via backward smoothing. In arXiv preprint arXiv:2010.01278 2020.
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning. 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference Machine Learning, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, 2020b.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of
deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. MMA training: Direct input space margin maximization through adversarial training. In International Conference on Learning Representations, 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Conference on Computer Vision and Pattern Recognition, 2018.
Yinpeng Dong, Zhijie Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. In Advances in Neural Information Processing Systems, 2020.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovie, Timothy Arthur Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In International Conference on Computer Vision, 2019.
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference Learning Representations, 2018.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.
10

---

Under review as a conference paper at ICLR 2022
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In AAAI Conference on Artificial Intelligence, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In International Conference Learning Representations, Workshop Track Proceedings, 2017.
Bai Li, Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial training. In arXiv preprint 0tt::2006.03089) 2020a.
Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. QEBA: query-efficieet boundary- based blackbox attack. In Conference on Computer Vision and Pattern Recognition, 2020b.
Huichen Li, Linyi Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. Nonlinear projection based gradient estimation for query efficient blackbox attacks. International Conference on Artificial Intelligence and Statistics, 2021.
Linyi Li, Zexuan Zhong. Bo Li, and Tao Xie. Robustra: Training provable robust neural networks over reference adversarial space. In International Joint Conference on Artificial Intelligence, 2019.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Conference on Computer Vision and Pattern Recognition, 2018.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks, In International Conference on Learning Representations, 2017.
Zihao Liu, Qi Liu, Tao Liu, Nuo Xu, Xue Lin, Yanzhi Wang, and Wujie Wen. Feature distillation: Dnn-oriented JPEG compression against adversarial examples. In Conference on Computer Vision and Pattern Recognition, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Aamir Mustafa, Salman Khan, Munawar Hayat, Jianbing Shen, and Ling Shao. Image super- resolution as a defense against adversarial attacks. In IEEE Transactions on Image Processing, 2019.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self- supervised approach for adversarial robustness. In Conference on Computer Vision and Pattern Recognition, 2020.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. In International Conference on Learning Representationss 2018.
Leslie Rice, Eric Wong, and J. Zico Kolter. Overfitting in adversarially robust deep learning, In International Conference on Machine Learning, 2020.
Vivek B. S. and R. Venkatesh Babu. Single-step adversarial training with dropout scheduling. In Conference on Computer Vision and Patlern Recognition, 2020.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, 2019.
Leslie N Smith. Cyclical learning rates for training neural networks. IEEE Winter Conference on Applications of Computer Vision, 2017.
11

---

Under review as a conference paper at ICLR 2022
Chuanbiao Song, Kun He, Jiadong Lin, Liwei Wang, and John E. Hoperoft, Robust local features for improving the generalization of adversarial training. International Conference on Learning Representations, 2020.
Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, and R Venkatesh Babu. Guided adversarial attack for evaluating and enhancing adversarial defenses. In Advances in Neural Information Processing Systems, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Representations, 2014.
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, and Jinwoo Shin. Consis-
tency regularization for adversarial robustness. In arXiv preprint arXiv:2/03.04623, 2021.
Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In Conference on Computer Vision and Pattern Recognition, 2021.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. International Conference on Learning Representations, 2020.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020.
Chaowei Xiao, Bo Li, Jun-Yan Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In International Joint Conference on Artificial Intelligence, 2018a.
Chaowei Xiao, Jun-Yan Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In International Conference on Learning Representations, 2018b.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan L. Yuille. Mitigating adversarial effects through randomization. In International Conference Learning Representations, 2018.
Dinghuai Zhang, Tianyuan Zhang. Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In Advances in Neural Information Processing Systems, 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Lauuret El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, 2019b.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 CALCULATION FOR I {v; p:}
In this section, we first provide the detailed calculation for I {uy pi} with PGD2 an example and then summarize a general algorithm for PGDk.
Taking PGD2 for example, there would be four cases of the sign for each pixel x; E z, namely (+1, +1), (+1, 1), (-1 1) and (-1, -1). Suppose the sign of the first iteration is either +1 or with equal probability, we could calculate the probability for each sign sequence as following:
P((+1, +1)) = p p 2
P((+1, 111 2 (1 - 1-p
2
2
P((-1, -1) = 2 p
P((-1,+1)) = (1 (1-p) 1 1-P 2
where p is the probability that a specific element z; obtains the same sign (i.e. (.... 1, +1 ....)..................... or
(... 1.-1....) in two consecutive iterations. The imitative value would be {0, 2) and could calculate the corresponding probability:
P(e 0) P((+1, P( =1-1
P(c = 2) = P
= =
Thus, we could obtain that Z[0] = P(u 5 = = and Z = P(u 2) - i.e. (0: p. 2: 13. For an arbitrary value of k we first traverse each possible sign sequence to calculate the corresponding probability. Then we calculate the probability for each potential imitative value by accumulating the probability of related sign sequences. To obtain the quantile for each possible value, we further accumulate the probability based on the imitative value in ascending order. The general calculation for any is summarized Algorithm
Algorithm 2 Calculation for imitative value and accumulated probability: (u; : pa}
Input: The number of iteration for imitative PGD, the probability that specific element obtains the same sign in two consecutive iterations
1: Initialize I = 0
2: for i-th binary sign sequence s; with value (+1. and size of is do
3: U; = the absolute value of the sum s,
4: Pr., the probability for the sign sequence 8;
5:
if u E Z then
6:
I[uy] 0
Initialize the probability for value by in I
7:
end if
8:
I(e:] Z[u]] + Pv.
i/ might be related to various sign sequences
9:
Sort I based on the key v in ascending order
for each (u, P:) do
> Accumulate the probability as the quantile
11: Tpu = 1j
12: end for
13: return I (v: p.}
14: end for
A.2 AN EXAMPLE OF CALCULATING THE QUANTILE
To better illustrate the algorithm, we also provide a simple example of imitating PGD2 with the probability p = 2/3 as follows.
= 0) -1 -1 P( = p= and calculate the imitative value and accumulated probability I {0 1/3.2 1). Suppose obtain
First, we could obtain the probability [[
0.2
-0.3
-0.1
the gradient matrix g
0.5
-0.6
0.3
at line of Algorithm we need to calculate the
0.7
0.1
0.9
13

---

Under review as a conference paper at ICLR 2022
start and end value for the range with the portion from 0 1/3 and from 1/3 1 the sorted array [0.1, 0.1, 0.2, 0.3, 0.3, 0.5, 0.6, 0.7,0.7... (the absolute values of g), respectively. Thus, we could get the quantile Q 0.2, 0.9) at Line 6 of Algorithm (0 is the minimal value for the portion of O, 0.2 is the smaller 3-quantile for the portion of 1/3 and 0.9 is the maximum value for the portion of
0 2 0
1.). After the for-loop at line 7-9 of Algorithm we have =
2 2 2
by calculating each
2 0 2
0.1
0.2
element g(t
Then we let share the same sign of g at line 10 and
2
0.2
S 0.9
0
-2
obtain ğ =
2
-2 2
.which could be used to generate the perturbation for adversarial training
at line 11-12 of Algorithm
A.3 EXPERIMENTAL SETTING
In this section, we provide more details about the experimental setting, including datasets, hyper- parameters for baselines and the attack settings.
Datasets. We adopt two widely investigated benchmark datasets for the evaluation, namely CIFAR- 10 and Tiny ImageNet. CIFAR-10 contains 10 classes with 5, 000 training and 1, 000 testing images of resolution 32 x 32 per class. Tiny ImageNet contains 120, 000 images of 200 classes downsized to 64 x 64 colored images, in which each class has 500 training images, 50 validation images and 50 testing images.
a 6, m and train the model for 200 epochs. We adopt a = 1.25 e for Fast-AT (Wong et al., 2020), set the regularization parameter for GradAlign 0.2 (Andriushchenko & Flammarion 2020) and utilize checkpoints for Kim er al. (Kim et al. 2021). We follow the setting of FastAdv+ (Li et al. (020a) which reports the best results among 200 epochs with piecewise learning rate and adopts PGD-IO to recover the catastrophic overfitting. For PGD&-AT, the step size a is set to max 2/255, e/k). For all the baselines, we evaluate the effectiveness of the models at the last training epoch (Wong et al. 2020 AAnriishhhhnko
Hyper-parameters for baselines. For Free-AT,
& Flammarion 2020 Kim et al. 2021).
Attack settings. CW implemented by adopting the margin- based loss function (Carlini & Wagner 2017) and using PGD for optimization. We use 20 and 50 steps for PGD and CW and 50 steps for MIM with 10 random restarts. The perturbation budget is set to 8 8/255 and step size a is set to 2/ 2/2 AutoAttack adopts the default setting (Croce & Hein 2020b) with four adversarial attacks, namely untargeted APGD-CE, targeted APGD-DLR, targeted FAB (Croce & Hein 2020a) and Squared attack (Andrrushchenko al. 2020
14


Reviews:
[{"review_id": "czqFhJqZ4gq", "reviewer": "Reviewer_Dz2K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros: \n\n1. The paper focuses on the catastrophic overfitting problem caused by the limited number of adversarial perturbation patterns generated by the single-step attack method during the adversarial training process, and proposes I-PGD-AT to alleviate the problem.\n\n2. The paper pays attention to the both problems of high computations in PGD-based adversarial training and limited attack patterns in FGSM-based adversarial training, and gives a compromise method to absorb the advantages of both methods and discard the disadvantages at the same time.\n\n3. This paper provides comprehensive experiments on CIFAR10 and Tiny ImageNet to demonstrate the effectiveness of I-PGD-AT. And further analysis on catastrophic overfitting problem is provided.\n\n##########################################################################\n\nCons: \n\n1. The author only focuses on $\\delta’$ and ignores $\\rho$ in Fig. 1 to demonstrate that PGDk can generate more diverse search space and further find more diverse adversarial examples in the process of attack. However, since both FGSM-RS and PGDk utilizes the random start in the beginning of the attack, I think both of the methods can explores the whole search space in \\Deta.\n\n2. Compared to Fast-AT, I-PGD-AT only slightly improves the robustness of the model under attack (average ~2% in Tab. 1), but the classification accuracy on clean images has dropped obviously (the drop also reaches ~2% in Tab. 1).\n\n3. The author has too few methods to compare in the experiments. Since a large number of methods in adversarial training haves been proposed in recent years, more methods should be compared to demonstrate the effectiveness of the I-PGD-AT.\n\n#########################################################################\n\nSome typos: \n\n(1) two lines above Sec. 3.4: \\mathcal{I}=\\{0: 1-p, 2: 1\\} -> \\mathcal{I}=\\{0: 1-p, 2: p\\}\n\n(2) the same problem with (1) in Appendix A.1\n\n(3) immediately after the problem of (2), “arbitra ry” should have no blank space\n\n\n\n", "summary_of_the_paper": "The paper notices that the existing methods of adversarial training has the problem of catastrophic overfitting, and proposes an efficient single-step adversarial training method I-PGD-AT by adopting I-PGD attack for training, in which I-PGD imitates the perturbation of PGD based on the magnitude of gradient. Extensive empirical evaluations on CIFAR-10 and Tiny ImageNet demonstrate that I-PGD-AT can improve the robustness compared with the baselines and delay catastrophic overfitting.", "summary_of_the_review": "Overall, I vote for weak reject. The paper proposes I-PGD-AT to imitate PGD virtually through single gradient calculation. My major concern is about the margin improvement and the limited compared methods in experiments (see cons below). Hopefully the author can address my concern in the rebuttal period. \n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nUPDATE\n\nAfter reading all the response and comments from other reviewers, I will keep my scores. My main concern is the insufficient experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_lGSPrq8o75", "reviewer": "Reviewer_bLbt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1 . I wonder if the approximation in (6) makes sense? In (6), the gradient in later iterations can go far outside the perturbation limit and thus make the update nowhere near the actual case where all PGD iterations’ gradients are strictly limited within the perturbation ball.\n\n2 . The authors claimed that one of the problems of one-step AT is that the perturbations are not diverse enough while in Figure 1, they showed that PGD-k should have diverse discrete perturbations. However, in practice, PGD attacks are usually located on the boundary with common choice attack step size (also mentioned in [1]).   \n\n[1] \"Parsimonious black-box adversarial attacks via efficient combinatorial optimization.\" International Conference on Machine Learning. PMLR, 2019. \n\nAlso, even if Figure 1 is correct or aligned with practice, to me, it still does not explain how adopting random initialization can mitigate such a phenomenon.\n\n3 . In the experiments, I notice that the authors actually used a large attack step size such as epsilon or 1.25epsilon, so that the final projection point is no longer located at those discrete points as in Figure 1. Rather, it will again be located on the boundary of the norm ball.\n\n4 . The robustness improvement is marginal compared with Fast-AT as shown in Table 1. Also, it seems that the proposed method is actually trading natural accuracy for robustness? \n\n5 . In table 2, PGD2-AT performance is even worse than Fast-AT which is clearly not reasonable. The authors might want to double-check their experimental results.\n\n6 . From figure 3, it seems that both I-PGD-AT and Fast-AT are experiencing catastrophic overfitting problems, just that Fast-AT happened first. Since this phenomenon may occur with randomness, it is hard to decide whether I-PGD-AT really solves the catastrophic overfitting problem. The authors should at present the result for repeated runs and see whether this is consistent.\n\n7 . The following two papers also work on fast adversarial training. The authors might also want to comment and compare with them:\n\n[2] \"Towards understanding fast adversarial training.\" arXiv preprint arXiv:2006.03089 (2020).\n[3] \"Efficient robust training via backward smoothing.\" arXiv preprint arXiv:2010.01278 (2020).\n\n\n\n", "summary_of_the_paper": "In this paper, the authors studied improving the efficiency of adversarial training. The authors first analyzed the difference of perturbation generated by FGSM-RS and PGD and proposed an efficient single-step adversarial training method I-PGD-AT by adopting I-PGD attack for training. Extensive empirical evaluations on CIFAR-10 and Tiny ImageNet demonstrate that the proposed I-PGD-AT can improve the robustness compared with the baselines and significantly delay catastrophic overfitting. ", "summary_of_the_review": "The intuition of the proposed method is not really making sense and the empirical results are not quite convincing, therefore, I recommend reject for this paper", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EWegQXiZOWY", "reviewer": "Reviewer_hVJW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n* Great empirical results by improving over Fast-AT with the same runtime as well as delaying catastrophic overfitting.\n* The paper is well written.\n* The idea of using the quantiles of the gradient magnitudes to produce multiple step sizes  within a single step adversarial method is a clever idea.\n* The code is attached to the supplementary material and looks usable for easy reproducibility.\n\n**Weaknesses**:\n1) Regarding equation (6) and \"We approximately ignore the projection operator at each iteration as the step size $\\alpha$ is expected to be small\", the theoretical justification does not seem well-supported. First, according to the experiment section, $\\alpha$ is not small enough compared to $\\epsilon$. Indeed, if the initial random perturbation $\\rho$ is not equal to 0, the perturbation can get \"stuck\" on the pertubation set boundary after a few steps. If $\\rho$ is close to $\\epsilon$, the perturbation boundary can even be reached after one step. Hence, we cannot ignore the projections. Second, if we cannot ignore the projections, we lose the grid structure (like in Figure 1) used in the rest of the analysis to determine the algorithm. I might be wrong for both points but it would require some clarifications.\n2) Regarding the sign consistency experiment. Could the high value for the sign consistency be due to the perturbation being \"stuck\" on the perturbation set boundary? If yes, the fact that the projections which were ignored in order to come up with the algorithm would be the responsible for the high value of the sign consistency. That would go against \"The proposed method I-PGD is based on the hypothesis that due to the neighborhood constraint by adversarial attacks, the probability $p$ that a specific element in the input obtains the same sign in two consecutive iterations would be high and stable\".\n3) The theoretical approach boils down to a simple result \"{0 : 1 − p, 2 : p}\" for  I-PGD2-AT and  \"{1 : 1 − p, 3 : p}\" for I-PGD3-AT. There might be other approaches leading to this simple result without making the assumptions discussed above which need clarifications. It could just be an heuristic approach such as: PGD leads to perturbations close to the boundary or closer to the random start. This last sentence translates into your result \"{0 : 1 − p, 2 : p}\" and would just require a sweep over the probability $p$. Such a sweep is done in Table 3 for I-PGD2-AT and it would be great to have the same for \"{1 : 1 − p, 3 : p}\" with I-PGD3-AT.\n4) The ablation study on the probability $p$ in the appendix should have been in the main paper, more than the catastrophic overfitting discussion.\n5) Minor. There are a few typos in the paper. Page 5 and in the appendix: {0 : 1 − p, 2 : 1} ->  {0 : 1 − p, 2 : p}. In Section 4.6, the subscripts of  \"I-PGD2-AT\" are lacking compared to Figure 5.\n6) Minor. Why doing the catastrophic overfitting study in Figure 4 on Fast-AT rather than on the proposed method?\n7) Minor. In Figure 5, you should add a curve \"PGD10 Acc on PGD10-AT\" to the plot for a more complete comparison.", "summary_of_the_paper": "**Few sentences summary**: the paper proposes a single-step adversarial training method called I-PGD-AT to improve over the Fast Adversarial Training method. Based on the fact that not all the perturbations lie on the boundary of the perturbation set when using multiple steps PGD, this method aims at replicating this behaviour while still using only one single adversarial training step. \n\nFor the **proposed method**, it uses:\n* Quantiles over the magnitudes of the gradient across the image.\n* A model approximating how the perturbation space is \"crowded\" after $k$ PGD steps under the assumption that gradients sign changes between iterations occur with a fixed probability $p$.\n* Quantiles over the gradient magnitudes and quantiles over the occupation of the perturbation space are matched such that the strongest gradients are assigned to the furthest position in the perturbation space.\n\nFor the **experiments**:\n* Evaluation on CIFAR-10 and TinyImageNet against other single-step methods: Fast-AT, GradAlign and Kim et al.(2021).\n* Evaluation on the extension of the proposed method to the PGD case.\n* The paper evaluates the sign consistency of the gradients between two consecutive iterations as the proposed method relies on the assumption that sign changes between iterations occur with a fixed probability $p$.\n* General investigation on catastrophic overfitting for single step methods.\n", "summary_of_the_review": "The paper proposes a method which could be very useful to the community as it provides a single-step adversarial training method with clearly improved performance compared to existing methods like Fast-AT with the same runtime. I am vouching for acceptance due to the great experimental results but currently I am not yet satisfied by the theoretical justification of the method as a few statements are not well-supported. I think that it can be clarified in the rebuttal as using the quantiles of the gradient magnitudes to produce multiple step sizes  within a single step adversarial method is a clever idea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7-Tb2sB4ozH", "reviewer": "Reviewer_AAHj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros: \n\n1. The problem the paper is trying to solve is an important one. Adversarial robustness is of great importance and adversarial training has\nbeen shown to be one of the most effective methods. However, the efficiency of multi-step adversarial training is a problem. \n\n \n2. The intuition behind the proposed method makes sense. The difference between PGD generated adversarial examples and FGSM generated ones comes from the perturbations. The proposed method introduces a heuristic way to imitate PGD adversarial perturbation based on one gradient query, which is efficient and novel. \n\n \n3. Experimental results on CIFAR10 and TinyImageNet show that I-PGD-AT outperforms several fast adversarial training methods against state-of-the-art attacks. Besides, it does increase the training time much compared to FAT. Studies on catastrophic overfitting are also done to investigate contributing factors to the issue.\n\n \n##########################################################################\n\nCons: \n\n \n1. My main concern is that the method is not evaluated on ImageNet. In the paper of Fast adversarial training (Wong et al., 2020), the method was tested on ImageNet. It might be hard to address this problem as performing adversarial training (even the fast version) may take several days.\n\n \n2. Overall, the paper is clearly written and well-organized, but some details are missing, making it hard to follow. (See detailed comments below.) But, this can addressed during rebuttal.\n \n##########################################################################\n\nDetailed Comments: \n\n(1) I like the fact that pseudo code of computing possible discrete values and their corresponding probabilities. However, the process of generating quantile set Q based on I is just one sentence. It would be better if some details of this process is provided in the appendix as well. (Maybe I'm slow in understanding this. I read the code to figure out the process.)\n\n(2) Section 3.3, \"We only need to consider the absolute value and the smaller |g|[i] is, the smaller value...\" is a little bit confusing. Suggestion: \"We only need to consider the absolute value. The smaller |g|[i] is, the smaller value...\"\n\n(3) What is \"the sign of uniform perturbation\"? It's not mentioned previously. Suddenly, the technique is used to deal with catastrophic overfitting. \n\nPlease address and clarify the cons and detailed comments above.", "summary_of_the_paper": "Summary:\n\nThe paper introduces a new adversarial training method to improve the performance of Fast adversarial training (FAT), an adversarial training method that depends on only one gradient query. Adversarial training using PGD attack is not efficient, but FAT performs worse than PGD adversarial training. To improve the performance of FAT and maintain the efficiency, the authors study the characteristics of PGD perturbation and proposes a method to imitate PGD perturbation based on only one gradient query. \n\n", "summary_of_the_review": "Reasons for score: \n\n \nOverall, I vote for weakly accepting. I like the idea of imitating the PGD perturbation to improve the performance of one-step adversarial training. Experiments show that the performance of the proposed method is better than fast adversarial training and has similar training\ntime. My major concern is that the proposed method is not evaluated on large image dataset, like ImageNet. Some methods perform well \non small datasets like CIFAR10 but do not perform well on large datasets. Since Fast adversarial training can be deployed on ImageNet \n(experiments can be found in the paper), why not compare I-PGD-AT with FAT on ImageNet?\n\n##########################################################################\nUpdates: Thanks for the authors' response. Part of my concerns are addressed by the author. The concern about lack experiments on ImageNet is hard to address during rebuttal period. Therefore, I'll keep my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to improve the efficiency of adversarial training. Specifically, by analyzing the differences between the adversarial perturbations generated by FGSM-RS and the adversarial perturbations generated by PGD, this paper proposes a new single-step attacker I-PGD (which imitates PGD by creating diverse adversarial perturbations) to accelerate adversarial training. Empirical results are provided on CIFAR-10 and Tiny ImageNet to support the effectiveness of the proposed method.

Overall, the reviewers think it is an interesting paper, but are severely concerned about some statements. During the discussion period, the authors actively clarify these points. However, the Reviewer bLbt is not fully convinced and believes 1) the approximation in Eq. (6) is incorrect and 2) the proposed method is loosely motivated by imitating the behavior of PGD. The authors fail to further follow up on this discussion. The Reviewer bLbt and the Reviewer Dz2K are also concerned that the proposed I-PGD-AT only yields margin improvements over Fast Adversarial Training. In addition, as suggested by the reviewer AAHj, given this work focuses on developing efficient adversarial training, it is important to include results on larger-scale datasets like ImageNet.

I encourage the authors to incorporate all the reviewers' comments and make a stronger submission next time.
Decision: Reject