Fast AdvProp
Authors: Jieru Mei
Yucheng Han
Yutong Bai
Yixiao Zhang
Yingwei Li
Xianhang Li
Alan Yuille
Cihang Xie
Venue: ICLR 2022 Poster (2022)
Keywords: Adversarial examples
efficient training
generalization

Abstract: Adversarial Propagation (AdvProp) is an effective way to improve recognition models, leveraging adversarial examples. Nonetheless, AdvProp suffers from the extremely slow training speed, mainly because: a) extra forward and backward passes are required for generating adversarial examples; b) both original samples and their adversarial counterparts are used for training (i.e., 2X data).  In this paper, we introduce Fast AdvProp, which aggressively revamps AdvProp's costly training components, rendering the method nearly as cheap as the vanilla training. Specifically, our modifications in Fast AdvProp are guided by the hypothesis that disentangled learning with adversarial examples is the key for performance improvements, while other training recipes (e.g., paired clean and adversarial training samples, multi-step adversarial attackers) could be largely simplified.   

Our empirical results show that, compared to the vanilla training baseline, Fast AdvProp is able to further model performance on a spectrum of visual benchmarks, without incurring extra training cost. Additionally, our ablations find Fast AdvProp scales better if larger models are used, is compatible with existing data augmentation methods (i.e., Mixup and CutMix), and can be easily adapted to other recognition tasks like object detection.  The code is available here: https://github.com/meijieru/fast_advprop.

Published as a conference paper at ICLR 2022
FAST ADVPROP
Jieru Mei', Yucheng Han', Yutong Bai', Yixiao Zhang', Yingwei Li', Xianhang Li
Alan Yuille' & Cihang Xie Johns Hopkins University
Nanyang Technological University
3 UC Santa Cruz
ABSTRACT
Adversarial Propagation (AdvProp) is an effective way to improve recognition models, leveraging adversarial examples. Nonetheless, AdvProp suffers from the extremely slow training speed, mainly because: a) extra forward and backward passes are required for generating adversarial examples; b) both original samples and their adversarial counterparts are used for training (i.e., 2x data). In this paper, we introduce Fast AdvProp, which aggressively revamps AdvProp's costly training components, rendering the method nearly cheap as the vanilla training. Specifically, our modifications Fast AdvProp are guided by the hypothesis that disentangled learning with adversarial examples is the key for performance improvements, while other training recipes (e.g., paired clean and adversarial training samples, multi-step adversarial attackers) could be largely simplified.
Our empirical results show that, compared the vanilla training baseline, Fast AdvProp is able to further model performance on a spectrum of visual bench- marks, without incurring extra training cost. Additionally, our ablations find Fast AdvProp scales better if larger models are used, is compatible with exist- ing data augmentation methods (i.e., Mixup and CutMix), and can be easily adapted to other recognition tasks like object detection. The code is available here: https://gituub.commmmjjjeeeuu//e aaateaaiierrafssssappopopo.
INTRODUCTION
Deep neural networks are highly successful for visual recognition. As fueled by powerful computa- tional resources and massive amounts of data, deep networks achieve compelling, sometimes even superhuman, performance on a wide range of visual benchmarks. However, when testing out of the box, these exemplary models are usually get criticized for lacking generalization/robustte increasing amount of works point out that deep neural networks are brittle at handling out-of-domain situations like natural image corruptions (Hendrycks & Dietterich, 2018), images with shifted styles (Geirhos et al., 2018; Hendrycks et al., 2020), etc.
an
Adversarial propagation (AdvProp) (Xie al., 2020), which additionally feeds networks with adversarial examples during training, emerged as one of the most effective ways to train not only accurate but also robust deep neural networks. The key in AdvProp is apply separate batch normalization (BN) layers (loffe & Szegedy, 2015) to clean training samples and adversarial training samples, they come from different underlying distributions. Later works further explore the potential of AdvProp on other learning tasks, including object detection (Chen et al., 202 Xu al., 2021), contrastive learning (Jiang al., 2020; Ho & Vasconcelos, 2020; Xu & Yang, 2020) and large-batch training (Liu et al., 2022).
However, the benefits brought by AdvProp do not come for "dree" introduces a significant amount of additional training cost, which is mainly incurred by generating and augmenting adversarial training samples. For instance, compared to the vanilla training baseline (where only clean images are vnvolved), the default setting in AdvProp (Xie et al., 2020) increase the total computational cost by factor of 7, i.e., 5/7 from generating adversarial examples, 1/7 from training adversarial examples, 1/7 from training clean images. This extremely high training cost not only limits the further explorations of AdvProp on larger networks (Xie et al., 2019; Brock et al., 2021; Dosovitskiy et al., 2020), with larger datasets (Sun et al., 2017; Kuznetsova et al., 2020), and for different learning tasks, but also makes the direct comparisons against other low-cost learning algorithms (Zhang et al., 2018; De Vries & Taylor, 2017; Yun et al., 2019; Cubuk et al., 2019b;a) seemingly unfair.

---

Published as a conference paper at ICLR 2022
Sampled
Sampled Batch
"hoo' spersiion
Huney
harward Bachordd
Clean Image
Adversarial Image
Image w/ Noise
Model Ingut
Model Input
Adotrap
(50
Figure 1: Comparison between AdvProp and Fast AdvProp. (a) AdvProp generates a paired adversar- ial image (with color orange) for each clean image (with color blue) in the sampled batch, therefore incurring heavy training cost. Morcover, in addition to clean images, adversarial images are also fed into networks for training, therefore further increasing the total training cost, i.e., 2x data is used here compared to the vanilla training. (b) Different from AdvProp. Fast AdvProp only uses a small portion of the sampled batch to generate adversarial examples. Moreover, during the generation of adversarial images, the gradient calculation input images and the gradient calculation of network parameters are merged into the same forward and backward pass as in (Shafahi et al., 2019; Zhang et al., 2019), therefore making generating adversarial examples for "free".
In this paper, we present Fast AdvProp. which can run as cheaply as the vanilla training baseline in practice. In particular, noting the most costly training components in AdvProp are (1) generating adversarial examples where multiple forward passes and backward passes are additionally required, and (2) training with both clean samples and their adversarial counterparts therefore the size of training data gets doubled, Fast AdvProp revamps the original training pipeline as the following:
Firstly, though both clean training samples and their adversarial counterparts are default components in traditional adversarial training (Goodfellow et al., 2015; Kurakin et al., 2017), we argue such pairing behavior is not a fundamental request by AdvProp. Specifically, in Fast AdvProp, we reposition adversarial examples solely as bonus part for network training, i.e., networks now are expected to train with a mixture of a large portion of clean images and a small portion of adversarial examples. This adjustment on training data helps lower down training cost. Though the total number of adversarial training examples is reduced, our empirical results verify this strategy is sufficient to let networks gain robust feature representations.
Secondly, we integrate the recent techniques accelerating adversarial training (Wong et al., 2020; Zhang et al., 2019; Shafahi et al., 2019) into AdvProp, mainly for reducing the complexity of generating adversarial examples. However, this is non- non-trivial- maivelly adopting these fast adversarial training techniques will collapse the training, resulting in suboptimal model performance. We identify such failure is caused by the "label leaking" effect (Kurakin et al., 2017), which largely weakens the regularization power imposed by adversarial training samples. We further note this leakage comes from the intra-batch communication among training samples in the same mini-batch, and resolve it via shuffling BN (He al., 2020). Additionally, we find a) re-balancing the importance between clean training samples and adversarial training samples and b) synchronizing parameter updating speed are the other two key ingredients for ensuring Fast AdvProp's improvements.
Our empirical results demonstrate that Fast AdvProp can successfully improve recognition models for "free". For instance, without incurring any extra training cost, Fast AdvProp helps ResNet-50 (He et al., 2016) outperforms its vanilla counterpart by 0.3% on ImageNet, 2.1% on ImageNet-C, 1.9% on ImageNet-R and 0.5% on be observed when Fast AdvProp is applied to networks at different scales, combined with various data augmentation strategies, and adapted to other recognition tasks. By easing the computational barriers, we hope this work can encourage the community to further explore the potential of AdvProp (or adversarial learning in general) developing better deep learning models.
Furthermore, such "free lunch" can consistently
2

---

Published as a conference paper at ICLR 2022
2
RELATED WORK
Adversarial training. Adversarial training (Szegedy ct al., 2014; Goodfellow et al., 2015), which trains networks with adversarial examples that are generated on the fly, is one of the most effective ways for defending against adversarial attacks. Nonetheless, compared to vanilla training, adversarial training significantly increases the computational overhead, mainly due to the high complexity of generating adversarial examples.
To this end, many efforts have been devoted to accelerating adversarial training. Both (Shafahi et al., 2019) and (Zhang et al., 2019) propose to merge the gradient for adversarial attacks and the gradient for network parameter updates into a single forward and backward pass reduce computations. Wong et al. (Wong et al., 2020) alternatively argue that the cheapest adversarial attacker, Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015), actually can train robust classifiers, if combined with random initialization. This work is further enhanced by (Andriushchenko & Flammarion, 2020) to explicitly maximizing the gradient alignment inside the perturbation set for enhancing the quality of the FGSM solution. In this work, we aim to integrate these fast adversarial training techniques into AdvProp, for reducing the overhead of generating adversarial training samples.
Adversarial propagation. It is generally believed that adversarial training hurts generalization (Raghunathan et al., 2019). Adversarial propagation (AdvProp) (Xie et al., 2020), a special form of adversarial training, challenges this belief by showing training with adversarial examples actually can improve recognition models. The key is to utilize an additional set of batch normalization layers exclusively for the adversarial images (or more importantly, as suggested in (Chen et al., 2021a), la), by applying different set of rescaling parameters in batch normalization layers), as they have different underlying distributions to clean examples. Later works further explore the potential of AdvProp on other recognition tasks (Chen et al., 2021 b; Xu et al., 2021; Shu et al., 2020; Chen et al., 2021 Xie & Yuille, 2020; Wang et al., 2020; Gong et al., 2021), under different learning paradigms (Jiang et al., 2020; Ho & Vasconcelos, 2020; Xu & Yang, 2020), with different adversarial data (Merchant et al., 2020; Li et al., 2020; Hermann et al., 2021), enabling extremely large-batcr batch training (Liu et al., 2022), etc. In this paper, rather than furthering performance, we aim make AdvProp "free".
Data augmentation. Data augmentation, which effectively increases the size and the diversity of the training dataset, is crucial for the success of deep neural networks (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016). Popular ways for augmenting data include geometric transformations (e.g., translation, rotation), color jittering (e.g.. brightness, contrast), mixing images (Zhang et al., 2018; Yun et al., 2019; DeVries & Taylor, 2017), etc.
Training with adversarial examples (Goodfellow et al., 2015; Xie et al., 2020; Chen et al., 2021b) can be regarded a special way augment dafa--iiferent from traditional data augmentation strategies which are usually fixed and model agnostic, the policy of generating adversarial examples is jointly evolved with the model updating throughout the whole training process. This behavior ensures the augmentation policy of adversarial examples stays current and relevant. Nonetheless, a significant drawback of augmenting adversarial examples is that the introduced computational overhead much more expensive than that of traditional augmentation strategies, We hereby aim to make training with adversarial examples as cheap as other data augmentation strategies.
FAST ADVPROP
We hereby present Fast AdvProp. which aggressively revamps the costly components in AdvProp. Particularly, our modifications mainly focus on reducing the computational overheads stemmed from adversarial examples, meanwhile (empirically) still attempt to retain the benefits brought by AdvProp.
3.1 REVISITING ADVPROP
AdvProp (Xie et al., 2020) demonstrates adversarial examples can improve recognition models. By noticing adversarial images and clean images have different underlying distributions, AdvProp bridges such distribution mismatch by using two BN scheme- the original BN layers are applied exclusively for clean images, and the auxiliary BN layers are applied exclusively for adversarial images. This scheme ensures each BN layer is executed on a single data source (i.e., either clean images or adversarial images). More concretely, in each iteration,
3

---

Published as a conference paper at ICLR 2022
Table The performance of vanilla training, AdvProp, same-budget AdvProp, and Fast AdvProp on various datasets. 1/4 indicate the higher/lower the better.
IMAGENET f
IMAGENET-C
IMAGENET-R
S-IMAGENET 1
TRAINING BUDGET
Vanilla Training
76.2
58.5
36.3
7.8
1x
PGD-5 AdvProp
77.0
52.0
42.3
12.0
7x
15-epoch PGD-S AdvProp
66.8
66.2
31.7
8.6
1x
PGD-1 AdvProp
77.5
39.5
3x
35-epoch PGD-1 AdvProp
73.9
59.6
36.5
1x
Fast-AdvPrup (ours)
76.5
56.4
38.2
8.3
1x
Step 1: After randomly sampling subset of the training data, AdvProp applies the ad- versarial attack to these clean images for generating the corresponding adversarial images, using the auxiliary BN layers.
The clean images and their adversarial counterparts are then fed into the network as a pair. Specifically, the original BN layers are applied exclusively on the clean images, and the auxiliary BN layers are applied exclusively on the adversarial images.
Step 2:
Step 3: The loss from adversarial images and clean images are jointly optimized for updating network parameters.
As shown in (Xie et al., 2020), AdvProp substantially improves both the clean images accuracy, as well as the model robustness. We confirm it in our re-implementation-aa shown in the second row of Table 1, AdvProp, using PGD-5 attacker, helps ResNet-50 beats its vanilla counterpart by 0.8% ImageNet (Russakovsky et al., 2015), 6.5% on ImageNet-C (Hendrycks & Dietterich, 2018). 10), 6.0% on ImageNet-R (Hendrycks et al., 2020) and 4.2% on Boylized-ImageNee (Geirhos al., 2018).
But meanwhile, we note AdvProp significantly increases the training cost. For example, our AdvProp re-implementation requires 7x more forward and backward passes than the vanilla baseline. Such heavy training cost not only limits the broader exploration with AdvProp, but also makes the comparisons other learning strategies (which are usually "free". e.g., (Yun al., 2019; Zhang et al., 2018; Cubuk et al., 2019a)) seemly unfair.
To reduce the computational cost, we first give a naive attempt to simplify AdvProp's training pipeline. Specifically, given PGD-5 AdvProp here is 7 more expensive than the vanilla baseline, we directly cut its total training epochs by a factor of 7 (i.e., from 105 epochs to 15 epochs). As shown in the third row of Table 1, this 15-epoch PGD-5 AdvProp severely degrades the original AdvProp's performance (i.e., 66.8% vs. 77.0% on ImageNet), even making the resulted model attains much lower performance than the vanilla training baseline. Moreover, PGD-1 training (i.e., FGSM + random initialization as in (Wong et al., 2020)) to AdvProp still leads to inferior performance. These results demonstrate that the task of accelerating AdvProp is non-trivial, therefore motivate us to explore more sophisticated solutions next.
verify that applying the cheapest
3.2 LIGHTENING ADVPROP
We hereby carefully diagnose the design choices of AdvProp, aiming to simplify/purge its costly training components. Note that our ablations, we always keep the disentangled learning behavior with adversarial training samples unchanged (i.e., keep separate BN layers for adversarial samples and clean samples), as we assume this is the key for gaining robust features from adversarial examples.
3.2.1 UNPAIRING TRAINING SAMPLES IN ADVPROP
Let's consider the training cost of one epoch. We denote the cost of single forward and backward pass for one image as 1 and the dataset size as N. Then the cost of vanilla training for one epoch
cost Vanilla) = N.
During standard training step. compute the gradient for all parameters used in the forward process; while for attacking the network, we only compute the gradient with respect to the input images (therefore less computations). Nonetheless, we ignore such differences for simplification in cost calculation.
4

---

Published as a conference paper at ICLR 2022
Table 2: The performance of AdvProp with different settings. +ADvPROP denotes the original AdvProp. +1 ITER denotes the AdvProp with PGD-1 attacker. +DECOUPLED denotes the decoupled training where only small portion (e.g.. 20% here) of training images are used for generating adversarial examples. The last column reports the corresponding training budget in one epoch.
+ADvPROP
+1 ITER
+DECOUPLED
IMAGENET t
IMAGENET-C
TRAINING BUDGET
76.2
58.5
N
77.0
51.9
7N
/
77.5
54.3
3N
76.6
56.8
1,2N
Similarly, the cost of AdvProp using PGD-K K attack (Madry et al., 2018) for one epoch is:
cost(AdvProp) N + K + N N = (K + + x N,
(2)
where the first part (i.e., N) refers to the training cost of clean samples, the second part (i.e., K * N) refers to the cost of generating adversarial examples, and the third part (i.e., N) refers to the training cost of adversarial examples. Note AdvProp by default use K 5 (Xie et al., 2020), therefore increasing the training cost by a factor of 7 compared to the vanilla training baseline. This high training cost makes further scaling AdvProp to the conge-compliig settings (Xie et al., 2019; Mahajan et al., 2018; Dosovitskiy et al., 2020) challenging.
PGD-1 attack. Firstly, rather than using K 5, training cost from 7N to 3N. Note this change simplify the PGD attacker to the FGSM attacker (Goodfellow et al., 2015) with random noise inirialization. As shown Table 2. compared to the default AdvProp, this simplification increases the top-1 accuracy by 0.5% on ImageNet, but at the cost of sacrificing the robustness on ImageNet-C (i.e., 2.4 higher mCE).
K the default setting to reduce the
Decoupled training. Secondly, AdvProp implicitly introduces constraint that, for each clean image, we should generate a paired adversarial image for jointly training. Though such paired training behavior is popular and standard in adversarial training (Goodfellow et al., 2015), interestingly, we empirically find this is not necessarily needed, i.e., models can still be benefited from training with non non-paired clean images and adversarial images. Moreover, if we choose to break such pairing behavior, the training cost will be reduced to:
(Fatt AdvProp) Palean N + Padv * (K + 11) +
(3)
where Peleon is the percentages of training images used clean examples, and Pado is the percentages of training images used as adversarial examples. Note AdvProp by default sets Pelean Padv = 1. While for Fast AdvProp, we exclusively set Padv of the training samples for generating adversarial examples and keep the rest as clean examples. The training cost now is
cost(Fast AdvProp) = (1 - Podv) * N + Pado * (K + 1) N (Pado * K + 1) N
(4)
As shown in Table 2, by setting Pade = 0.2 and apply PGD-1 1 attacker, our Fast AdvProp not only largely reduces the training cost to 1. 1.2N, but also beats the vanilla training baseline by 0.4% top-1 accuracy on ImageNet and by 1.7 mCE ImageNet-C.
3.2.2 INCORPORATING FREE ADVERSARIAL TRAINING TECHNIQUES
Though decoupled training (using Equation 4) enables AdvProp to use the same number of training images as the vanilla setting, the resulted strategy still incurs extra training cost. This is because additional forward-backer backward passes are exclusively reserved for generating adversarial examples.
Inspired by the recent works on accelerating adversarial training by recycling the gradient (Shafahi et al., 2019; Zhang et al., 2019), we aim to improve the gradient utilization in generating adversarial examples, i.e., such gradients will also be used for updating network parameters. In this way, we are able to let networks additionally train with the samples that are in the intermediate state of generating adversarial examples (e.g.. clean image with random noise), and more importantly, for "free". Note that, following Shafahi et al. (2019), in order to re-using gradient, we need to switch the attack mode from targeted attack untargeted attack in Fast AdvProp.
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: Pseudo code of Fast AdvProp for T epochs, given some radius importance
re-weight parameter B3 learning rate y, and ratio of adversarial examples Podv-
Data: A set of clean images with labels;
Result: Network parameter!
for epoch 1,..., T (Pado + do
Sample clean image mini-batch with label Y
Split X into X1, X2 with the ratio 1 Padv and Pado, respectively
Generate the adversarial examples Xade using X₂
8 c() 98 < Val(a + 66, y,y,0)
8' < 8 siggngg))
Xade = X2 + clip(0'. 1.'00--0 EC
Calculate gradients for X2, Xudos note Po ghoise and 9s could be compute simultaneously
96 gglenn + Ex, Vol(z, 0)]
90 ggoise + ExEx, Vol(x (z + 6, y, 0)]
98 gfde +- E. VVl((x y, 0)]
Re-balance the gradients
g0 x gleean + 3 ghoise + B 90
ade
Update 1 using gradient descent
0 -- 0 - 790
end
As analyzed by Equation 4, the training cost of Fast AdvProp now is (Pade * K + 1) N for one epoch. To let the training cost of Fast AdvProp exactly match the training cost of the vanilla setting, we further calibrate the training by reducing the total number of epochs by a factor of (Pado K + 1).
Following the design principles above, interestingly, found naively recycling the gradients makes the training of Fast AdyProp unstable. Next, as summarized in Algorithm 1, we stabilize the network training using the following techniques:
Bridging the gap between the running/batch statistics in BN. Adversarial attackers by default attack models using their testing mode, e.g., the running statistics is applied in BN. Nonetheless, we observe unstable training with NaN loss if we reuse the gradient and using BN's running statistics at the same time. We conjecture this comes from the inconsistent gradient paths- we use the batch statistics for training with the clean examples and adversarial examples, but use the running statistics for generating adversarial examples. To remove that inconsistency, we now use the batch statistics for generating adversarial examples.
Shuffling BN for dealing with informa- tion leakage When attacking using batch statistics, we observe the training accuracy is unreasonably high, shown in Figure 2. The network could classify images by "cheating" since use the same batch for attacking and training, and the intra-batch information exchange introduced by BN leaks information. We use shuffling BN (He al., 2020) to resolve this problem. Before feeding the adversarial images on each GPU into the network, we shuffle the generated adversarial examples across the multiple GPUs. This ensures the batch statistics for the images with random noise and the adversarial images come from dif- ferent subsets, therefore preventing information leakage.
wie Shufflng BN
Train for anversrrial esample(())
Shuffling
80
60
20.
10
20
30
40
50
Epoch
Figure 2: Illustration of the information leakage.

---

Published as a conference paper at ICLR 2022
Re-balancing training samples. By reusing the gradient, the images used for adversarial attack involve two forward and backward passes (i.e., one for generating adversarial examples, and one for training with adversarial examples), while the clean images involve only one forward and backward pass. With the intuition that each image within the batch should have the same importance, we propose to halve the importance of the images used for adversarial attack. In another perspective, the adversarial images and their intermediate products (i.e., the images with random noise) are two kinds of augmentations on the original images. Based on the argument in (Hoffer et al., 2020) that the overall importance of an image should be the same w/ or w/o the repeated augmentations, we therefore need to adjust the importance of the images with random noise and the importance of the adversarial examples accordingly (i.e., setting 3 0.5 in Algorithm 1).
Synchronizing parameter updating speed. The decoupled training strategy enables us to substan- tially reduce the training cost. Nonetheless, it causes problems when combining with the auxiliary BNs scheme. Specifically, the parameters of original BNs only receive the gradients from clean images, the parameters of auxiliary BNs only receive the gradients from images with random noise and adversarial examples, while the parameters of the shared layers receive gradients from all ex- amples. The ratio between the gradient magnitude of shared layers/original BNs/auxiliary BNs is 1: (1 Padu): Podv- The inconsistent updating speed of network parameters harms the performance. To solve this problem, we re-se the gradient to ensure the similar parameter updating speed.
4 EXPERIMENTS
4.1 EXPERIMENTS SETUP
Dataset. We evaluate model performance on ImageNet classification, and the robustness dif- ferent specialized benchmarks including ImageNet-C. ImageNet-R, and Stylized ImageNet. Ima- geNet dataset contains 1.2 million training images and 50000 images for validation of 1000 classes. ImageNet-C measures the network robustness on 15 common corruption types, each with 5 severity, ImageNet-R contains stylish renditions like cartoons, art, and sketches of 200 ImageNet classes resulting in 30000 images. Eyylized-Imagell dataset keeps the global shape information while removing the local texture information using the AdaIN style transfer (Huang & Belongie, 2017).
Implementation Details We use the renowned ResNet family (He et al., 2016) as our default architectures. We use a SGD optimizer with momentum 0.9 and train for 105 epochs. The learning rate starts from 0.1 and decays at 30. 60, 90, 100 epochs by 0.1. We use batch size of 64 per GPU for vanilla training. For decoupled training setting, we use a batch size of 64/(1 Padv)
) per GPU, keeping the same 64 batch size per GPU for the original BNs. Pads is set to 0.2 if not specified.
For strictly fair comparison, we scale the total epochs and decay epochs by the relative training cost to vanilla training (i.e., Podu * K + 1 in Fast AdvProp, K + 2 in AdvProp) in the same budget setting. To generate adversarial images, we using the PGD attacker with random initialization. We attack for one step (K = 1) and set the perturbation size to 1.0. As discussed in Section 3.2.2, to ensure the same importance of each example within a batch, we set 3 0.5 for halving the importance of the images with random noise and the adversarial images. Additionally, we re-scale the gradient to achieve the 1: 1 1 ratio for ensuring similar updating speed of all parameters.
4.2 MAIN RESULTS
Table I compares our method with AdvProp and the vanilla training using ResNet-50. Firstly, our ieimplementation of AdvProp achieves 77.0% top-1 accuracy on ImageNet, 0.8% higher than the vanilla training baseline. We also evaluate robustness generalization on ImageNet-C, ImageNet- R, Stylized-ImageNet;; AdvProp also substantially outperforms the vanilla training baseline here. However, the comparison is unfair AdvProp using 7x training budget. In the fair comparison setting, the performance of 15-epoch AdvProp (dividing the total epochs by 7) degrades significantly, i.e,, its ImageNet accuracy is only 66.8%, which is 9.4% lower than the vanilla training baseline.
On the contrary, using exactly the same training budget as the vanilla setting, Fast AdvProp achieves 76.5% accuracy on ImageNet, which is 0.3% higher than the vanilla training baseline and significantly higher than the 15-epoch AdvProp. As shown in Table 1, our Fast AdvProp also shows stronger robustness than both the vanilla training baseline and the 15-epochs AdvProp.
7

---

Published as a conference paper at ICLR 2022
Table The ImageNet accuracy and robustness generalization vanilla training and Fast AdvProp.
IMAGENET
IMAGENET-CI
IMAGENET-R
S-IMAGENET
ResNet-50
76.2
58.5
36.3
7.8
15-epoch PGD-5 AdvProp
66.8
66.2
31.7
8.6
Fast AdvProp
76.5
56.4
38.2
8.3
ResNet-101
77.8
54.7
40.1
9.1
+ 15-epoch PGD-5 AdvProp
68.6
61.8
34.0
12.3
Fast AdvProp
77.8
51.7
41.0
10.8
ResNet-152
78.3
52.3
40.5
10.5
15-epoch PGD-5 AdvProp
69.0
59.6
36.2
13.2
AdvProp
78.6
42.0
12.4
Additionally, our method scales better with larger networks. As shown Table 3, Fast AdvProp helps the large 152 achieve 78.6% top-1 accuracy ImageNet. 49.8 mCE on ImageNet-C, 42.0% top-1 accuracy on ImageNet-R and 12.4% top-1 accuracy on Stylized-Iman Miaiiieedmmmaeeet,, beating its vanilla counterpart by 0.3% ImageNet, 2.5% ImageNet-C. 1.5% ImageNet-R and 1.9% on Erylized-ImageNer respectively. This observation also holds when comparing Fast AdvProp to the 15-epoch AdvProp baseline (which has the same training for all ResNet models, Fast AdvProp attains much higher performance than the 15-epoch AdvProp baseline on ImtgeNet, ImageNet-C and ImageNet-R. The only exception is where 15-epoch AdvProp always attains the best performance. We conjecture this is mainly due to larger percentage of adversarial data is used during training. therefore letting the learned feature representation be biased towards shape cues (Geirhos et al., 2018; Zhang & Zhu, 2019).
ours). Table shows that,
4.3 ABLATION STUDY
The importance decoupled training. In Table we take a close look at the effect of Podo- We can draw clear conclusion that the larger the Pade: the more inferior top-1 accuracy we achieved. Specifically, using 50% images as adversarial examples only gets 75.4% accuracy, which is 0.9% lower than the vanilla training. This comes from the fact that as Padv increases, the training cost in one epoch increases (calculated using Equation 4), therefore the total training epoch needs to be reduced for keeping the training
unchanged. In addition, reducing Pado from 0.20 to 0.11 does not further decrease the model accuracy, suggesting its value is fairly robust in a certain range. If we further decrease Pado 0, then our method degenerates the vanilla training baseline.
The decoupled training strategy enables us to train the networks with mixture of a large portion of clean images and small portion of adversarial examples, which helps the network go through the dataset with enough epochs. Based on Table 4, we choose Padv = 0.2 as our default setup. Table 4: Ablation study the influence of the percentage of the adversarial images.
Pode
IMAGENET
EPOCHS
0.00
76.2
105
76.5
94
0.20
76.5
87
0..3
76.0
79
0.50
75.4
70
The importance of example re-balancing and updating speed synchronization. In the default setting, the weight ratio between the clean/noise/adversarfi examples is without adjustment. With Pade 0.2, the ratio of updating speed between the shared/clean only/adversarial only parame- ters therefore is 0.8 0.2. We cannot observe any performance gain this setting as shown in Table 5, i.e., its ImageNet accuracy is 76.2%, almost the same the vanilla training.
We find is important to simultaneously adopt those two strategies, which achieves 76.53% ImageNet accuracy. Ignoring the re-balancing for the examples hurts the accuracy by 0. 18%; this may because some examples now are more important than the others, therefore violating the assumption that the overall importance for each example should be the same (Hoffer et al., 2020). Removing the updating speed synchronization hurts the performance as well, leading to 76.27% ImageNet accuracy, which is 0.26% lower than the setting with consistent parameter updating speed.
8

---

Published as a conference paper at ICLR 2022
Table 5: Ablation study of the effects on re-balancing samples and synchronizing updating speed.
SYNCHRONIZING U'DATE SPEED
SAME IMPORTANCE
IMAGENET
76.20
76.27
76.35
76.53
Information leakage problem. We find that shuffling BN is important for gradient reusing. Without shuffling BN, the training accuracy on the adversarial images reaches 88.0%, even higher than the training accuracy on the clean images (73.3%). This observation is counter-intuitive, as the adversarial images are much more difficult than the clean images. The shuffling BN technique resolves this problem effectively-we observe smooth and reasonable training curve in Figure 2. In addition, the validation accuracy boosts from 74.5% to 75.1%.
Combining with other data augmentations. AdvProp could be viewed as a data augmentation method from the perspective of increasing the size and diversity of the dataset using adversarial examples. We next combine Fast AdvProp with common data augmentation methods including Mixup (Zhang al., 2018) and CutMix (Yun al., 2019). Specifically, compared with the vanilla training baseline, Mixup and CutMix require extra training epochs to fully cope with their strong regularization. For Mixup, we train 180 epochs and decay the learning rate by 0.1 every 60 epochs. For CutMix, we train for 210 epochs and decay the learning rate at 75, 155 and 180, respectively. To properly integrate them with Fast AdyProp, we apply the extra data augmentations on clean images only, Note that the training cost is the same w/ w/o method.
Fast AdvProp achieves comparable performance on ImageNet but much better robustness. As shown in Table 6, when combining with CutMix, our method beats the baseline by 2.1% on ImageNet-C, 1.3% on ImageNet-R, 2.5% on lrfizedd-mmaeeet... These results suggest Fast AdyProp is compatible with existing data augmentations for furthering the network performance and robustness.
Table 6: Robustness when combining our Fast AdvProp with existing data augmentation methods.
IMAGENET-CI
IMAGENET-R R f
|S- S-IMAGENET
CutMix
58.9
35.2
5.5
+ Fast AdvProp
55.0
36.5
7.0
MixUp
53.4
41.0
10.0
Fast AdvProp
52.2
41.7
10.9
Object detection results. We implement Fast AdvProp on object detection and evaluate it on COCO dataset (Lin al., 2014). We adopt RetinaNet (Lin et al., 2020) as the detection framework without freezing the BN's rumning statistics. We train 24 epochs for the baseline and 20 epochs for Fast AdvProp. We benchmark both the standard performance on COCO and the robustness on COCO-C with 15 common corruption types (Michaelis et al., 2019). As shown in Table 7, compared to the baseline, our method achieves the similar performance on COCO detection, but much better robustness (i.e., +1.4%) in COCO-C.
Table 7: Object detection mAP(%) and robustness measurements on COCO and COCO-C.
COCO (mAP clean)
COCO-C (mAP corr.)
Vanilla Training
35.8
17.6
+ Adst-p
35.8
19.0
5 CONCLUSION
AdvProp is an effective method to get accurate and robust networks. However, it suffers from extremely high training costs. We propose the decoupled training where networks are expected to train with only small portion of adversarial examples but large portion of clean images, therefore reducing the training cost significantly. In addition, we succeed in reusing the gradient Fast AdvProp effectively and efficiently
when generating adversarial examples. Empirically, helps network gain better performance without incurring extra training cost. We believe our method will speed up the iteration of future works and accelerating the research adversarial learning.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work was supported by ONR N00014-21-1-2122. Cihang Xie was supported by a gift grant from Open Philanthropy.
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In NeurlPS, 2020.
Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. Sigh-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171, 2021.
Tianlong Chen, Yu Cheng, Zhe Gan, Jianfeng Wang, Lijuan Wang. Zhangyang Wang, and Jingjing Liu. Adversarial feature augmentation and normalization for visual recognition. arXiv preprint arXiv:2103.12171, 20
Xiangning Chen, Cihang Xie, Mingxing Tan, Li Zhang, Cho-Jui Hsich, and Boqing Gong, Robust and accurate object detection via adversarial learning. In CVPR, 2021b.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019a.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719 2019b.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint
04552, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2018.
Xinyu Gong, Wuyang Chen, Tianlong Chen, and Zhangyang Wang. Sandwich batch normalization. arXiv preprint 1r0. 1332, 2021.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamnessing adversarial examples. In ICLR, 2015.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint 180.0018, 2018.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of dit-----ittiittiion generalization. arXiv preprint arXiv:2006. 16241, 2020.
Charles Herrmann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krish- nan, and Deqing Sun. Pyramid adversarial training improves vit performance. arXiv preprint 1 2021.
Chih-Hui Ho and Nuno Vasconcelos. Contrastive learning with adversarial examples. arXiv preprint arXiv:2010. /2050, 2020.
10

---

Published as a conference paper at ICLR 2022
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR 2020, pp. IEEE, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. pp. 1510-1519, IEEE Computer Society, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. arXiv preprint arXiv:2010.13337 2020.
Alex Krizhevsky, Πly Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. NIPS, 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR, 2017.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. IJCV, 2020.
Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and Cihang Xie. Shape-texture debiased neural network training. arXiv preprint arXiv:2010.05981, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), ECCV, volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. PAMI, 42(2):318-33 327, 2020.
Yong Liu, Xiangning Chen, Minhao Cheng, Cho-Jui Hsich, and Yang You. Concurrent adversarial learning for large-batch training. In ICLR, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
Amil Merchant, Barret Zoph, and Ekin Dogus Cubuk. Does data augmentation benefit from split batchnorms. arXiv preprini arXiv:2010.07810. 2020.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv: 1907.0744 2019.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang. John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint 1rX1v:1906.06032, 06032, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurIPS, 2019.
Manli Shu, Zuxuan WU, Micah Goldblum, and Tom Goldstein. preparing for the worst: Making networks less brittle with adversarial batch normalization. arXiv preprint vrXiv:2009008655 2020.
11

---

Published as a conference paper at ICLR 2022
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2013.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Haotao Wang, Tianlong Chen, Shupeng Gui, Ting-Kuei Hu, Ji Liu, and Zhangyang Wang. Once-for- all adversarial training: In-situ tradeoff between robustness and accuracy for free. arXiv preprint arXiv:2010.11822 2020.
Eric Wong, Leslie Rice, and Zico Kolter. Fast better than free: Revisiting adversarial training. In ICLR, 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. /CLR, 2020.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoe V. Le. Adversarial examples improve image recognition. In CVPR, 2020.
Qizhe Xie, Eduard Hovy, Minh-Thame Luong, and Quoc Le. Self-training with noisy student improves imagenet classification. arXiv preprint
2019.
Cong Xu and Min Yang. arXiv:2012.13154 2020.
Adversarial momentum-contraativv pre-training. arXiv preprint
Weipeng Xu, Pengzhi Chu, Renhao Xie, Xiongziyan Xiao, and Hongcheng Huang. Robust and accurate object detection via self-knowledge distillation. arXiv preprint arXiv:2111.07239, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Painless adversarial training using maximal principle. arXiv preprint 17Xiv:1905.00877 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural networks. In ICML, 2019.
A APPENDIX
A.1 DO WE NEED RANDOM INITIALIZATION IN PGD ATTACK?
Here focus ablating the effects of the random initialization in PGD attacker on Fast AdvProp. Note that removing the random initialization simplifies the PGD-1 attacker to the FGSM attacker. Though, discussed in (Wong al., 2020), the random initialization is the key step for enabling successful adversarial training, we observe Fast AdvProp behaves robustly under such situation achieves 76.46% top-1 ImageNet accuracy (using FGSM), which closely matches the situation when random initialization is applied (i.e., 76.53% top-1 ImageNet accuracy, using PGD-1).
12


Reviews:
[{"review_id": "ayo3yDChlv", "reviewer": "Reviewer_e6SS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength \n1. The motivation of the paper is solid and the paper clearly quantifies and compares the training budget between the proposed method and AdvProp. \n2. Figure 1 and Algorithm is clear and demonstrates the overall idea of the paper.\n3. The paper is well organized and well presented.\n4. The proposed method demonstrates better performance under the same computation cost and generalizes to various tasks.\n5. The ablation study is thorough and cover different aspects of the proposed method. \n\nWeakness\n1. While reading the text, it is unclear what is the rationale of introducing g^{noise} in Algorithm 1. The optimization of g^{noise} could be similar to g^{clean}, but instead operating on X_2 without adding noise (changing the purple block to blue block in Figure 1(b)). What is the performance under such modification? The author is suggested to provide more detail about the introduction of g^{noise}.\n2. Table 4 shows the performance of various p_adv value. What is the performance for p_adv < 0.2 ?\n3. The robustness for advprop is suggested to add to Table 3 for comparison.\n", "summary_of_the_paper": "This paper aims to improve the training speed and decrease the computation cost of AdvProp, which is a method that leverages the adversarial example to improve the image recognition accuracy. AdvProp uses separate batchnorm for clean and adversarial examples respectively. In this work, the proposed method Fast AdvProp reduces the computation cost by reusing some gradient computation during training. In the experiment section, Fast AdvProp demonstrates better image recognition and object detection performance under the same training budget and can be combined with existing training strategies, such as mix up. Overall, this paper proposed an efficient training strategy that can be combined with various existing data augmentation for various tasks.", "summary_of_the_review": "This paper is enjoyable to read and provides good insight to the problem of large computation cost of AdvProp. The proposed solution Fast AdvProp demonstrates good performance with less computation cost and can potentially be applied to various vision tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "UMX25DFc0i-", "reviewer": "Reviewer_7oqV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of this paper: \n\nHighlighted several \"tricks\" to save on computational cost for AdvProp. I recognized that the discovery of these tricks and that they work to a certain extent is non-trivial work and appreciate those insights. Namely, they are (a) decoupling does not hurt as much, (b) 1-step adversarial does not cause as much degradation as previously thought, (c) recycling gradient works better with (i) non-targeted attack, (ii) using batch statistic for adversarial image generation, (iii) using shuffle BN to avoid label leaks, (iv) rebalancing examples and (v) synchronizing the update speed of parameters.\n\nThe paper is also written very clearly and the motivation is well communicated.\n\nWeaknesses of this paper:\n\nI felt there will be questions on what's the \"new scientific discovery\" here. This paper is mostly a report on empirical insights, that decoupling or reducing the number of adversarial samples does not really hurt performance that much. Most of the gradient recycling discussed in the paper is prior work but with engineering insights. \n\nOn experimental results. I take some issues with this \"The AdvProp also shows compelling robustness compared with the vanilla training, verifying its effectiveness. However, the comparison is unfair as AdvProp using 7× training budget. In the same budget setting, the performance of 15-epoch AdvProp (dividing the total epochs by 7) degrades significantly.\" The motivation of the paper is that you can retain the advantage of AdvProp without paying the computational cost, i.e., as the authors called it, a \"free lunch\". If you can't retain the advantage of advprop, then I question what is the purpose.", "summary_of_the_paper": "AdvProp is not efficient. This paper tries to overcome that with several measures: (1) decoupling, where they now do not need to have one adversarial sample with each clean image, (2) one step adversarial rather than multi-steps, and (3) reuse recent work on accelerating advprop such as recycling gradient, for which they provide several insights.", "summary_of_the_review": "I am ambivalent based on what I mentioned above. The paper provides empirical insights, but IMO not huge enough for ICLR, and is well written. However, I am wondering whether there is any new scientific discovery here -- it will be nice for those \"tricks\" mentioned in the paper, the authors can contrast clearly why they are novel as it seems like they have been previously proposed. I also question the AdvProp vs Fast AdvProp results where we have to significantly reduce AdvProp to have the Fast version beat it. I will be interested to see what the other reviewers think and I am opened to discussions and changing my feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RMSVtzjOTjs", "reviewer": "Reviewer_QrYp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper actually has resolved one of the main issues of AdvProp, making it more efficient. The empirical study shows very promising results. However, I still have some concerns about the work:\n\n1. The novelty and technique strength is limited. The way to combine clean and adversarial batches is empirical. The trick used for fast adversarial training has been explored by many other previous works. \n2. The empirical study only shows the effectiveness of fast AdvProp on the ResNet family, but not EfficientNets. In addition, on the detection task, the performance improvement is not that significant. \n3. Some claims and empirical supports of the paper are confusing. For example, [chen et al. 2021a] uses an adversarial feature augmentation instead of doing something like AdvProp to generate real adversarial samples. For me, AdvProp has more advantages to improve the clean accuracy. But the paper shows some experiments on the robustness side. \n\nA few notice about more technical details regarding AdvProp: 1. for the TensorFlow version, PGD-5 is better than PGD-1. But the Pytorch version [a] has a reversed conclusion; 2. the two BN trick was analyzed by the paper [b], I am not sure if the author has a similar observation on that. \n\na.https://github.com/tingxueronghua/pytorch-classification-advprop\nb.Adversarial Masking: Towards Understanding Robustness Trade-off for Generalization", "summary_of_the_paper": "This paper proposed an improved version of AdvProp, to speed up the training process. Specifically, it proposes to use less adversarial examples in a mixed batch and utilizes a list of recent fast adversarial training techniques. The experiments have been conducted on ImageNet sets with different backbones, demonstrating the effectiveness of the proposed fast model. ", "summary_of_the_review": "Overall, I think the paper is interesting but has some technical details to be addressed. In addition, the novelty of the paper is increamental. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BFj4CF-z1e-", "reviewer": "Reviewer_63kv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n- The paper is very well written and the used techniques are well explained and each of the modifications to Advprop is justified by ablation experiments.\n- The author(s) do(es) a good job at counting the number of forward- and backward propagations to demonstrate the increased performance of Fast Advprop compared to Advprop.\n- The overall experimental results are promising. The author(s) show(s) that Fast Advprop is able to improve the ResNet performance on some ImageNet-like datasets while keeping the training budget comparable to the budget for vanilla training. \n\nWEAKNESSES:\n- My main criticism is that the proposed Fast AdvProp is a potpourri of existing methods (adversarial propagation (Xie et al., 2020), free adversarial training (Shafahi et al., 2019) and Shuffling BN (He et al., 2020)) and simple modifications (splitting the training data into clean images and ones for which adversarial perturbations are calculated, rebalancing the losses of clean and adversarial images, synchronizing update speed) which limits the novelty of this approach. However, each of the modifications is well justified by ablation studies and the overall experimental evaluation is convincing.\n- Table 1 points out the gained/lost accuracy in comparison with the vanilla training for some of the entries. However, for most of the entries this difference is not stated. Is there a reason for this? Why are some of the differences printed in green and not in red? What does the red arrow indicate? This should be explained in the caption of the table.\n- Although, the methods used in this paper are well explained and justified, I would have liked some more explanations in Algorithm 1, e.g., which line corresponds to which paragraph. \"Fuse the gradients\" could for example be expressed as \"Rebalancing gradients\" to make the connection to the relevant paragraph.\n\nMINOR REMARKS:\n- missing word on page 6: \"When attack using running statistics and non-targeted attack\"\n- missing/wrong word on page 6: \"Before feed the adversarial images on each GPU into the network\" \n- page 8: \"shuffling BN\" vs. \"Shuffling BN\"\n- capitalization of references: \"equation 4\" vs. \"Table 3\"\n", "summary_of_the_paper": "The paper proposes Fast Advprop which is an modified implementation of adversarial propagation. Adversarial propagation aims at improving the robustness and generalization abilities of deep neural network classifiers by performing adversarial training with additional batch normalization layers that are solely used for the adversarial examples which are used during training. The paper suggests to apply different existing techniques and small modifications to increase the training speed while using AdvProp.", "summary_of_the_review": "Overall, I would recommend to accept this submission into ICLR 2022. The paper is well-written and the results are good. I would be happy to see some of the points, that I stated in my review, would be implemented in a future version of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper improves the training speed and decrease the computation cost of AdvProp, which is a method that leverages the adversarial example to improve the image recognition accuracy. The method achieves the speedup by leveraging a collection of practical heuristics, including reusing some gradient computation during training. The paper is well written, well justified with empirical supports, and can be potentially useful in many vision tasks. On the other hand, some novelty of the method is incremental, and the issues regarding empirical results and claims pointed out by the reviewers need  to be addressed in the revision.
Decision: Accept (Poster)