Fast AdvProp

1 INTRODUCTION . Deep neural networks are highly successful for visual recognition . As fueled by powerful computational resources and massive amounts of data , deep networks achieve compelling , sometimes even superhuman , performance on a wide range of visual benchmarks . However , when testing out of the box , these exemplary models are usually get criticized for lacking generalization/robustness—an increasing amount of works point out that deep neural networks are brittle at handling out-of-domain situations like natural image corruptions ( Hendrycks & Dietterich , 2018 ) , images with shifted styles ( Geirhos et al. , 2018 ; Hendrycks et al. , 2020 ) , etc . Adversarial propagation ( AdvProp ) ( Xie et al. , 2020 ) , which additionally feeds networks with adversarial examples during training , emerged as one of the most effective ways to train not only accurate but also robust deep neural networks . The key in AdvProp is to apply separate batch normalization ( BN ) layers ( Ioffe & Szegedy , 2015 ) to clean training samples and adversarial training samples , as they come from different underlying distributions . Later works further explore the potential of AdvProp on other learning tasks , including object detection ( Chen et al. , 2021b ; Xu et al. , 2021 ) , contrastive learning ( Jiang et al. , 2020 ; Ho & Vasconcelos , 2020 ; Xu & Yang , 2020 ) and large-batch training ( Liu et al. , 2022 ) . However , the benefits brought by AdvProp do not come for “ free ” —AdvProp introduces a significant amount of additional training cost , which is mainly incurred by generating and augmenting adversarial training samples . For instance , compared to the vanilla training baseline ( where only clean images are involved ) , the default setting in AdvProp ( Xie et al. , 2020 ) increase the total computational cost by factor of 7 , i.e. , 5/7 from generating adversarial examples , 1/7 from training adversarial examples , 1/7 from training clean images . This extremely high training cost not only limits the further explorations of AdvProp on larger networks ( Xie et al. , 2019 ; Brock et al. , 2021 ; Dosovitskiy et al. , 2020 ) , with larger datasets ( Sun et al. , 2017 ; Kuznetsova et al. , 2020 ) , and for different learning tasks , but also makes the direct comparisons against other low-cost learning algorithms ( Zhang et al. , 2018 ; DeVries & Taylor , 2017 ; Yun et al. , 2019 ; Cubuk et al. , 2019b ; a ) seemingly unfair . Sampled Batch + R an d o m N o is e Model Input “ Free ” operation Heavy operation Forward & Backward Clean Image Adversarial Image Image w/ Noise Sampled Batch Model Input ( a ) AdvProp ( b ) Fast AdvProp … Figure 1 : Comparison between AdvProp and Fast AdvProp . ( a ) AdvProp generates a paired adversarial image ( with color orange ) for each clean image ( with color blue ) in the sampled batch , therefore incurring heavy training cost . Moreover , in addition to clean images , adversarial images are also fed into networks for training , therefore further increasing the total training cost , i.e. , 2x data is used here compared to the vanilla training . ( b ) Different from AdvProp , Fast AdvProp only uses a small portion of the sampled batch to generate adversarial examples . Moreover , during the generation of adversarial images , the gradient calculation of input images and the gradient calculation of network parameters are merged into the same forward and backward pass as in ( Shafahi et al. , 2019 ; Zhang et al. , 2019 ) , therefore making generating adversarial examples for “ free ” . In this paper , we present Fast AdvProp , which can run as cheaply as the vanilla training baseline in practice . In particular , noting the most costly training components in AdvProp are ( 1 ) generating adversarial examples where multiple forward passes and backward passes are additionally required , and ( 2 ) training with both clean samples and their adversarial counterparts therefore the size of training data gets doubled , Fast AdvProp revamps the original training pipeline as the following : • Firstly , though both clean training samples and their adversarial counterparts are default components in traditional adversarial training ( Goodfellow et al. , 2015 ; Kurakin et al. , 2017 ) , we argue such pairing behavior is not a fundamental request by AdvProp . Specifically , in Fast AdvProp , we reposition adversarial examples solely as a bonus part for network training , i.e. , networks now are expected to train with a mixture of a large portion of clean images and a small portion of adversarial examples . This adjustment on training data helps lower down training cost . Though the total number of adversarial training examples is reduced , our empirical results verify this strategy is sufficient to let networks gain robust feature representations . • Secondly , we integrate the recent techniques on accelerating adversarial training ( Wong et al. , 2020 ; Zhang et al. , 2019 ; Shafahi et al. , 2019 ) into AdvProp , mainly for reducing the complexity of generating adversarial examples . However , this is non-trivial—naively adopting these fast adversarial training techniques will collapse the training , resulting in suboptimal model performance . We identify such failure is caused by the “ label leaking ” effect ( Kurakin et al. , 2017 ) , which largely weakens the regularization power imposed by adversarial training samples . We further note this leakage comes from the intra-batch communication among training samples in the same mini-batch , and resolve it via shuffling BN ( He et al. , 2020 ) . Additionally , we find a ) re-balancing the importance between clean training samples and adversarial training samples and b ) synchronizing parameter updating speed are the other two key ingredients for ensuring Fast AdvProp ’ s improvements . Our empirical results demonstrate that Fast AdvProp can successfully improve recognition models for “ free ” . For instance , without incurring any extra training cost , Fast AdvProp helps ResNet-50 ( He et al. , 2016 ) outperforms its vanilla counterpart by 0.3 % on ImageNet , 2.1 % on ImageNet-C , 1.9 % on ImageNet-R and 0.5 % on Stylized-ImageNet . Furthermore , such “ free lunch ” can consistently be observed when Fast AdvProp is applied to networks at different scales , combined with various data augmentation strategies , and adapted to other recognition tasks . By easing the computational barriers , we hope this work can encourage the community to further explore the potential of AdvProp ( or adversarial learning in general ) on developing better deep learning models . 2 RELATED WORK . Adversarial training . Adversarial training ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , which trains networks with adversarial examples that are generated on the fly , is one of the most effective ways for defending against adversarial attacks . Nonetheless , compared to vanilla training , adversarial training significantly increases the computational overhead , mainly due to the high complexity of generating adversarial examples . To this end , many efforts have been devoted to accelerating adversarial training . Both ( Shafahi et al. , 2019 ) and ( Zhang et al. , 2019 ) propose to merge the gradient for adversarial attacks and the gradient for network parameter updates into a single forward and backward pass to reduce computations . Wong et al . ( Wong et al. , 2020 ) alternatively argue that the cheapest adversarial attacker , Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) , actually can train robust classifiers , if combined with random initialization . This work is further enhanced by ( Andriushchenko & Flammarion , 2020 ) to explicitly maximizing the gradient alignment inside the perturbation set for enhancing the quality of the FGSM solution . In this work , we aim to integrate these fast adversarial training techniques into AdvProp , for reducing the overhead of generating adversarial training samples . Adversarial propagation . It is generally believed that adversarial training hurts generalization ( Raghunathan et al. , 2019 ) . Adversarial propagation ( AdvProp ) ( Xie et al. , 2020 ) , a special form of adversarial training , challenges this belief by showing training with adversarial examples actually can improve recognition models . The key is to utilize an additional set of batch normalization layers exclusively for the adversarial images , as they have different underlying distributions to clean examples . Later works further explore the potential of AdvProp on other recognition tasks ( Chen et al. , 2021b ; Xu et al. , 2021 ; Shu et al. , 2020 ; Chen et al. , 2021a ; Xie & Yuille , 2020 ; Wang et al. , 2020 ; Gong et al. , 2021 ) , under different learning paradigms ( Jiang et al. , 2020 ; Ho & Vasconcelos , 2020 ; Xu & Yang , 2020 ) , with different adversarial data ( Merchant et al. , 2020 ; Li et al. , 2020 ; Herrmann et al. , 2021 ) , enabling extremely large-batch training ( Liu et al. , 2022 ) , etc . In this work , rather than furthering AdvProp on performance , we aim to develop a “ free ” version of it . Data augmentation . Data augmentation , which effectively increases the size and the diversity of the training dataset , is crucial for the success of deep neural networks ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ) . Popular ways for augmenting data include geometric transformations ( e.g. , translation , rotation ) , color jittering ( e.g. , brightness , contrast ) , mixing images ( Zhang et al. , 2018 ; Yun et al. , 2019 ; DeVries & Taylor , 2017 ) , etc . Training with adversarial examples ( Goodfellow et al. , 2015 ; Xie et al. , 2020 ; Chen et al. , 2021b ) can be regarded as a special way to augment data—different from traditional data augmentation strategies which are usually fixed and model agnostic , the policy of generating adversarial examples is jointly evolved with the model updating throughout the whole training process . This behavior ensures the augmentation policy of adversarial examples stays current and relevant . Nonetheless , a significant drawback of augmenting adversarial examples is that the introduced computational overhead is much more expensive than that of traditional augmentation strategies . We hereby aim to make training with adversarial examples as cheap as other data augmentation strategies . 3 FAST ADVPROP . We hereby present Fast AdvProp , which aggressively revamps the costly components in AdvProp . Particularly , our modifications mainly focus on reducing the computational overheads stemmed from adversarial examples , meanwhile ( empirically ) still attempt to retain the benefits brought by AdvProp . 3.1 REVISITING ADVPROP . AdvProp ( Xie et al. , 2020 ) demonstrates adversarial examples can improve recognition models . By noticing adversarial images and clean images have different underlying distributions , AdvProp bridges such distribution mismatch by using two BN scheme—the original BN layers are applied exclusively for clean images , and the auxiliary BN layers are applied exclusively for adversarial images . This scheme ensures each BN layer is executed on a single data source ( i.e. , either clean images or adversarial images ) . More concretely , in each iteration , As shown in ( Xie et al. , 2020 ) , AdvProp substantially improves both the clean images accuracy , as well as the model robustness . We confirm it in our re-implementation—as shown in the second row of Table 1 , AdvProp , using PGD-5 attacker , helps ResNet-50 beats its vanilla counterpart by 0.8 % on ImageNet ( Russakovsky et al. , 2015 ) , 6.5 % on ImageNet-C ( Hendrycks & Dietterich , 2018 ) , 6.0 % on ImageNet-R ( Hendrycks et al. , 2020 ) and 4.2 % on Stylized-ImageNet ( Geirhos et al. , 2018 ) . But meanwhile , we note AdvProp significantly increases the training cost . For example , our AdvProp re-implementation requires 7× more forward and backward passes than the vanilla baseline . Such heavy training cost not only limits the broader exploration with AdvProp , but also makes the comparisons to other learning strategies ( which are usually “ free ” , e.g. , ( Yun et al. , 2019 ; Zhang et al. , 2018 ; Cubuk et al. , 2019a ) ) seemly unfair . To reduce the computational cost , we first give a naive attempt to simplify AdvProp ’ s training pipeline . Specifically , given PGD-5 AdvProp here is 7× more expensive than the vanilla baseline , we directly cut its total training epochs by a factor of 7 ( i.e. , from 105 epochs to 15 epochs ) . As shown in the third row of Table 1 , this 15-epoch PGD-5 AdvProp severely degrades the original AdvProp ’ s performance ( i.e. , 66.8 % vs. 77.0 % on ImageNet ) , even making the resulted model attains much lower performance than the vanilla training baseline . Moreover , we verify that applying the cheapest PGD-1 training ( i.e. , FGSM + random initialization as in ( Wong et al. , 2020 ) ) to AdvProp still leads to inferior performance . These results demonstrate that the task of accelerating AdvProp is non-trivial , therefore motivate us to explore more sophisticated solutions next .

Summary: This paper aims to improve the training speed and decrease the computation cost of AdvProp, which is a method that leverages the adversarial example to improve the image recognition accuracy. AdvProp uses separate batchnorm for clean and adversarial examples respectively. In this work, the proposed method Fast AdvProp reduces the computation cost by reusing some gradient computation during training. In the experiment section, Fast AdvProp demonstrates better image recognition and object detection performance under the same training budget and can be combined with existing training strategies, such as mix up. Overall, this paper proposed an efficient training strategy that can be combined with various existing data augmentation for various tasks.