Fast AdvProp

1 INTRODUCTION . Deep neural networks are highly successful for visual recognition . As fueled by powerful computational resources and massive amounts of data , deep networks achieve compelling , sometimes even superhuman , performance on a wide range of visual benchmarks . However , when testing out of the box , these exemplary models are usually get criticized for lacking generalization/robustness—an increasing amount of works point out that deep neural networks are brittle at handling out-of-domain situations like natural image corruptions ( Hendrycks & Dietterich , 2018 ) , images with shifted styles ( Geirhos et al. , 2018 ; Hendrycks et al. , 2020 ) , etc . Adversarial propagation ( AdvProp ) ( Xie et al. , 2020 ) , which additionally feeds networks with adversarial examples during training , emerged as one of the most effective ways to train not only accurate but also robust deep neural networks . The key in AdvProp is to apply separate batch normalization ( BN ) layers ( Ioffe & Szegedy , 2015 ) to clean training samples and adversarial training samples , as they come from different underlying distributions . Later works further explore the potential of AdvProp on other learning tasks , including object detection ( Chen et al. , 2021b ; Xu et al. , 2021 ) , contrastive learning ( Jiang et al. , 2020 ; Ho & Vasconcelos , 2020 ; Xu & Yang , 2020 ) and large-batch training ( Liu et al. , 2022 ) . However , the benefits brought by AdvProp do not come for “ free ” —AdvProp introduces a significant amount of additional training cost , which is mainly incurred by generating and augmenting adversarial training samples . For instance , compared to the vanilla training baseline ( where only clean images are involved ) , the default setting in AdvProp ( Xie et al. , 2020 ) increase the total computational cost by factor of 7 , i.e. , 5/7 from generating adversarial examples , 1/7 from training adversarial examples , 1/7 from training clean images . This extremely high training cost not only limits the further explorations of AdvProp on larger networks ( Xie et al. , 2019 ; Brock et al. , 2021 ; Dosovitskiy et al. , 2020 ) , with larger datasets ( Sun et al. , 2017 ; Kuznetsova et al. , 2020 ) , and for different learning tasks , but also makes the direct comparisons against other low-cost learning algorithms ( Zhang et al. , 2018 ; DeVries & Taylor , 2017 ; Yun et al. , 2019 ; Cubuk et al. , 2019b ; a ) seemingly unfair . Sampled Batch + R an d o m N o is e Model Input “ Free ” operation Heavy operation Forward & Backward Clean Image Adversarial Image Image w/ Noise Sampled Batch Model Input ( a ) AdvProp ( b ) Fast AdvProp … Figure 1 : Comparison between AdvProp and Fast AdvProp . ( a ) AdvProp generates a paired adversarial image ( with color orange ) for each clean image ( with color blue ) in the sampled batch , therefore incurring heavy training cost . Moreover , in addition to clean images , adversarial images are also fed into networks for training , therefore further increasing the total training cost , i.e. , 2x data is used here compared to the vanilla training . ( b ) Different from AdvProp , Fast AdvProp only uses a small portion of the sampled batch to generate adversarial examples . Moreover , during the generation of adversarial images , the gradient calculation of input images and the gradient calculation of network parameters are merged into the same forward and backward pass as in ( Shafahi et al. , 2019 ; Zhang et al. , 2019 ) , therefore making generating adversarial examples for “ free ” . In this paper , we present Fast AdvProp , which can run as cheaply as the vanilla training baseline in practice . In particular , noting the most costly training components in AdvProp are ( 1 ) generating adversarial examples where multiple forward passes and backward passes are additionally required , and ( 2 ) training with both clean samples and their adversarial counterparts therefore the size of training data gets doubled , Fast AdvProp revamps the original training pipeline as the following : • Firstly , though both clean training samples and their adversarial counterparts are default components in traditional adversarial training ( Goodfellow et al. , 2015 ; Kurakin et al. , 2017 ) , we argue such pairing behavior is not a fundamental request by AdvProp . Specifically , in Fast AdvProp , we reposition adversarial examples solely as a bonus part for network training , i.e. , networks now are expected to train with a mixture of a large portion of clean images and a small portion of adversarial examples . This adjustment on training data helps lower down training cost . Though the total number of adversarial training examples is reduced , our empirical results verify this strategy is sufficient to let networks gain robust feature representations . • Secondly , we integrate the recent techniques on accelerating adversarial training ( Wong et al. , 2020 ; Zhang et al. , 2019 ; Shafahi et al. , 2019 ) into AdvProp , mainly for reducing the complexity of generating adversarial examples . However , this is non-trivial—naively adopting these fast adversarial training techniques will collapse the training , resulting in suboptimal model performance . We identify such failure is caused by the “ label leaking ” effect ( Kurakin et al. , 2017 ) , which largely weakens the regularization power imposed by adversarial training samples . We further note this leakage comes from the intra-batch communication among training samples in the same mini-batch , and resolve it via shuffling BN ( He et al. , 2020 ) . Additionally , we find a ) re-balancing the importance between clean training samples and adversarial training samples and b ) synchronizing parameter updating speed are the other two key ingredients for ensuring Fast AdvProp ’ s improvements . Our empirical results demonstrate that Fast AdvProp can successfully improve recognition models for “ free ” . For instance , without incurring any extra training cost , Fast AdvProp helps ResNet-50 ( He et al. , 2016 ) outperforms its vanilla counterpart by 0.3 % on ImageNet , 2.1 % on ImageNet-C , 1.9 % on ImageNet-R and 0.5 % on Stylized-ImageNet . Furthermore , such “ free lunch ” can consistently be observed when Fast AdvProp is applied to networks at different scales , combined with various data augmentation strategies , and adapted to other recognition tasks . By easing the computational barriers , we hope this work can encourage the community to further explore the potential of AdvProp ( or adversarial learning in general ) on developing better deep learning models . 2 RELATED WORK . Adversarial training . Adversarial training ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , which trains networks with adversarial examples that are generated on the fly , is one of the most effective ways for defending against adversarial attacks . Nonetheless , compared to vanilla training , adversarial training significantly increases the computational overhead , mainly due to the high complexity of generating adversarial examples . To this end , many efforts have been devoted to accelerating adversarial training . Both ( Shafahi et al. , 2019 ) and ( Zhang et al. , 2019 ) propose to merge the gradient for adversarial attacks and the gradient for network parameter updates into a single forward and backward pass to reduce computations . Wong et al . ( Wong et al. , 2020 ) alternatively argue that the cheapest adversarial attacker , Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) , actually can train robust classifiers , if combined with random initialization . This work is further enhanced by ( Andriushchenko & Flammarion , 2020 ) to explicitly maximizing the gradient alignment inside the perturbation set for enhancing the quality of the FGSM solution . In this work , we aim to integrate these fast adversarial training techniques into AdvProp , for reducing the overhead of generating adversarial training samples . Adversarial propagation . It is generally believed that adversarial training hurts generalization ( Raghunathan et al. , 2019 ) . Adversarial propagation ( AdvProp ) ( Xie et al. , 2020 ) , a special form of adversarial training , challenges this belief by showing training with adversarial examples actually can improve recognition models . The key is to utilize an additional set of batch normalization layers exclusively for the adversarial images , as they have different underlying distributions to clean examples . Later works further explore the potential of AdvProp on other recognition tasks ( Chen et al. , 2021b ; Xu et al. , 2021 ; Shu et al. , 2020 ; Chen et al. , 2021a ; Xie & Yuille , 2020 ; Wang et al. , 2020 ; Gong et al. , 2021 ) , under different learning paradigms ( Jiang et al. , 2020 ; Ho & Vasconcelos , 2020 ; Xu & Yang , 2020 ) , with different adversarial data ( Merchant et al. , 2020 ; Li et al. , 2020 ; Herrmann et al. , 2021 ) , enabling extremely large-batch training ( Liu et al. , 2022 ) , etc . In this work , rather than furthering AdvProp on performance , we aim to develop a “ free ” version of it . Data augmentation . Data augmentation , which effectively increases the size and the diversity of the training dataset , is crucial for the success of deep neural networks ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ) . Popular ways for augmenting data include geometric transformations ( e.g. , translation , rotation ) , color jittering ( e.g. , brightness , contrast ) , mixing images ( Zhang et al. , 2018 ; Yun et al. , 2019 ; DeVries & Taylor , 2017 ) , etc . Training with adversarial examples ( Goodfellow et al. , 2015 ; Xie et al. , 2020 ; Chen et al. , 2021b ) can be regarded as a special way to augment data—different from traditional data augmentation strategies which are usually fixed and model agnostic , the policy of generating adversarial examples is jointly evolved with the model updating throughout the whole training process . This behavior ensures the augmentation policy of adversarial examples stays current and relevant . Nonetheless , a significant drawback of augmenting adversarial examples is that the introduced computational overhead is much more expensive than that of traditional augmentation strategies . We hereby aim to make training with adversarial examples as cheap as other data augmentation strategies . 3 FAST ADVPROP . We hereby present Fast AdvProp , which aggressively revamps the costly components in AdvProp . Particularly , our modifications mainly focus on reducing the computational overheads stemmed from adversarial examples , meanwhile ( empirically ) still attempt to retain the benefits brought by AdvProp . 3.1 REVISITING ADVPROP . AdvProp ( Xie et al. , 2020 ) demonstrates adversarial examples can improve recognition models . By noticing adversarial images and clean images have different underlying distributions , AdvProp bridges such distribution mismatch by using two BN scheme—the original BN layers are applied exclusively for clean images , and the auxiliary BN layers are applied exclusively for adversarial images . This scheme ensures each BN layer is executed on a single data source ( i.e. , either clean images or adversarial images ) . More concretely , in each iteration , As shown in ( Xie et al. , 2020 ) , AdvProp substantially improves both the clean images accuracy , as well as the model robustness . We confirm it in our re-implementation—as shown in the second row of Table 1 , AdvProp , using PGD-5 attacker , helps ResNet-50 beats its vanilla counterpart by 0.8 % on ImageNet ( Russakovsky et al. , 2015 ) , 6.5 % on ImageNet-C ( Hendrycks & Dietterich , 2018 ) , 6.0 % on ImageNet-R ( Hendrycks et al. , 2020 ) and 4.2 % on Stylized-ImageNet ( Geirhos et al. , 2018 ) . But meanwhile , we note AdvProp significantly increases the training cost . For example , our AdvProp re-implementation requires 7× more forward and backward passes than the vanilla baseline . Such heavy training cost not only limits the broader exploration with AdvProp , but also makes the comparisons to other learning strategies ( which are usually “ free ” , e.g. , ( Yun et al. , 2019 ; Zhang et al. , 2018 ; Cubuk et al. , 2019a ) ) seemly unfair . To reduce the computational cost , we first give a naive attempt to simplify AdvProp ’ s training pipeline . Specifically , given PGD-5 AdvProp here is 7× more expensive than the vanilla baseline , we directly cut its total training epochs by a factor of 7 ( i.e. , from 105 epochs to 15 epochs ) . As shown in the third row of Table 1 , this 15-epoch PGD-5 AdvProp severely degrades the original AdvProp ’ s performance ( i.e. , 66.8 % vs. 77.0 % on ImageNet ) , even making the resulted model attains much lower performance than the vanilla training baseline . Moreover , we verify that applying the cheapest PGD-1 training ( i.e. , FGSM + random initialization as in ( Wong et al. , 2020 ) ) to AdvProp still leads to inferior performance . These results demonstrate that the task of accelerating AdvProp is non-trivial , therefore motivate us to explore more sophisticated solutions next .

Summary: This paper proposed an improved version of AdvProp, to speed up the training process. Specifically, it proposes to use less adversarial examples in a mixed batch and utilizes a list of recent fast adversarial training techniques. The experiments have been conducted on ImageNet sets with different backbones, demonstrating the effectiveness of the proposed fast model.