Defending Against Image Corruptions Through Adversarial Augmentations

1 INTRODUCTION . By following a process known as Empirical Risk Minimization ( ERM ) ( Vapnik , 1998 ) , neural networks are trained to minimize the average error on a training set . ERM has enabled breakthroughs in a wide variety of fields and applications ( Goodfellow et al. , 2016 ; Krizhevsky et al. , 2012 ; Hinton et al. , 2012 ) , ranging from ranking content on the web ( Covington et al. , 2016 ) to autonomous driving ( Bojarski et al. , 2016 ) via medical diagnostics ( De Fauw et al. , 2018 ) . ERM is based on the principle that the data used during training is independently drawn from the same distribution as the one encountered during deployment . In practice , however , training and deployment data may differ and models can fail catastrophically . Such occurrence is commonplace as training data is often collected through a biased process that highlights confounding factors and spurious correlations ( Torralba et al. , 2011 ; Kuehlkamp et al. , 2017 ) , which can lead to undesirable consequences ( e.g. , http : //gendershades.org ) . As such , it has become increasingly important to ensure that deployed models are robust and generalize to various input corruptions . Unfortunately , even small corruptions can significantly affect the performance of existing classifiers . For example , Recht et al . ( 2019 ) ; Hendrycks et al . ( 2019 ) show that the accuracy of IMAGENET models is severely impacted by changes in the data collection process , while imperceptible deviations to the input , called adversarial perturbations , can cause neural networks to make incorrect predictions with high confidence ( Carlini & Wagner , 2017a ; b ; Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ; Szegedy et al. , 2014 ) . Methods to counteract such effects , which mainly consist of using random or adversarially-chosen data augmentations , struggle . Training against corrupted data only forces the memorization of such corruptions and , as a result , these models fail to generalize to new corruptions ( Vasiljevic et al. , 2016 ; Geirhos et al. , 2018 ) . Recent work from Hendrycks et al . ( 2020b ) ( also known as AugMix ) argues that basic pre-defined corruptions can be composed to improve the robustness of models to common corruptions . Another line of work , DeepAugment ( Hendrycks et al. , 2020a ) , corrupts images by passing them through two specific image-to-image models while distorting the models ’ parameters and activations using an extensive range of manually defined heuristic operations . While both methods perform well on average on the common corruptions present in CIFAR-10-C and IMAGENET-C , they generalize poorly to the adversarial setting . Most recently , Laidlaw et al . ( 2021 ) proposed an adversarial training method based on bounding a neural perceptual distance ( i.e. , an approximation of the true perceptual distance ) , under the acronym of PAT for Perceptual Adversarial Training . Their method performs well against five diverse adversarial attacks , but , as it specifically addresses robustness to pixel-level attacks that directly manipulate image pixels , it performs worse than AugMix on common corruptions . In this work , we address this gap . We focus on training models that are robust to adversarially-chosen corruptions that preserve semantic content . We go beyond conventional random data augmentation schemes ( exemplified by Hendrycks et al. , 2020b ; a ) and adversarial training ( exemplified by Madry et al. , 2018 ; Gowal et al. , 2019 ; Laidlaw et al. , 2021 ) by leveraging image-to-image models that can produce a wide range of semantically-preserving corruptions ; in contrast to related works , our method does not require the manual creation of heuristic transformations . Our contributions are as follows : • We formulate an adversarial training procedure , named AdversarialAugment ( or AdA for short ) which finds adversarial examples by optimizing over the weights of any pre-trained image-to-image model ( i.e . over the weights of arbitrary autoencoders ) . • We give sufficient conditions for the consistency of idealized versions of our method and DeepAugment , and provide PAC-Bayesian performance guarantees , following Neyshabur et al . ( 2017 ) . Our theoretical considerations highlight the potential advantages of AdA over previous work ( DeepAugment ) , as well as the combination of the two . We also establish links to Invariant Risk Minimization ( IRM ) ( Arjovsky et al. , 2020 ) , Adversarial Mixing ( AdvMix ) ( Gowal et al. , 2019 ) and Perceptual Adversarial Training ( Laidlaw et al. , 2021 ) . • We improve upon the known state-of-the-art on CIFAR-10-C by achieving a mean corruption error ( mCE ) of 7.83 % when using our method in conjunction with others ( vs. 23.51 % for Perceptual Adversarial Training ( PAT ) , 10.90 % for AugMix and 8.11 % for DeepAugment ) . On IMAGENET we show that our method can leverage 4 pre-trained image-to-image models simultaneously ( VQVAE ( van den Oord et al. , 2017 ) , U-Net ( Ronneberger et al. , 2015 ) , EDSR ( Lim et al. , 2017 ) & CAE ( Theis et al. , 2017 ) ) to yield the largest increase in robustness to common image corruptions , among all evaluated models . • On ` 2 and ` ∞ norm-bounded perturbations we significantly improve upon previous work ( DeepAugment & AugMix ) using AdA ( EDSR ) , while slightly improving generalization performance on both IMAGENET-V2 and on CIFAR-10.1 . 2 RELATED WORK . Data augmentation . Data augmentation has been shown to reduce the generalization error of standard ( non-robust ) training . For image classification tasks , random flips , rotations and crops are commonly used ( He et al. , 2016a ) . More sophisticated techniques such as Cutout of DeVries & Taylor ( 2017 ) ( which produces random occlusions ) , CutMix of Yun et al . ( 2019 ) ( which replaces parts of an image with another ) and mixup of Zhang et al . ( 2018a ) ; Tokozume et al . ( 2018 ) ( which linearly interpolates between two images ) all demonstrate extremely compelling results . Guo et al . ( 2019 ) improved upon mixup by proposing an adaptive mixing policy . Works , such as AutoAugment ( Cubuk et al. , 2019 ) and the related RandAugment ( Cubuk et al. , 2020 ) , learn augmentation policies from data directly . These methods are tuned to improve standard classification accuracy and have been shown to work well on CIFAR-10 , CIFAR-100 , SVHN and IMAGENET . However , these approaches do not necessarily generalize well to larger data shifts and perform poorly on benign corruptions such as blur or speckle noise ( Taori et al. , 2020 ) . Robustness to synthetic and natural data shift . Several works argue that training against corrupted data only forces the memorization of such corruptions and , as a result , models fail to generalize to new corruptions ( Vasiljevic et al. , 2016 ; Geirhos et al. , 2018 ) . This has not prevented Geirhos et al . ( 2019 ) ; Yin et al . ( 2019 ) ; Hendrycks et al . ( 2020b ) ; Lopes et al . ( 2019 ) ; Hendrycks et al . ( 2020a ) from demonstrating that some forms of data augmentation can improve the robustness of models on IMAGENET-C , despite not being directly trained on these common corruptions . Most works on the topic focus on training models that perform well in expectation . Unfortunately , these models remain vulnerable to more drastic adversarial shifts ( Taori et al. , 2020 ) . Robustness to adversarial data shift . Adversarial data shift has been extensively studied ( Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ; Szegedy et al. , 2014 ; Moosavi-Dezfooli et al. , 2019 ; Papernot et al. , 2016 ; Madry et al. , 2018 ) . Most works focus on the robustness of classifiers to ` p-norm bounded perturbations . In particular , it is expected that a robust classifier should be invariant to small perturbations in the pixel space ( as defined by the ` p-norm ) . Goodfellow et al . ( 2015 ) and Madry et al . ( 2018 ) laid down foundational principles to train robust networks , and recent works ( Zhang et al. , 2019 ; Qin et al. , 2019 ; Rice et al. , 2020 ; Wu et al. , 2020 ; Gowal et al. , 2020 ) continue to find novel approaches to enhance adversarial robustness . However , approaches focused on ` p-norm bounded perturbations often sacrifice accuracy on non-adversarial images ( Raghunathan et al. , 2019 ) . Several works ( Baluja & Fischer , 2017 ; Song et al. , 2018 ; Xiao et al. , 2018 ; Qiu et al. , 2019 ; Wong & Kolter , 2021b ; Laidlaw et al. , 2021 ) go beyond these analytically defined perturbations and demonstrate that it is not only possible to maintain accuracy on non-adversarial images but also to reduce the effect of spurious correlations and reduce bias ( Gowal et al. , 2019 ) . Unfortunately , most aforementioned approaches perform poorly on CIFAR-10-C and IMAGENET-C . 3 DEFENSE AGAINST ADVERSARIAL CORRUPTIONS . In this section , we introduce AdA , our approach for training models robust to image corruptions through the use of adversarial augmentations while leveraging pre-trained autoencoders . In Appendix A we detail how our work relates to AugMix ( Hendrycks et al. , 2020b ) , DeepAugment ( Hendrycks et al. , 2020a ) , Invariant Risk Minimization ( Arjovsky et al. , 2020 ) , Adversarial Mixing ( Gowal et al. , 2019 ) and Perceptual Adversarial Training ( Laidlaw et al. , 2021 ) . Corrupted adversarial risk . We consider a model fθ : X → Y parametrized by θ . Given a dataset D ⊂ X × Y over pairs of examples x and corresponding labels y , we would like to find the parameters θ which minimize the corrupted adversarial risk : E ( x , y ) ∼D [ max x′∈C ( x ) L ( fθ ( x ′ ) , y ) ] , ( 1 ) where L is a suitable loss function , such as the 0-1 loss for classification , and C : X → 2X outputs a corruption set for a given example x . For example , in the case of an image x , a plausible corruption set C ( x ) could contain blurred , pixelized and noisy variants of x . In other words , we seek the optimal parameters θ∗ which minimize the corrupted adversarial risk so that fθ∗ is invariant to corruptions ; that is , fθ∗ ( x′ ) = fθ∗ ( x ) for all x′ ∈ C ( x ) . For example if x is an image classified to be a horse by fθ∗ , then this prediction should not be affected by the image being slightly corrupted by camera blur , Poisson noise or JPEG compression artifacts . AdversarialAugment ( AdA ) . Our method , AdA , uses image-to-image models to generate adversarially corrupted images . At a high level , this is similar to how DeepAugment works : DeepAugment perturbs the parameters of two specific image-to-image models using heuristic operators , which are manually defined for each model . Our method , instead , is more general and optimizes directly over perturbations to the parameters of any pre-trained image-to-image model . We denote these image-to-image models as corruption networks . We experiment with four corruption networks : a vector-quantised variational autoencoder ( VQ-VAE ) ( van den Oord et al. , 2017 ) ; a convolutional U-Net ( Ronneberger et al. , 2015 ) trained for image completion ( U-Net ) , a super-resolution model ( EDSR ) ( Lim et al. , 2017 ) and a compressive autoencoder ( CAE ) ( Theis et al. , 2017 ) . The latter two models are used in DeepAugment as well . Additional details about the corruption networks are provided in the Appendix in section C. Formally , let cφ : X → X be a corruption network with parameters φ = { φi } Ki=1 which , when its parameters are perturbed , acts upon clean examples by corrupting them . Here each φi corresponds to the vector of parameters in the i-th layer , and K is the number of layers . Let δ = { δi } Ki=1 be a weight perturbation set , so that a corrupted variant of x can be generated by c { φi+δi } Ki=1 ( x ) . With a slight abuse of notation , we shorten c { φi+δi } Ki=1 to cφ+δ . Clearly , using unconstrained perturbations can result in exceedingly corrupted images which have lost all discriminative information and are not useful for training . For example , if cφ is a multi-layer perceptron , trivially setting δi = −φi would yield fully zero , uninformative outputs . Hence , we restrict the corruption sets by defining a maximum relative perturbation radius ν > 0 , and define the corruption set of AdA as C ( x ) = { cφ+δ ( x ) | ‖δ‖2 , φ ≤ ν } , where the norm ‖ · ‖2 , φ is defined as ‖δ‖2 , φ = maxi∈ { 1 , ... , K } ‖δi‖2/‖φi‖2 . Finding adversarial corruptions . For a clean image x with label y , a corrupted adversarial example within a bounded corruption distance ν is a corrupted image x′ = cφ+δ ( x ) generated by the corruption network c with bounded parameter offsets ‖δ‖2 , φ ≤ ν which causes fθ to misclassify x : fθ ( x′ ) 6= y . Similarly to Madry et al . ( 2018 ) , we find an adversarial corruption by maximizing a surrogate loss L̃ to L , for example , the cross-entropy loss between the predicted logits of the corrupted image and its clean label . We optimize over the perturbation δ to c ’ s parameters φ : max ‖δ‖2 , φ≤ν L̃ ( fθ ( cφ+δ ( x ) ) , y ) . ( 2 ) In practice , we solve this optimization problem ( approximately ) using projected gradient ascent to enforce that perturbations δ lie within the feasible set ‖δ‖2 , φ ≤ ν . Examples of corrupted images obtained by AdA are shown in Figure 1 . Adversarial training . Given the model f parameterized by θ , minimizing the corrupted adversarial risk from ( 1 ) results in parameters θ∗ obtained by solving the following optimization problem : θ∗= arg min θ E ( x , y ) ∼D [ max ‖δ‖2 , φ≤ν L̃ ( fθ ( cφ+δ ( x ) ) , y ) ] . ( 3 ) We also provide a full algorithm listing of our method in Appendix B . Meaningful corruptions . A crucial element of AdA is setting the perturbation radius ν to ensure that corruptions are varied enough to constitute a strong defense against common corruptions , while still being meaningful ( i.e. , without destroying semantics ) . We measure the extent of corruption induced by a given ν through the structural similarity index measure ( SSIM ) ( Wang et al. , 2004 ) between clean and corrupted images ( details on how SSIM is computed can be found in section E ) .We plot the distributions of SSIM over various perturbation radii in Figure 2 for corrupted images produced by AdA using two backbones ( EDSR and CAE ) on CIFAR-10 . We find that a relative perturbation radius of ν = .015 yields enough variety in the corruptions for both EDSR and CAE . This is demonstrated for EDSR by having a large SSIM variance compared to , e.g . ν = 0.009375 , without destroying semantic meaning ( retaining a high mean SSIM ) . We guard against unlikely but too severe corruptions ( i.e . with too low SSIM ) using an efficient approximate line-search procedure ( details can be found Appendix E ) . A similar approach for restricting the SSIM values of samples during adversarial training was used by Hameed ( 2020 ) ; Hameed & György ( 2021 ) .

Summary: The main contribution of the paper is the proposed AdversarialAugment (AdA) method. AdA generates augmented versions of an input by passing the input through an corruption network (such as a pretrained image-to-image model) while adding a worst-case perturbation to the _weights_ of this pretrained network. The paper thus aims at using worst-case perturbations for increasing average case out-of-distribution generalization such as common corruption robustness. Tuning the weight perturbation radius of the corruption network is done based on controlling the SSIM of the augmented to the clean input.  The paper provides theoretical considerations that state assumptions under which AdA is well-behaved (converges) and how it is related to prior work such as DeepAugment.  The paper presents an extensive evaluation on common corruption benchmarks (CIFAR10-C and ImageNet-C), domain shift (ImageNet-R), resampled test sets (CIFAR-10.1 and ImageNet-v2), and worst-case robustness against $\ell_p$ perturbations.