Defending Against Image Corruptions Through Adversarial Augmentations

1 INTRODUCTION . By following a process known as Empirical Risk Minimization ( ERM ) ( Vapnik , 1998 ) , neural networks are trained to minimize the average error on a training set . ERM has enabled breakthroughs in a wide variety of fields and applications ( Goodfellow et al. , 2016 ; Krizhevsky et al. , 2012 ; Hinton et al. , 2012 ) , ranging from ranking content on the web ( Covington et al. , 2016 ) to autonomous driving ( Bojarski et al. , 2016 ) via medical diagnostics ( De Fauw et al. , 2018 ) . ERM is based on the principle that the data used during training is independently drawn from the same distribution as the one encountered during deployment . In practice , however , training and deployment data may differ and models can fail catastrophically . Such occurrence is commonplace as training data is often collected through a biased process that highlights confounding factors and spurious correlations ( Torralba et al. , 2011 ; Kuehlkamp et al. , 2017 ) , which can lead to undesirable consequences ( e.g. , http : //gendershades.org ) . As such , it has become increasingly important to ensure that deployed models are robust and generalize to various input corruptions . Unfortunately , even small corruptions can significantly affect the performance of existing classifiers . For example , Recht et al . ( 2019 ) ; Hendrycks et al . ( 2019 ) show that the accuracy of IMAGENET models is severely impacted by changes in the data collection process , while imperceptible deviations to the input , called adversarial perturbations , can cause neural networks to make incorrect predictions with high confidence ( Carlini & Wagner , 2017a ; b ; Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ; Szegedy et al. , 2014 ) . Methods to counteract such effects , which mainly consist of using random or adversarially-chosen data augmentations , struggle . Training against corrupted data only forces the memorization of such corruptions and , as a result , these models fail to generalize to new corruptions ( Vasiljevic et al. , 2016 ; Geirhos et al. , 2018 ) . Recent work from Hendrycks et al . ( 2020b ) ( also known as AugMix ) argues that basic pre-defined corruptions can be composed to improve the robustness of models to common corruptions . Another line of work , DeepAugment ( Hendrycks et al. , 2020a ) , corrupts images by passing them through two specific image-to-image models while distorting the models ’ parameters and activations using an extensive range of manually defined heuristic operations . While both methods perform well on average on the common corruptions present in CIFAR-10-C and IMAGENET-C , they generalize poorly to the adversarial setting . Most recently , Laidlaw et al . ( 2021 ) proposed an adversarial training method based on bounding a neural perceptual distance ( i.e. , an approximation of the true perceptual distance ) , under the acronym of PAT for Perceptual Adversarial Training . Their method performs well against five diverse adversarial attacks , but , as it specifically addresses robustness to pixel-level attacks that directly manipulate image pixels , it performs worse than AugMix on common corruptions . In this work , we address this gap . We focus on training models that are robust to adversarially-chosen corruptions that preserve semantic content . We go beyond conventional random data augmentation schemes ( exemplified by Hendrycks et al. , 2020b ; a ) and adversarial training ( exemplified by Madry et al. , 2018 ; Gowal et al. , 2019 ; Laidlaw et al. , 2021 ) by leveraging image-to-image models that can produce a wide range of semantically-preserving corruptions ; in contrast to related works , our method does not require the manual creation of heuristic transformations . Our contributions are as follows : • We formulate an adversarial training procedure , named AdversarialAugment ( or AdA for short ) which finds adversarial examples by optimizing over the weights of any pre-trained image-to-image model ( i.e . over the weights of arbitrary autoencoders ) . • We give sufficient conditions for the consistency of idealized versions of our method and DeepAugment , and provide PAC-Bayesian performance guarantees , following Neyshabur et al . ( 2017 ) . Our theoretical considerations highlight the potential advantages of AdA over previous work ( DeepAugment ) , as well as the combination of the two . We also establish links to Invariant Risk Minimization ( IRM ) ( Arjovsky et al. , 2020 ) , Adversarial Mixing ( AdvMix ) ( Gowal et al. , 2019 ) and Perceptual Adversarial Training ( Laidlaw et al. , 2021 ) . • We improve upon the known state-of-the-art on CIFAR-10-C by achieving a mean corruption error ( mCE ) of 7.83 % when using our method in conjunction with others ( vs. 23.51 % for Perceptual Adversarial Training ( PAT ) , 10.90 % for AugMix and 8.11 % for DeepAugment ) . On IMAGENET we show that our method can leverage 4 pre-trained image-to-image models simultaneously ( VQVAE ( van den Oord et al. , 2017 ) , U-Net ( Ronneberger et al. , 2015 ) , EDSR ( Lim et al. , 2017 ) & CAE ( Theis et al. , 2017 ) ) to yield the largest increase in robustness to common image corruptions , among all evaluated models . • On ` 2 and ` ∞ norm-bounded perturbations we significantly improve upon previous work ( DeepAugment & AugMix ) using AdA ( EDSR ) , while slightly improving generalization performance on both IMAGENET-V2 and on CIFAR-10.1 . 2 RELATED WORK . Data augmentation . Data augmentation has been shown to reduce the generalization error of standard ( non-robust ) training . For image classification tasks , random flips , rotations and crops are commonly used ( He et al. , 2016a ) . More sophisticated techniques such as Cutout of DeVries & Taylor ( 2017 ) ( which produces random occlusions ) , CutMix of Yun et al . ( 2019 ) ( which replaces parts of an image with another ) and mixup of Zhang et al . ( 2018a ) ; Tokozume et al . ( 2018 ) ( which linearly interpolates between two images ) all demonstrate extremely compelling results . Guo et al . ( 2019 ) improved upon mixup by proposing an adaptive mixing policy . Works , such as AutoAugment ( Cubuk et al. , 2019 ) and the related RandAugment ( Cubuk et al. , 2020 ) , learn augmentation policies from data directly . These methods are tuned to improve standard classification accuracy and have been shown to work well on CIFAR-10 , CIFAR-100 , SVHN and IMAGENET . However , these approaches do not necessarily generalize well to larger data shifts and perform poorly on benign corruptions such as blur or speckle noise ( Taori et al. , 2020 ) . Robustness to synthetic and natural data shift . Several works argue that training against corrupted data only forces the memorization of such corruptions and , as a result , models fail to generalize to new corruptions ( Vasiljevic et al. , 2016 ; Geirhos et al. , 2018 ) . This has not prevented Geirhos et al . ( 2019 ) ; Yin et al . ( 2019 ) ; Hendrycks et al . ( 2020b ) ; Lopes et al . ( 2019 ) ; Hendrycks et al . ( 2020a ) from demonstrating that some forms of data augmentation can improve the robustness of models on IMAGENET-C , despite not being directly trained on these common corruptions . Most works on the topic focus on training models that perform well in expectation . Unfortunately , these models remain vulnerable to more drastic adversarial shifts ( Taori et al. , 2020 ) . Robustness to adversarial data shift . Adversarial data shift has been extensively studied ( Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ; Szegedy et al. , 2014 ; Moosavi-Dezfooli et al. , 2019 ; Papernot et al. , 2016 ; Madry et al. , 2018 ) . Most works focus on the robustness of classifiers to ` p-norm bounded perturbations . In particular , it is expected that a robust classifier should be invariant to small perturbations in the pixel space ( as defined by the ` p-norm ) . Goodfellow et al . ( 2015 ) and Madry et al . ( 2018 ) laid down foundational principles to train robust networks , and recent works ( Zhang et al. , 2019 ; Qin et al. , 2019 ; Rice et al. , 2020 ; Wu et al. , 2020 ; Gowal et al. , 2020 ) continue to find novel approaches to enhance adversarial robustness . However , approaches focused on ` p-norm bounded perturbations often sacrifice accuracy on non-adversarial images ( Raghunathan et al. , 2019 ) . Several works ( Baluja & Fischer , 2017 ; Song et al. , 2018 ; Xiao et al. , 2018 ; Qiu et al. , 2019 ; Wong & Kolter , 2021b ; Laidlaw et al. , 2021 ) go beyond these analytically defined perturbations and demonstrate that it is not only possible to maintain accuracy on non-adversarial images but also to reduce the effect of spurious correlations and reduce bias ( Gowal et al. , 2019 ) . Unfortunately , most aforementioned approaches perform poorly on CIFAR-10-C and IMAGENET-C . 3 DEFENSE AGAINST ADVERSARIAL CORRUPTIONS . In this section , we introduce AdA , our approach for training models robust to image corruptions through the use of adversarial augmentations while leveraging pre-trained autoencoders . In Appendix A we detail how our work relates to AugMix ( Hendrycks et al. , 2020b ) , DeepAugment ( Hendrycks et al. , 2020a ) , Invariant Risk Minimization ( Arjovsky et al. , 2020 ) , Adversarial Mixing ( Gowal et al. , 2019 ) and Perceptual Adversarial Training ( Laidlaw et al. , 2021 ) . Corrupted adversarial risk . We consider a model fθ : X → Y parametrized by θ . Given a dataset D ⊂ X × Y over pairs of examples x and corresponding labels y , we would like to find the parameters θ which minimize the corrupted adversarial risk : E ( x , y ) ∼D [ max x′∈C ( x ) L ( fθ ( x ′ ) , y ) ] , ( 1 ) where L is a suitable loss function , such as the 0-1 loss for classification , and C : X → 2X outputs a corruption set for a given example x . For example , in the case of an image x , a plausible corruption set C ( x ) could contain blurred , pixelized and noisy variants of x . In other words , we seek the optimal parameters θ∗ which minimize the corrupted adversarial risk so that fθ∗ is invariant to corruptions ; that is , fθ∗ ( x′ ) = fθ∗ ( x ) for all x′ ∈ C ( x ) . For example if x is an image classified to be a horse by fθ∗ , then this prediction should not be affected by the image being slightly corrupted by camera blur , Poisson noise or JPEG compression artifacts . AdversarialAugment ( AdA ) . Our method , AdA , uses image-to-image models to generate adversarially corrupted images . At a high level , this is similar to how DeepAugment works : DeepAugment perturbs the parameters of two specific image-to-image models using heuristic operators , which are manually defined for each model . Our method , instead , is more general and optimizes directly over perturbations to the parameters of any pre-trained image-to-image model . We denote these image-to-image models as corruption networks . We experiment with four corruption networks : a vector-quantised variational autoencoder ( VQ-VAE ) ( van den Oord et al. , 2017 ) ; a convolutional U-Net ( Ronneberger et al. , 2015 ) trained for image completion ( U-Net ) , a super-resolution model ( EDSR ) ( Lim et al. , 2017 ) and a compressive autoencoder ( CAE ) ( Theis et al. , 2017 ) . The latter two models are used in DeepAugment as well . Additional details about the corruption networks are provided in the Appendix in section C. Formally , let cφ : X → X be a corruption network with parameters φ = { φi } Ki=1 which , when its parameters are perturbed , acts upon clean examples by corrupting them . Here each φi corresponds to the vector of parameters in the i-th layer , and K is the number of layers . Let δ = { δi } Ki=1 be a weight perturbation set , so that a corrupted variant of x can be generated by c { φi+δi } Ki=1 ( x ) . With a slight abuse of notation , we shorten c { φi+δi } Ki=1 to cφ+δ . Clearly , using unconstrained perturbations can result in exceedingly corrupted images which have lost all discriminative information and are not useful for training . For example , if cφ is a multi-layer perceptron , trivially setting δi = −φi would yield fully zero , uninformative outputs . Hence , we restrict the corruption sets by defining a maximum relative perturbation radius ν > 0 , and define the corruption set of AdA as C ( x ) = { cφ+δ ( x ) | ‖δ‖2 , φ ≤ ν } , where the norm ‖ · ‖2 , φ is defined as ‖δ‖2 , φ = maxi∈ { 1 , ... , K } ‖δi‖2/‖φi‖2 . Finding adversarial corruptions . For a clean image x with label y , a corrupted adversarial example within a bounded corruption distance ν is a corrupted image x′ = cφ+δ ( x ) generated by the corruption network c with bounded parameter offsets ‖δ‖2 , φ ≤ ν which causes fθ to misclassify x : fθ ( x′ ) 6= y . Similarly to Madry et al . ( 2018 ) , we find an adversarial corruption by maximizing a surrogate loss L̃ to L , for example , the cross-entropy loss between the predicted logits of the corrupted image and its clean label . We optimize over the perturbation δ to c ’ s parameters φ : max ‖δ‖2 , φ≤ν L̃ ( fθ ( cφ+δ ( x ) ) , y ) . ( 2 ) In practice , we solve this optimization problem ( approximately ) using projected gradient ascent to enforce that perturbations δ lie within the feasible set ‖δ‖2 , φ ≤ ν . Examples of corrupted images obtained by AdA are shown in Figure 1 . Adversarial training . Given the model f parameterized by θ , minimizing the corrupted adversarial risk from ( 1 ) results in parameters θ∗ obtained by solving the following optimization problem : θ∗= arg min θ E ( x , y ) ∼D [ max ‖δ‖2 , φ≤ν L̃ ( fθ ( cφ+δ ( x ) ) , y ) ] . ( 3 ) We also provide a full algorithm listing of our method in Appendix B . Meaningful corruptions . A crucial element of AdA is setting the perturbation radius ν to ensure that corruptions are varied enough to constitute a strong defense against common corruptions , while still being meaningful ( i.e. , without destroying semantics ) . We measure the extent of corruption induced by a given ν through the structural similarity index measure ( SSIM ) ( Wang et al. , 2004 ) between clean and corrupted images ( details on how SSIM is computed can be found in section E ) .We plot the distributions of SSIM over various perturbation radii in Figure 2 for corrupted images produced by AdA using two backbones ( EDSR and CAE ) on CIFAR-10 . We find that a relative perturbation radius of ν = .015 yields enough variety in the corruptions for both EDSR and CAE . This is demonstrated for EDSR by having a large SSIM variance compared to , e.g . ν = 0.009375 , without destroying semantic meaning ( retaining a high mean SSIM ) . We guard against unlikely but too severe corruptions ( i.e . with too low SSIM ) using an efficient approximate line-search procedure ( details can be found Appendix E ) . A similar approach for restricting the SSIM values of samples during adversarial training was used by Hameed ( 2020 ) ; Hameed & György ( 2021 ) .

Summary: This paper provides a data augmentation method that augments samples by perturbing parameters of a generative model. The perturbations are found by an adversarial loss, and are constrained based on a perceptual similarity distance to guard from the outliers. In addition to thorough empirical evaluations, this paper provides formalization of their method and a closely related one (deep augment), adding theoretical insights , and provide interesting convergence properties and PAC-Bayesian analysis.