Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation

1 INTRODUCTION Modern deep neural networks can achieve impressive performance at classifying images in curated datasets ( Karpathy , 2011 ; Krizhevsky et al. , 2012 ; Tan & Le , 2019 ) . Yet , they lack robustness to various forms of distribution shift that typically occur in real-world settings . For example , neural networks are sensitive to small translations and changes in scale ( Azulay & Weiss , 2018 ) , blurring and additive noise ( Dodge & Karam , 2017 ) , small objects placed in images ( Rosenfeld et al. , 2018 ) , and even different images from a distribution similar to the training set ( Recht et al. , 2019 ; 2018 ) . For models to be useful in the real world , they need to be both accurate on a high-quality held-out set of images , which we refer to as “ clean accuracy , ” and robust on corrupted images , which we refer to as “ robustness. ” Most of the literature in machine learning has focused on architectural changes ( Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Zoph & Le , 2017 ; Szegedy et al. , 2017 ; Han et al. , 2017 ; Zoph et al. , 2017 ; Hu et al. , 2017 ; Liu et al. , 2018 ) to im- prove clean accuracy but interest has recently shifted toward robustness as well . Research in neural network robustness has tried to quantify the problem by establishing benchmarks that directly measure it ( Hendrycks & Dietterich , 2018 ; Gu et al. , 2019 ) and comparing the performance of humans and neural networks ( Geirhos et al. , 2018b ; Elsayed et al. , 2018 ) . Others have tried to understand robustness by highlighting systemic failure modes of current methods . For instance , networks exhibit excessive invariance to visual features ( Jacobsen et al. , 2018 ) , texture bias ( Geirhos et al. , 2018a ) , sensitivity to worst-case ( adversarial ) perturbations ( Goodfellow et al. , 2014 ) , and a propensity to rely on non-robust , but highly predictive features for classification ( Doersch et al. , 2015 ; Ilyas et al. , 2019 ) . Of particular relevance , Ford et al . ( 2019 ) has established connections between popular notions of adversarial robustness and some measures of distribution shift considered here . Another line of work has attempted to increase model robustness performance , either by projecting out superficial statistics ( Wang et al. , 2019 ) , via architectural improvements ( Cubuk et al. , 2017 ) , pretraining schemes ( Hendrycks et al. , 2019 ) , or with the use of data augmentations . Data augmentation increases the size and diversity of the training set , and provides a simple way to learn invariances that are challenging to encode architecturally ( Cubuk et al. , 2017 ) . Recent work in this area includes learning better transformations ( DeVries & Taylor , 2017 ; Zhang et al. , 2017 ; Zhong et al. , 2017 ) , inferring combinations of them ( Cubuk et al. , 2018 ) , unsupervised methods ( Xie et al. , 2019 ) , theory of data augmentation ( Dao et al. , 2018 ) , and applications for one-shot learning ( Asano et al. , 2019 ) . Despite these advances , individual data augmentation methods that improve robustness do so at the expense of reduced clean accuracy . Further , achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets : the current state of the art in robustness benchmarks involves creating a custom dataset with styled-transferred images before training ( Geirhos et al. , 2018a ) , and still incurs a significant drop in clean accuracy . The ubiquity of reported robustness/accuracy trade-offs in the literature have even led to the hypothesis that these trade-offs may be inevitable ( Tsipras et al. , 2018 ) . Because of this , many recent works focus on improving either one or the other ( Madry et al. , 2017 ; Geirhos et al. , 2018a ) . In this work we propose a simple data augmentation method that overcomes this trade-off , achieving improved robustness while maintaining clean accuracy . Our contributions are as follows : • We characterize a trade-off between robustness and accuracy in standard data augmentations Cutout and Gaussian ( Section 2.1 ) . • We describe a simple data augmentation method ( which we term Patch Gaussian ) that allows us to interpolate between the two augmentations above ( Section 3.1 ) . Despite its simplicity , Patch Gaussian achieves a new state of the art in the Common Corruptions robustness benchmark ( Hendrycks & Dietterich , 2018 ) , while maintaining clean accuracy , indicating current methods have not reached this fundamental trade-off ( Section 4.1 ) . • We demonstrate that Patch Gaussian can be combined with other regularization strategies ( Section 4.2 ) and data augmentation policies ( Section 4.3 ) . • We perform a frequency-based analysis ( Yin et al. , 2019 ) of models trained with Patch Gaussian and find that they can better leverage high-frequency information in lower layers , while not being too sensitive to them at later ones ( Section 5.1 ) . • We show a similar method can be used in adversarial training , suggesting under-explored questions about training distributions ’ effect on out-of-distribution robustness ( Section 5.2 ) . 2 PRELIMINARIES . We start by considering two data augmentations : Cutout ( DeVries & Taylor , 2017 ) and Gaussian ( Grandvalet & Canu , 1997 ) . The former sets a random patch of the input image to a constant ( mean pixel in the dataset ) in order to improve clean accuracy . The latter adds independent Gaussian noise to each pixel of the input image , which directly increases robustness to Gaussian noise . 2.1 CUTOUT AND GAUSSIAN EXHIBIT A TRADE-OFF BETWEEN ACCURACY AND ROBUSTNESS . We compare the effectiveness of Gaussian and Cutout data augmentation for accuracy and robustness by measuring the performance of models trained with each on clean as well as corrupted data . Here , robustness is defined as average accuracy of the model , when tested on data corrupted by various σ ( 0.1 , 0.2 , 0.3 , 0.5 , 0.8 , 1.0 ) of Gaussian noise , relative to the clean accuracy : Relative Gaussian Robustness = E σ ( Accuracy on Data Corrupted by σ ) − Clean Accuracy Fig . 2 highlights an apparent trade-off in using these methods . In accordance to previous work ( DeVries & Taylor , 2017 ) , Cutout improves accuracy on clean test data . Despite this , we find it does not lead to increased robustness . Conversely , training with higher σ of Gaussian can lead to increased robustness to Gaussian noise , but also leads to decreased accuracy on clean data . Therefore , any robustness gains are offset by poor overall performance : a model with a perfect Relative Robustness of 0 , but whose clean accuracy dropped to 50 % will be wrong half the time , even on clean data . At first glance , these results seem to reinforce the findings of previous work ( Tsipras et al. , 2018 ) , indicating that robustness comes at the cost of generalization , which would offset any benefits of improved robustness . In the following sections , we will explore whether there exists augmentation strategies that do not exhibit this limitation . 3 METHOD . Each of the two methods seen so far achieves one half of our stated goal : either improving robustness or slightly improving/maintaining clean test accuracy , but never both . To explore whether this observed trade-off is fundamental , we introduce Patch Gaussian , a technique that combines the noise robustness of Gaussian with the slightly improved clean accuracy of Cutout . Our method is intentionally simple but , as we ’ ll see , it ’ s powerful enough to overcome the limitations described and beats complex training schemes designed to provide robustness . 3.1 PATCH GAUSSIAN . Patch Gaussian works by adding a W × W patch of Gaussian noise to the image ( Figure 3 ) . As with Cutout , the center of the patch is sampled to be within the image . By varying the size of this patch and the maximum standard deviation of noise sampled σmax , we can interpolate between Gaussian ( which applies additive Gaussian noise to the whole image ) and an approximation of Cutout ( which removes all information inside the patch ) . See Fig . 9 for more examples . 3.2 HYPER-PARAMETER SELECTION . Our goal is to learn models that achieve both good clean accuracy and improved robustness to corruptions . Prior work has optimized for one or the other but , as noted before , to meaningfully improve robustness to other distributions , a method can ’ t incur a significant drop in clean accuracy . Therefore , when selecting hyper-parameters , we focus on identifying the models that are most robust while still achieving a minimum accuracy ( Z ) on the clean test data . Values of Z are selected to incur negligible decrease in clean accuracy . As such , they vary per dataset and model , and can be found in the Appendix ( Table 5 ) . If no model has clean accuracy ≥ Z , we report the model with highest clean accuracy , unless otherwise specified . We find that patch sizes around 25 on CIFAR ( ≤250 on ImageNet , i.e . : uniformly sampled with maximum value 250 ) with σ ≤ 1.0 generally perform the best . A complete list of selected hyperparameters for all augmentations can be found in Table 5 . We are interested in out-of-distribution robustness , and report performance of selected models on Common Corruption ( Hendrycks & Dietterich , 2018 ) . However , when selecting hyper-parameters , we use Relative Gaussian Robustness as a stand-in for “ robustness. ” Ford et al . ( 2019 ) indicates that this metric is correlated with performance on Common Corruptions , so selected models should be generally robust beyond Gaussian corruptions . By picking models based on robustness to Gaussian noise , we ensure that our selection process does not overfit to the Common Corruptions benchmark . 4 RESULTS . Models trained with Patch Gaussian overcome the observed trade-off and gain robustness to Gaussian noise while maintaining clean accuracy ( Fig . 1 ) . Because Gaussian robustness is only used for hyper-parameter selection , we omit these results , but refer the curious reader to Appendix Fig . 7 . Instead , we report how this Gaussian robustness translates into better Common Corruption robustness , which is in line with reports of the correlation between the two ( Ford et al. , 2019 ) . In doing so , we establish a new state of the art in the Common Corruptions benchmark ( Section 4.1 ) , despite the simplicity of our method when compared with the previous best ( Geirhos et al. , 2018a ) . We then show that Patch Gaussian can be used in complement to other common regularization strategies ( Section 4.2 ) and data augmentation policies ( Cubuk et al. , 2018 ) ( Section 4.3 ) . 4.1 TRAINING WITH PATCH GAUSSIAN IMPROVES COMMON CORRUPTION ROBUSTNESS . In this section , we look at how our augmentations impact robustness to corruptions beyond Gaussian noise . Rather than focusing on adversarial examples that are worst-case bounded perturbations , we focus on a more general set of corruptions ( Gilmer et al. , 2018 ) that models are likely to encounter in real-world settings : the Common Corruptions benchmark ( Hendrycks & Dietterich , 2018 ) . This benchmark , also referred to as CIFAR-C and ImageNet-C , is composed of images transformed with 15 corruptions , at 5 severities each . Each is designed to model transformations commonly found in real-world settings , such as brightness , different weather conditions , and different kinds of noise . Table 1 shows that Patch Gaussian achieves state of the art on both of these benchmarks in terms of mean Corruption Error ( mCE ) . A “ Corruption Error ” is a model ’ s average error over 5 severities of a given corruption , normalized by the same average of a baseline model . However , ImageNet-C was released in compressed JPEG format ( ECMA International , 2009 ) , which alters the corruptions applied to the raw pixels . Therefore , we report results on the benchmark as-released ( “ Original mCE ” ) 1 as well as a version of 12 corruptions without the extra compression ( “ mCE ” ) . Additionally , because Patch Gaussian is a noise-based augmentation , we wanted to verify whether its gains on this benchmark were solely due to improved performance on noise-based corruptions ( Gaussian Noise , Shot Noise , and Impulse Noise ) . To do this , we also measure the models ’ average performance on all other corruptions , reported as “ Original mCE ( -noise ) ” , and “ mCE ( -noise ) ” . The models used to normalize Corruption Errors are the “ Baselines ” trained with only flip and crop data augmentation . The one exception is Original mCE ImageNet , where we use the AlexNet baseline to be directly comparable with previous work ( Hendrycks & Dietterich , 2018 ; Geirhos et al. , 2018a ) . On CIFAR , we compare with an adversarially-trained model ( Madry et al. , 2017 ) . On ImageNet , we compare with a model trained with Random Erasing ( Zhong et al. , 2017 ) , as well as a shape-biased model “ SIN+IN ftIN ” ( Geirhos et al. , 2018a ) . Finally , previous work ( Yin et al. , 2019 ) has found that augmentation diversity is a key component of robustness gains . To confirm that Patch Gaussian ’ s gains aren ’ t simply a result of using multiple augmentations , we also report results on training with Cutout and Gaussian applied in sequence ( “ Cutout & Gaussian ” and “ Gaussian & Cutout ” ) , as well as to 50 % of batches ( “ Gaussian or Cutout ” ) . We observe that Patch Gaussian outperforms all models , even on corruptions like fog where Gaussian hurts performance ( Ford et al. , 2019 ) . Scores for each corruption can be found in the 1We note that , in our implementation , Original mCE used the 299× 299 images provided by the benchmark , which were originally intended for Inception models . Preliminary experiments indicate this does not change our qualitative analysis , but we will update the paper with evaluations on 224× 224 when possible . Appendix ( Tables 6 & 7 ) . We observe small gains on low capacity models ( ResNet-50 & Wide ResNet ) , but find large robustness improvements with higher capacity ones ( ResNet-200 & Shake 112 ) . These results are surprising : achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets . Training shape-biased models ( Geirhos et al. , 2018a ) involves creating a custom dataset of style-transferred images , which is a computationallyexpensive operation . Even with these , the most robust model reported SIN+IN displays a significant drop in clean accuracy . Because of this , our main comparison is with SIN+IN ftIN , which is fine-tuned on ImageNet . A comparison with SIN+IN can be found in Appendix Table 8 . In sum , despite its simplicity , Patch Gaussian achieves a substantial decrease in mCE relative to other models , indicating that current methods have not reached the theoretical trade-off ( Tsipras et al. , 2018 ) , and that complex training schemes ( Geirhos et al. , 2018a ) are not needed for robustness .

Summary: This paper proposes a hybrid approach for adding noise to training images of an image classification model. Instead of either cutting out a patch or adding gaussian noise, the authors propose to adding a patch of gaussian noise to the images. Although possibly useful practically, this proposal lacks theoretical base on how and why it would be better, besides the claim that hopefully the combination will combine the benefit and subtract the weakness. The experiments are rather limitted to support the claim.