DJMix: Unsupervised Task-agnostic Augmentation for Improving Robustness

1 INTRODUCTION CNNs are the de facto standard components of image recognition tasks and achieve excellent performance . However , CNNs are vulnerable to unseen noise on input images . Such harmful noise includes not only adversarially generated noise ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ) , but also naturally possible noise such as blur by defocusing and artifacts generated by JPEG compression ( Vasiljevic et al. , 2016 ; Hendrycks & Dietterich , 2019 ) . Natural noise on input images is inevitable in the real world ; therefore , making CNNs robust to natural noise is crucial for practitioners . A simple approach to solving this problem is adding noise to training images , but this does not make models generalize to unseen corruptions and perturbations ( Vasiljevic et al. , 2016 ; Geirhos et al. , 2018 ; Gu et al. , 2019 ) . For example , even if Gaussian noise of a certain variance is added during training , models fail to generalize to Gaussian noise of other variances . Nonetheless , some data augmentation methods are effective for improving robustness . For example , Yin et al . reported that extensive augmentation , such as AutoAugment ( Cubuk et al. , 2019 ) , improves the robustness . Similarly , Hendrycks et al . proposed to mix differently augmented images during training to circumvent the vulnerability . We will further review previous approaches in Section 2 . Despite the effectiveness of these data augmentation and mixing approaches , these methods require handcrafted image transformations , such as rotation and solarizing . Particularly when geometrical transformations are used , the mixed images can not have trivial targets in non classification tasks , for instance , semantic segmentation and detection . This lack of applicability to other tasks motivates us to introduce robust data augmentation without such transformations . In this paper , we propose Discretizing and Joint Mixing ( DJMix ) which mixes original and discretized training images to improve the robustness . The difference between the original and obtained images is nearly imperceptible , as shown in Figure 1 , which enables the use of DJMix in various image recognition tasks . In Section 3 , we will introduce DJMix and analyze it empirically and theoretically . We show that DJMix reduces mutual information between inputs and internal representations to ignore harmful features and improve CNNs ’ resilience to test-time noise . To benchmark the robustness of CNNs to unseen noise , Hendrycks & Dietterich ( 2019 ) introduced ImageNet-C as a corrupted counterpart of the ImageNet validation set ( Russakovsky et al. , 2015 ) . CNN models are evaluated using this dataset on the noisy validation set , whereas they are trained without any prior information on the corruptions on the original training set . Similarly , Geirhos et al . created noisy ImageNet and compared different behaviors between humans and CNN models with image noise . In addition to these datasets designed for classification , we cre- ated Segmentation-C and Detection-C datasets , which are corrupted counterparts of the PASCAL-VOC validation sets ( Everingham et al. , 2015 ) . We demonstrate the robustness of CNN models trained with DJMix on various tasks using these benchmark datasets in Section 4 . Additionally , we perform experimental analyses , including ablation studies , to verify DJMix in Section 5 . In summary , our contributions are summarized as follows : 1 . We introduce DJMix , a simple task-agnostic data augmentation method for improving robustness . DJMix mixes the original and discretized training images and can be straightforwardly adapted to various image recognition tasks . We empirically demonstrate the effectiveness of this approach . 2 . We analyze DJMix theoretically from the Information Bottleneck perspective , which could help analyze other robust methods . We also investigate DJMix from the Fourier sensitivity perspective . 3 . We create datasets , Segmentation-C and Detection-C , to benchmark the robustness of CNN models on semantic segmentation and detection tasks . 2 RELATED WORK Small corruptions or perturbations on images can drastically change the predictions of CNN models . While adversarially generated noise , i.e. , adversarial examples ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ) , can be thought of as the worst case , natural noise also harms the performance of CNNs . Such natural noise includes blurs and artifacts generated by JPEG compression ( Vasiljevic et al. , 2016 ; Hendrycks & Dietterich , 2019 ) . Because of this vulnerability , CNN models sometimes predict inconsistently among adjacent video frames ( Gu et al. , 2019 ; Hendrycks & Dietterich , 2019 ) . For the real-world application of CNNs , this vulnerability needs to be overcome . A strong defense against adversarial examples is adversarial training , where CNN models are trained with adversarial examples ( Goodfellow et al. , 2014 ; Madry et al. , 2018 ) . Unfortunately , this approach fails in natural noise , because CNNs trained on a specific type of noise do not generalize to other types of noise ( Geirhos et al. , 2018 ; Vasiljevic et al. , 2016 ) . Instead , we need robust methods that are agnostic to test-time noise a priori ( Hendrycks & Dietterich , 2019 ) . Data augmentation is a practical approach to improving the generalization ability on clean data , e.g. , by randomly flipping and cropping ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , mixing different images ( Zhang et al. , 2018 ; Tokozume et al. , 2018 ) , or erasing random regions ( DeVries & Taylor ; Zhong et al. , 2020 ) . Some types of data augmentation are also reported to improve robustness . For example , strong data augmentation , namely , AutoAugment ( Cubuk et al. , 2019 ) , can improve the robustness ( Yin et al. , 2019 ) . Similarly , AugMix is a data augmentation method to alleviate the problem by mixing images that are differently augmented from each input image . CNNs exploit the texture or higher-frequency domains of images ( Jo & Bengio , 2017 ; Ilyas et al. , 2019 ; Wang et al. , 2020 ) , and thus , CNNs trained on detextured ImageNet images by style transfer show robustness to noise on input images ( Geirhos et al. , 2019 ) . Orthogonal to manipulating input images , enhancing CNN architectures or components is also a possible direction . Larger CNN models with feature aggregation used in DenseNet ( Huang et al. , 2017 ) and ResNeXt ( Xie et al. , 2017 ) show better robustness to natural noise ( Gu et al. , 2019 ; Hendrycks & Dietterich , 2019 ) . MaxBlur has been proposed to improve the shift invariance of subsampling operations used in pooling operations , e.g. , MaxPooling , and enhance the robust performance ( Zhang , 2019 ) . Our approach , DJMix , belongs to the first ones , which use data augmentation to enhance robustness . Unlike previous methods , DJMix applies imperceptible and task-agnostic augmentation to images . This property allows us to use DJMix for various image recognition tasks . 3 DJMIX FOR ROBUST DATA AUGMENTATION A CNN model fθ : RD → RD ′ is usually trained to minimize the task loss ℓ ( fθ ( x ) , y ) , where x ∈ RD is an input image , and y ∈ RD′ is its target . When the task is a D′-category classification task , y is a one-hot vector and ℓ is cross-entropy . DJMix uses a pair of loss functions , the task loss ℓ ( fθ ( x̂ ) , y ) and the consistency loss d ( fθ ( x̂ ) , fθ ( x ) ) . Then , CNN models are trained to minimize ℓ ( fθ ( x̂ ) , y ) + γd ( fθ ( x̂ ) , fθ ( x ) ) , ( 1 ) where γ is a positive coefficient . x̂ ∈ RD is a discretized image of an input image x , which we will describe in Section 3.1 , and d is a divergence , such as the Jensen–Shannon divergence ( Section 3.2 ) . We will discuss why DJMix improves robustness in Sections 3.3 and 3.4 , both theoretically and empirically . 3.1 DISCRETIZATION OF IMAGES DJMix discretizes each input image x into g ( x ) , where g : RD → RD is a discretizing autoencoder ( DAE ) , whose bottleneck is discretized . Specifically , we used the Vector-Quantized Variational AutoEncoder used by van den Oord et al . ( 2017 ) and Razavi et al . ( 2019 ) . This DAE g has a bottleneck of dimension C and discretizes the features by vector quantization with the codebook size of 2K . DAE is pretrained on training data to minimize Ex‖g ( x ) − x‖22 in an unsupervised manner . As we will show in Section 5 , mixing each input image and its discretized one improves the robustness . More precisely , instead of using x̂ = g ( x ) , we use x̂ = βx+ ( 1− β ) g ( x ) , ( 2 ) where β ∈ [ 0 , 1 ] is sampled from a random distribution . Following Zhang et al . ( 2018 ) , we adopt Beta distribution . Although this mixing strategy is similar to that of AugMix , some differences exist . A major difference is the discrepancy between x and x̂ . Because AugMix applies geometric and color-enhancing operations to obtain x̂ , its appearance is different from x , whereas DJMix yields a nearly identical x̂ from x . A minor difference is the task loss : DJMix uses ℓ ( fθ ( x̂ ) , y ) , whereas AugMix uses ℓ ( fθ ( x ) , y ) . We will analyze this difference in Section 5 . 3.2 CONSISTENCY LOSS The consistency loss d ( fθ ( x̂ ) , fθ ( x ) ) forces a CNN model to map x and x̂ closely and make these representations indistinguishable . Following Hendrycks et al . ( 2020 ) , we use the Jensen–Shannon ( JS ) divergence as the divergence d. We compare other divergences and distances with the JS divergence in Section 5 . DJMix appears similar to AugMix ( Hendrycks et al. , 2020 ) , as both use the mixing of images and the consistency loss . However , the details of the mixing processes are different : whereas AugMix yields a different looking x̂ from x , DJMix augments a similar x̂ to x. Owing to this property , DJMix can be used in various image recognition tasks , as we will show in Section 4 . Additionally , the task loss of DJMix uses mixed images as ℓ ( fθ ( x̂ ) , y ) , whereas that of AugMix uses the original image as ℓ ( fθ ( x ) , y ) . Empirically , we found that ℓ ( fθ ( x̂ ) , y ) improves the robustness compared with ℓ ( fθ ( x ) , y ) , which we will show in Section 5.1 . 3.3 FROM INFORMATION BOTTLENECK PERSPECTIVE The Information Bottleneck objective ( Tishby et al. , 2000 ) can be written with an intermediate feature z of a model fθ as maxθ I ( z , y ; θ ) s.t . I ( x , z ; θ ) ≤ I , where I ( w , v ) : = DKL ( p ( w , v ) ||p ( w ) p ( v ) ) is mutual information between w and v , and I is a positive constraint . Supervised training is expected to maximize I ( z , y ; θ ) . However , without the constraint , z highly likely contains unnecessary details of the input ; then the models learn vulnerable representation ( Alemi et al. , 2017 ; Fisher & Alemi , 2020 ) . Importantly , DJMix introduces this constraint to improve the robustness by ignoring task-irrelevant details . For the following theorem , we assume that β in Equation ( 2 ) is 0 , and fθ and g have enough capacities to achieve training losses being 0 . Theorem 1 . Let z be fθ ( x ) . After convergence of the model fθ trained with DJMix , mutual information is constrained by the logarithm of the codebook size , i.e. , I ( x , z ; θ ) ≤ K. ( 3 ) Proof . After convergence , d ( fθ ( x̂ ) , fθ ( x ) ) becomes 0 , or equivalently , fθ ( x̂ ) = fθ ( x ) from the assumption . x is quantized into a codeword x̂ ∈ { 1 , 2 , . . . , 2K } in the DAE g. Therefore , we obtain K = H ( Uniform { 1 , 2 , . . . , 2K } ) ( H : entropy ) = H ( x̂ ) ≥ H ( x̂ ) −H ( x̂ | fθ ( x̂ ) ) = I ( fθ ( x̂ ) , x̂ ) ≥ I ( fθ ( x ) , x ) ( from Data Processing Inequality ) = I ( x , z ) 3.4 FROM FOURIER SENSITIVITY PERSPECTIVE Figure 2 presents the sensitivity of CNN models trained with and without DJMix to additive noise of Fourier-basis vectors ( Yin et al. , 2019 ) . Here , we used WideResNet trained on CIFAR10 . As can be seen , DJMix improves robustness to a wide range of frequencies : from lower frequencies , depicted in the center area , to higher frequencies , which appearing in the edges . These results imply why CNN models trained with DJMix show robustness to input images with noise . The experiments discussed in Section 4 further demonstrate more empirical robustness of DJMix . 4 EXPERIMENTS AND RESULTS In this section , we present experimental results . We first introduce experimental settings and new datasets , Segmentation-C , and Detection-C , whose input images are artificially corrupted to measure the robustness . Then , we present empirical results and comparisons with other methods in Section 4.1 for classification , Section 4.2 for semantic segmentation , and Section 4.3 for detection . We conducted the experiments three times with different random seeds for each setting and reported the averaged values , except for ImageNet experiments . We describe the additional details of the experiments in Appendix B . IMPLEMENTATION We implemented DJMix as well as CNN models using PyTorch ( Paszke et al. , 2019 ) and used FAISS ( Johnson et al. , 2017 ) to make the nearest neighbor search in DAE faster . We used classification models used by Hendrycks et al . ( 2020 ) 1 . For segmentation and detection tasks , we used DeepLab-v3 and Faster-RCNN from torchvision2 , whose backbone networks are ResNet-50 ( He et al. , 2016 ) pretrained on ImageNet ( Russakovsky et al. , 2015 ) . DAE is pretrained on each dataset for the classification task and on ImageNet for other tasks . We set a dictionary size to 512 , i.e. , K = 9 , following Razavi et al . ( 2019 ) . We set the parameters of Beta distribution ( β0 , β1 ) for mixing in Equation ( 2 ) to ( 1.0 , 0.5 ) , and the coefficient for the consistency loss γ to 1.0 . DJMix is a task-agnostic method and can improve robustness by itself . Additionally , DJMix can be incorporated with task-specific data augmentation . We introduce a DJMix variant that applies random data augmentation ( DJMix+RA ) , consisting of AugMix ’ s augmentation operations . We describe more details of RA in Appendix B.5 . DATASETS For classification , we used three datasets , CIFAR10 , CIFAR100 ( Krizhevsky , 2009 ) , and ImageNet ( Russakovsky et al. , 2015 ) , consisting of 10 , 100 , and 1,000 categories , respectively . We trained CNN models on clean training sets and evaluated the models using the accompanying clean test sets . We also evaluated the models on corrupted test sets ( CIFAR10-C , CIFAR100-C , and ImageNet-C ) proposed by Hendrycks & Dietterich ( 2019 ) , which are created to measure the behavior of CNN models with 15 common corruptions . To benchmark the robustness in segmentation and detection tasks , we created Segmentation-C and Detection-C datasets from PASCAL VOC-2012 ( Everingham et al. , 2015 ) . Namely , we degenerated images of the test set of PASCAL VOC-2012 for segmentation and the validation set of PASCAL VOC-2012 for detection using 10 degeneration operations used in ImageNet-C : gaussian_noise , shot_noise , impulse_noise , snow , frost , fog , brightness , contrast , pixelate , and jpeg_compression . We omitted five blur operations , namely defocus_blur , glass_blur , motion_blur , zoom_blur , and gaussian_blur , because the expected outputs for segmentation and detection under corruption are not trivial . Examples of Segmentation-C are presented in Figure 3 . Following the convention , we trained models on the augmented dataset of PASCAL-VOC ( Hariharan et al. , 2011 ) for segmentation and the union of train and validation datasets of VOC-2007 and VOC-2012 for detection . Similar to 1https : //github.com/google-research/augmix 2https : //github.com/pytorch/vision/tree/v0.7.0 . Detection-C , Michaelis et al . ( 2019 ) also created corrupted test images of detection datasets for autonomous driving . METRICS For the classification task , we use the error rates as the metric on the original test sets . On the corrupted data , we measure the corrupted error rate Ec , s , where c is the corruption type , e.g. , Gaussian noise , and s is the severity level , and report the statistics . Precisely , we use the averaged scores Ec , sEc , s for CIFAR10-C and CIFAR100-C , and Corruption Error EsEc , s/EsEAlexNetc , s for ImageNet-C , following Hendrycks & Dietterich ( 2019 ) , where EAlexNetc , s is the error rate of AlexNet ( Krizhevsky et al. , 2012 ) . For the segmentation task , we report the mean intersection over union ( mIoU ) on the clean data . On Segmentation-C , we use corrupted mIoU Ic , s and report averaged mIoU EsIc , s . Similarly , for the detection task , we report mean average precision ( mAP ) on the clean data and averaged corrupted mAP EsAc , s of corrupted mAP Ac , s on Detection-C. 4.1 CLASSIFICATION We trained models on CIFAR10 , CIFAR100 , and ImageNet . For CIFAR10 and CIFAR100 , we used DenseNet-BC ( k = 12 , d = 100 ) ( Huang et al. , 2017 ) , WideResNet-28-2 ( Zagoruyko & Komodakis , 2016 ) , and ResNeXt-29 ( Xie et al. , 2017 ) . For ImageNet , we used ResNet-50 ( He et al. , 2016 ) . The results on CIFAR10 and CIFAR100 are presented in Table 1 with comparison with the baseline , mixup ( Zhang et al. , 2018 ) , AugMix , and AugMix without geometric transformations ( GT ) . Table 1 shows that AugMix without GTs degenerates performance both on clean and corrupted data from AugMix , indicating that the robustness of AugMix heavily depends on GTs . DJMix shows balanced performance between clean and corrupted data , without such GTs . DJMixwith RA further decreases error rates on corrupted datasets , as well as on clean datasets . We present the results on ImageNet in Table 2 . Again , DJMix decreases Corruption Errors , particularly when strong data augmentation is introduced ( DJMix+RA ) . 4.2 SEMANTIC SEGMENTATION We trained DeepLab-v3 ( Chen et al. , 2017 ) on PASCAL-VOC . The logits before upsampling are used for the consistency loss . Table 3 shows the mIoU comparison of baseline , AugMix without GTs ( AugMix∗ ) , DJMix , and DJMixwith random augmentation without GTs ( DJMix+RA∗ ) . We omitted GTs , because defining the targets for mixed images that differently applied GTs are not trivial for segmentation and detection tasks . AugMix w/o GT uses pairs of original and augmented images , i.e. , the width is set to 2 . As can be seen , DJMix improves the robustness , especially when combined with extra data augmentation . In some cases , such as Gaussian noise , shot noise , and impulse noise , DJMix+RA markedly enhances the performance from DJMix and AugMix w/o GT , which implies the importance of the combination of task-specific and task-agnostic augmentation in practice . 4.3 DETECTION We trained Faster-RCNN ( Ren et al. , 2015 ) on PASCAL-VOC . The consistency loss between the output logits of the backbone network is used for training . Table 4 shows that DJMix yields better performance on almost all corruption types . As semantic segmentation , we compare baseline , AugMix∗ , DJMix , and DJMix+RA∗ . Similarly to semantic segmentation , DJMix markedly improves the robustness in the detection task . 5 ANALYSIS 5.1 ABLATION STUDIES DESIGN OF TASK LOSS : The task loss of DJMix presented in Equation ( 1 ) is ℓ ( fθ ( x̂ ) , y ) , but ℓ ( fθ ( x ) , y ) , as AugMix , is also a possible choice . We compare these choices in Table 5 ( a , b ) . ℓ ( fθ ( x̂ ) , y ) improves the robustness compared with ℓ ( fθ ( x ) , y ) . CHOICE OF CONSISTENCY LOSS : DJMix uses JS divergence as the consistency loss , but other divergences can also be used as the loss function . Here , we compare the performance when JS divergence is replaced with KL divergence and L2 distance . As can be seen from Table 5 ( a , c ) , JS and KL show similar performance , whereas L2 shows performance degeneration on corrupted data . EFFECT OF DISCRETIZATION : We verify the effect of the discretization of DJMix using DAE by substituting the standard autoencoder for DAE . Namely , we removed vector quantization modules of DAE and pretrained the AE on the training data to minimize the reconstruction error as DAE . Table 5 ( a , d ) shows that discretization improves CNNs ’ robustness as expected from the Information Bottleneck perspective presented in Section 3.3 . EFFECT OF MIXING : Table 5 ( e ) shows test error rates of DJMix without mixing , i.e. , β = 0 in Equation ( 2 ) where only discretized images are used . The results show that mixing is indispensable to retain the performance on clean data . We present further experiments on betas in Appendix B . 5.2 COMPUTATIONAL OVERHEAD OF DJMIX We find that the computational overhead by the DAE is negligible . However , the number of forward passes affects the training time . For instance , the standard training on CIFAR10 using WideResNet for 200 epoch requires approximately 1 hour in our environment . DJMix with two forward passes per update takes about 2 hours , and AugMix with three forward passes per update takes about 3 hours . Importantly , DJMix does not modify the components of CNNs as AugMix ; therefore , these methods do not affect the test-time speed , which is preferable for the real-world applications . 6 CONCLUSION In this paper , we have proposed DJMix , a novel task-agnostic approach to make CNN models robust to test-time corruption . To achieve task-agnostic robustness , we have used an autoencoder with a discretization bottleneck . Unlike previous approaches , the image modification of DJMix does not affect the appearance of images , which is useful for non classification tasks . Experiments have shown that DJMix improves the robustness of CNN models to input noise in semantic segmentation and detection , in addition to classification . We have found that combining task-specific and taskagnostic augmentation methods further improves performance on noisy images . We hope that data augmentation for robustness , including DJMix , bridges research and the real-world practice of deep learning . REFERENCES Alexander A. Alemi , Ian Fischer , Joshua V. Dillon , and Kevin Murphy . Deep variational information bottleneck . In ICLR , 2017 . Liang-Chieh Chen , George Papandreou , Florian Schroff , and Hartwig Adam . Rethinking Atrous Convolution for Semantic Image Segmentation . arXiv , 2017 . Ekin D. Cubuk , Barret Zoph , Dandelion Mane , Vijay Vasudevan , and Quoc V. Le . Autoaugment : Learning augmentation strategies from data . In CVPR , 2019 . Terrance DeVries and Graham W. Taylor . Improved Regularization of Convolutional Neural Networks with Cutout . arXiv . M Everingham , S M A Eslami , L VanGool , C K I Williams , J Winn , and A Zisserman . The Pascal Visual Object Classes Challenge : A Retrospective . In ICCV , 2015 . Ian Fisher and Alexander A. Alemi . Ceb improves model robustness . In ICLR , 2020 . Robert Geirhos , Heiko H. Schütt , Carlos R. Medina Temme , Matthias Bethge , Jonas Rauber , and Felix A. Wichmann . Generalisation in humans and deep neural networks . In NeurIPS , 2018 . Robert Geirhos , Claudio Michaelis , Felix A. Wichmann , Patricia Rubisch , Matthias Bethge , and Wieland Brendel . Imagenet-trained CNNs are biased towards texture ; increasing shape bias improves accuracy and robustness . In ICLR , 2019 . Ian J. Goodfellow , Jonathon Shlens , and Christian Szegedy . Explaining and Harnessing Adversarial Examples . arXiv , 2014 . Keren Gu , Brandon Yang , Jiquan Ngiam , Quoc Le , and Jonathon Shlens . Using Videos to Evaluate Image Model Robustness . arXiv , 2019 . B Hariharan , P Arbeláez , L Bourdev , S Maji , and J Malik . Semantic contours from inverse detectors . In ICCV , 2011 . Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep Residual Learning for Image Recognition . In CVPR , 2016 . Dan Hendrycks and Thomas Dietterich . Benchmarking neural network robustness to common corruptions and perturbations . In ICLR , 2019 . Dan Hendrycks , Norman Mu , Ekin D. Cubuk , Barret Zoph , Justin Gilmer , and Balaji Lakshminarayanan . AugMix : A Simple Data Processing Method to Improve Robustness and Uncertainty . In ICLR , 2020 . Gao Huang , Zhuang Liu , Kilian Q. Weinberger , and Laurens van der Maaten . Densely Connected Convolutional Networks . In CVPR , 2017 . Andrew Ilyas , Shibani Santurkar , Dimitris Tsipras , Logan Engstrom , Brandon Tran , and Aleksander Madry . Adversarial examples are not bugs , they are features . In NeurIPS , 2019 . Jason Jo and Yoshua Bengio . Measuring the tendency of cnns to learn surface statistical regularities . arXiv , 2017 . Jeff Johnson , Matthijs Douze , and Hervé Jégou . Billion-scale similarity search with GPUs . arXiv , 2017 . Alex Krizhevsky . Learning Multiple Layers of Features from Tiny Images . Technical report , 2009 . Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . ImageNet Classification with Deep Convolutional Neural Networks . In NIPS , 2012 . Ilya Loshchilov and Frank Hutter . SGDR : Stochastic Gradient Descent with Warm Restarts . In ICLR , 2016 . Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu . Towards Deep Learning Models Resistant to Adversarial Attacks . In ICLR , 2018 . Claudio Michaelis , Benjamin Mitzkus , Robert Geirhos , Evgenia Rusak , Oliver Bringmann , Alexander S. Ecker , Matthias Bethge , and Wieland Brendel . Benchmarking robustness in object detection : Autonomous driving when winter is coming . arXiv , 2019 . Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , Alban Desmaison , Andreas Kopf , Edward Yang , Zachary DeVito , Martin Raison , Alykhan Tejani , Sasank Chilamkurthy , Benoit Steiner , Lu Fang , Junjie Bai , and Soumith Chintala . PyTorch : An Imperative Style , High-Performance Deep Learning Library . In NeurIPS , 2019 . Ali Razavi , Aäron van den Oord , and Oriol Vinyals . Generating diverse high-fidelity images with VQ-VAE-2 . In NeurIPS , 2019 . Shaoqing Ren , Kaiming He , Ross Girshick , and Jian Sun . Faster R-CNN : Towards Real-Time Object Detection with Region Proposal Networks . In NIPS , 2015 . Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei . ImageNet Large Scale Visual Recognition Challenge . IJCV , 2015 . Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Goodfellow , and Rob Fergus . Intriguing properties of neural networks . In ICLR , 2014 . Naftali Tishby , Fernando C Pereira , and William Bialek . The information bottleneck method . In The 37th Annual Allerton Conference on Communication , Control , and Computing , 2000 . Yuji Tokozume , Yoshitaka Ushiku , and Tatsuya Harada . Between-class Learning for Image Classification . In ICLR , 2018 . Aaron van den Oord , Oriol Vinyals , and Koray Kavukcuoglu . Neural Discrete Representation Learning . In NIPS , 2017 . Igor Vasiljevic , Ayan Chakrabarti , and Gregory Shakhnarovich . Examining the Impact of Blur on Recognition by Convolutional Networks . arXiv , 2016 . URL http : //arxiv.org/abs/ 1611.05760 . Haohan Wang , Xindi Wu , Zeyi Huang , and Eric P. Xing . High-frequency component helps explain the generalization of convolutional neural networks . In CVPR , June 2020 . Saining Xie , Ross Girshick , and Piotr Doll . Aggregated Residual Transformations for Deep Neural Networks . In CVPR , 2017 . Dong Yin , Raphael Gontijo Lopes , Jon Shlens , Ekin Dogus Cubuk , and Justin Gilmer . A fourier perspective on model robustness in computer vision . In NeurIPS , 2019 . Sergey Zagoruyko and Nikos Komodakis . Wide Residual Networks . In BMVC , 2016 . Hongyi Zhang , Moustapha Cisse , Yann N. Dauphin , and David Lopez-Paz . mixup : Beyond Empirical Risk Minimization . In ICLR , 2018 . Richard Zhang . Making convolutional networks shift-invariant again . In ICML , 2019 . Zhun Zhong , Liang Zheng , Guoliang Kang , Shaozi Li , and Yi Yang . Random erasing data augmentation . In AAAI , 2020 . A ADDITIONAL ABLATION STUDIES A.1 THE EFFECT OF BETA DISTRIBUTION PARAMETERS Main experiments used ( β0 , β1 ) = ( 1.0 , 0.5 ) of Beta distribution for mixing x̂ = βx + ( 1 − β ) g ( x ) , where β ∼ Beta ( β0 , β1 ) . Figure 5 shows test error rates on different combinations of the parameters using WideResNet . Larger β0 and smaller β1 yield x̂ close to x , and vice versa , which is reflected in the results on CIFAR10 , i.e. , clean data . We used ( β0 , β1 ) = ( 1.0 , 0.5 ) , which balances performance on clean and corrupted data . B EXPERIMENTAL SETTINGS DETAILS This section describes additional experimental settings . B.1 DISCRETIZING AUTOENCODER We trained the autoencoder for 200 epochs to minimize the reconstruction error , and its codebook is updated by exponential moving average . The hyperparameters are identical to Razavi et al . ( 2019 ) . B.2 CLASSIFICATION We trained models on CIFAR10 , CIFAR100 , and ImageNet . For CIFAR10 and CIFAR100 , we used DenseNet-BC ( k = 12 , d = 100 ) ( Huang et al. , 2017 ) , WideResNet-28-2 ( Zagoruyko & Komodakis , 2016 ) and ResNeXt-29 ( Xie et al. , 2017 ) . We trained these networks for 200 epochs using stochastic gradient descent with a momentum of 0.9 and setting an initial learning rate to 0.1 that decays by cosine annealing with warm restart ( Loshchilov & Hutter , 2016 ) . We set a weight decay to 1× 10−4 and a batch size to 128 . We used data augmentation of random horizontal flipping , random cropping , and random erasing ( Zhong et al. , 2020 ) by default . For ImageNet , we used ResNet-50 ( He et al. , 2016 ) and trained it for 100 epochs using SGD with a momentum of 0.9 and a weight decay of 1 × 10−4 . We set the batch size to 1,024 and an initial learning rate to 0.4 that decays at 30 , 60 , and 90th epochs . We used random cropping and horizontal flipping as the base data augmentation . When training ResNet-50 on ImageNet , we used automatic mixed precision ( AMP ) implemented in PyTorch v1.6 to save the GPU memory consumption . We also used AMP for semantic segmentation and detection tasks . B.3 SEMANTIC SEGMENTATION We trained DeepLab-v3 ( Chen et al. , 2017 ) for 30 epochs with a batch size of 32 and a learning rate of 1.0×10−3 . We used SGD with a momentum of 0.9 and set its initial learning rate to 0.02 . The learning rate is multiplied by a factor of ( 1 − iterationtotal iteration ) 0.9 as Chen et al . ( 2017 ) . See https : // github.com/pytorch/vision/tree/master/references/segmentation for further details . B.4 DETECTION We trained Faster-RCNN ( Ren et al. , 2015 ) for 26 epochs with a batch size of 32 and a learning rate of 1.0× 10−3 . The learning rate is divided by 10 at 16th and 22nd epochs , while the first 1,000 iterations are the warmup period . See https : //github.com/pytorch/vision/tree/ master/references/detection for further details . B.5 RANDOM AUGMENTATION We used random augmentation ( RA ) as task-specific data augmentation , which is orthogonal to DJMix . Basically , we followed the data augmentation module of AugMix , and the only difference is the width . Whereas AugMix sets the width to 3 , DJMix uses the width of 1 , i.e. , only a single stream of operations is applied to each input image . Each image augmented by RA is used as an input x to DJMix .

Summary: This paper proposes an alternative data augmentation method for robust image classification. Training on augmented data has been shown to improve robustness. The proposed method mixes discretized images with real images and uses a consistency loss to enforce smoothness among those predictions. Experimental results include classification, detection and segmentation. Section 5 includes ablation studies on the choice of consistency loss and choice of discretization.