DJMix: Unsupervised Task-agnostic Augmentation for Improving Robustness
Authors: Ryuichiro Hataya
Hideki Nakayama
Venue:  (2021)
Keywords: robustness
uncertainty
discretization
data augmentation

Abstract: Convolutional Neural Networks (CNNs) are vulnerable to unseen noise on input images at the test time, and thus improving the robustness is crucial. In this paper, we propose DJMix, a data augmentation method to improve the robustness by mixing each training image and its discretized one. Discretization is done in an unsupervised manner by an autoencoder, and the mixed images are nearly impossible to distinguish from the original images. Therefore, DJMix can easily be adapted to various image recognition tasks. We verify the effectiveness of our method using classification, semantic segmentation, and detection using clean and noisy test images.

Under review as a conference paper at ICLR 2021
DJMIX: UNSUPERVISED TASK-AGNOSTIC AUGMEN- TATION FOR IMPROVING ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional Neural Networks (CNNs) are vulnerable to unseen noise on input images the test time, and thus improving the robustness is crucial. In this pa- per, we propose DJMix, a data augmentation method to improve the robustness by mixing each training image and its discretized one. Discretization is done unsupervised manner by an autoencoder, and the mixed images are nearly im- possible to distinguish from the original images. Therefore, DJMix x can easily be adapted to various image recognition tasks. We verify the effectiveness of our method using classification, semantic segmentation, and detection using clean and noisy test images.
INTRODUCTION
CNNs are the de facto standard components of image recognition tasks and achieve excellent per- formance. However, CNNs are vulnerable unseen noise input images. Such harmful noise includes not only adversarially generated noise (Szegedy et al., 2014; Goodfellow et al., 2014), but also naturally possible noise such as blur by defocusing and artifacts generated by JPEG com- pression (Vasiljevic et al., 2016; Hendrycks & Dietterich, 2019). Natural noise on input images is inevitable in the real world; therefore, making CNNs robust to natural noise is crucial for practition- ers.
simple approach to solving this problem is adding noise to training images, but this does not make models generalize unseen corruptions and perturbations (Vasiljevic et al., 2016; Geirhos et al., 2018: Gu al., 2019). For example, even if Gaussian noise of a certain variance added during training, models fail to generalize to Gaussian noise of other variances. Nonetheless, some data augmentation methods are effective for improving robustness. For example, Yin et al. reported that extensive augmentation, such Aut oAugment (Cubuk al., 2019), improves the robust- ness. Similarly, Hendrycks et al. proposed to mix differently augmented images during training to circumvent the vulnerability. We will further review previous approaches in Section 2.
Despite the effectiveness of these data augmentation and mixing approaches, these methods require handcrafted image transformations, such rotation and solarizing. Particularly when geometrical transformations are used, the mixed images cannot have trivial targets in non classification tasks, for instance, semantic segmentation and detection. This lack of applicability to other tasks motivates us to introduce robust data augmentation without such transformations.
In this paper, we propose Discretizing and Joint Mixing (DJMix) which mixes original and dis- cretized training images to improve the robustness. The difference between the original and ob- tained images nearly imperceptible, as shown in Figure 1, which enables the use of DJMix in various image recognition tasks. In Section 3, we will introduce DJMi x and analyze it empirically and theoretically. We show that DJMi reduces mutual information between inputs and internal representations to ignore harmful features and improve CNNs' resilience to test-time noise.
To benchmark the robustness CNNs to unseen noise, Hendrycks & Dietterich (2019) introduced ImageNet-C
corrupted counterpart of the ImageNet validation set (Russakovsky et al., 2015). CNN models are evaluated using this dataset on the noisy validation set, whereas they are trained without any prior information on the coruptions Geirhos et al. created noisy ImageNet and compared different behaviors between humans and CNN models with image noise. In addition to these datasets designed for classification, cre-
the original training set. Similarly,

---

Under review as a conference paper at ICLR 2021
Reconstructed Image a
CNNfè
Classification "cat"
Image C
Task Loss
Segmentation
Label y
Consistency Loss
Discretizing AE !
Detection
Figure 1: Schematic view of DJMix. In DJMix, CNN models trained to minimize divergence between the features of each input image fa( and its discretized image (o()), as well as the task- specific loss between fa) and the label y. This pipeline barely affects the appearance of input images, and thus, can be used for various image recognition tasks, e.g., classification, (semantic) segmentation, and detection.
ated aegmentation- and Detection-T datasets, which are corrupted counterparts of the PASCAL-VOS validation sets (Everingham al., 2015).
We demonstrate the robustness of CNN models trained with DJMix on various tasks using these benchmark datasets in Section 4. Additionally, we perform experimental analyses, including ab- lation studies, to verify DJMix in Section 5. In summary, our contributions are summarized as follows:
We introduce DJMix, simple task-agnostic data augmentation method for improving robustness. DJMix mixes the original and discretized training images and can be straight- forwardly adapted to various image recognition tasks. We empirically demonstrate the effectiveness of this approach.
2. We analyze DJMix theoretically from the Information Bottleneck perspective, which could help analyze other robust methods. We also investigate DJMix from the Fourier sensitivity perspective.
3. We create datasets, degmentation-- and Detection-F benchmark the robust- ness of CNN models on semantic segmentation and detection tasks.
RELATED WORK
Small corruptions or perturbations on images can drastically change the predictions of CNN models. While adversarially generated noise, i.e., adversarial examples (Szegedy al., 2014; Goodfellow et al., 2014), can be thought of as the worst case, natural noise also harms the performance of CNNs. Such natural noise includes blurs and artifacts generated by JPEG compression (Vasiljevic et al., 2016; Hendrycks & Dietterich, 2019). Because of this vulnerability, CNN models sometimes predict inconsistently among adjacent video frames (Gu al., 2019; Hendrycks & Dietterich, 2019). For the real-world application of CNNs, this vulnerability needs to be overcome. A strong defense against adversarial examples is adversarial training. where CNN models are trained with adversarial examples (Goodfellow et al., 2014; Madry et al., 2018). Unfortunately, this approach fails in natural noise, because CNNs trained on specific type of noise do not generalize to other types of noise (Geirhos et al., 2018; Vasiljevic et al., 2016). Instead, we need robust methods that are agnostic to test-time noise a priori (Hendrycks & Dietterich, 2019).
Data augmentation is practical approach to improving the generalization ability clean data, e.g., by randomly flipping and cropping (Krizhevsky et al., 2012; He et al., 2016), mixing different images (Zhang et al.. 2018; Tokozume al., 2018), erasing random regions VDevries & Taylor; Zhong et al., 2020). Some types of data augmentation are also reported to improve robustness. For example, strong data augmentation, namely, Aut Augment (Cubuk et al., 2019), improve the
2

---

Under review as a conference paper at ICLR 2021
robustness (Yin et al., 2019). Similarly. AugMix is data augmentation method to alleviate the problem by mixing images that are differently augmented from each input image. CNNs exploit the texture higher-frequency domains of images (Jo & Bengio, 2017; Ilyas et al., 2019; Wang et al., 2020), and thus, CNNs trained on detextured mageNeet images by style transfer show robustness noise on input images (Geirhos et al., 2019).
Orthogonal to manipulating input images, enhancing CNN architectures or components also a possible direction. Larger CNN models with feature aggregation used DenseNet (Huang al., 2017) and ResNeXt (Xie et al.. 2017) show better robustness to natural noise (Gu et al.. 2019; Hendrycks & Dietterich. 2019). MaxBlur has been proposed improve the shift invariance of subsampling operations used in pooling operations, e.g., MaxPooling, and enhance the robust per- formance (Zhang, 2019).
Our approach, DJMix, belongs to the first ones, which use data augmentation to enhance robustness. Unlike previous methods, DJMix x applies imperceptible and agg-osoc augmentation images. This property allows us to use DJMix for various image recognition tasks.
DJMIX FOR ROBUST DATA AUGMENTATION
CNN model so : RDD RD' usually trained to minimize the task loss {(fo(a)) y), where a E R is an input image, and y E IR'' is its target. When the task is D' -cattgoyyy classification task, y is one-hot vector and l is cross-entropy.
DJMix uses a pair of loss functions, the task loss (()( y) and the consistency loss d(fo (.), fo(x)). Then, CNN models are trained to minimize
((fo()).)) + +odd+++d(o))))) (d([o(t)
(I)
where y is positive coefficient. & E R.P is a discretized image of an input image æ, which we will describe in Section 3.1, and is divergence, such the Shasnn-Shannon divergence (Section 3.2). We will discuss why DJMix improves robustness in Sections 3.3 and 3.4, both theoretically and empirically.
3.1 DISCRETIZATION OF IMAGES
DJMix discretizes each input image æ into g(æ), where g: RD - P is discretizing autoencoder (DAE), whose bottleneck is discretized. Specifically, we used the Vector-Quantized Variational AutoEncoder used by van den Oord et al. (2017) and Razavi et al. (2019). This DAE has a bottleneck of dimension C' and discretizes the features by vector quantization with the codebook size of 2 DAE is pretrained on training data to minimize E, (l((z))) x| in an unsupervised manner.
As we will show in Section 5, mixing each input image and its discretized one improves the robust- ness. More precisely, instead of using å g(x),, we
= 3x + (1 (1- ()g(x)
(2)
where 3 E [0, 1] is sampled from a random distribution. Following Zhang et al. (2018), we adopt Beta distribution. Although this mixing strategy is similar that of AugMix, some differences exist. A major difference is the discrepancy between æ and ă. Because AugMix applies geometric and color-enhancing operations to obtain ă, its appearance is different from æ, whereas DJMix yields a nearly identical 4 from A minor difference is the task loss: DJMix x uses ((f(()) v), whereas AugMix uses {(fo(x), y). We will analyze this difference Section 5.
3.2 CONSISTENCY LOSS
The consistency loss dz)) fo())) forces a CNN model to map æ and & closely and make these representations indistinguishable. Following Hendrycks al. (2020), we use the Jensen- lensen-Shannon
3

---

Under review as a conference paper at ICLR 2021
(IS) divergence the divergence d. We compare other divergences and distances with the JS diver- gence in Section 5.
DJMix appears similar to AugMix (Hendrycks et al., 2020), both the mixing of images and the consistency loss. However, the details of the mixing processes are different: whereas AugMix yields a different looking E from æ. DJMix augments similar & DJMiX x can be used in various image recognition tasks, as we will show in Section 4. Additionally, the task loss of DJMix uses mixed images f(fo (.). y). whereas that of AugMi x uses the original image as {[f((z) y). Empirically, we found that ((ff (.); y) improves the robustness compared with f( (z). y), which we will show Section 5.1.
Owing to this property,
3.3 FROM INFORMATION BOTTLENECK PERSPECTIVE
The Information Bottleneck objective (Tishby et al., 2000) can be written with an intermedi- ate feature x a model fo as max, I(z, y: 0) s.t. I(x, x; I I. where I(w, v) = (pl(p(w, ") ()p((())((v) is mutual information between w and and I is positive constraint. Supervised training is expected to maximize Σ(z, y; 0). However, without the constraint, z highly likely contains unnecessary details of the input; then the models learn vulnerable representation (Alemi et al.. 2017: Fisher & Alemi, 2020). Importantly, DJMix introduces this constraint to im- prove the robustness by ignoring task-irrelevant details. For the following theorem, we assume that 3 in Equation (2) is O, and fo and have enough capacities to achieve training losses being 0.
Theorem 1. Let z be fx(x). After convergence of the model fe trained with DJMix, mutual infor- mation is constrained by the logarithm of the codebook size, i.c.,
I(x, 2; 0) S K.
Proof. After convergence, d((f(()), (x)) becomes O. eguivalently, fo fe(x) from the assumption. æ is quantized into a codeword à E (1,2.... 2 in the DAE g. Therefore, we obtain
K = H(Uniform 21, (H: entropy)
H())
2 H(x)) H(d [o()))
I()) ())
2 I(fo(x),
(from Data Processing Inequality)
I(x,2)
3.4 FROM FOURIER SENSITIVITY PERSPECTIVE
Figure presents the sensitivity of CNN models trained with and without DJMix additive noise of Fourier-basis vectors (Yin et al., 2019). Here, we used WideReNeet trained on CIFAR10 O. As can be seen, DJMix x improves robustness depicted in the center area, to higher frequencies, which appearing in the edges. These results imply why CNN models trained with DJMix show robustness to input images with noise. The experiments discussed in Section 4 further demonstrate more empirical robustness of DJMix.
wide range of frequencies: from lower frequencies.
4 EXPERIMENTS AND RESULTS
In this section, we present experimental results. We first introduce experimental settings and new datasets, Segmentation C, and Detection-CC whose input images are artificially corrupted to measure the robustness. Then, present empirical results and comparisons with other methods in Section 4.1 for classification, Section 4.2 for semantic segmentation, and Section 4.3 for detection. We conducted the experiments three times with different random seeds for each setting and reported the averaged values, except for ImageNet experiments. We describe the additional details of the experiments in Appendix B.
4

---

Under review as a conference paper at ICLR 2021
CNN trained w/o DJMix
CNN trained w/ DJMix
Frequenyy Noise
0.25
Error Rates
0.20
High-Frequency Noise
0.15
0.05
Figure 2: DJMix x reduces the sensitivity of CNN models to additive noise. Fourier-basis vectors are added to 2,048 randomly sampled test samples of CIFAR1 whose image size is 32 x 32, and pixels present the averaged error rates. Each pixel corresponds to certain Fourier basis, e.g., the center area lower frequencies and the edges of higher frequencies. The leftmost images examples of images with additive noise enhanced for visualization.
whose L2 norm
IMPLEMENTATION
We implemented DJMix well as CNN models using PyTorch (Paszke et al., 2019) and used FAISS (Johnson et al., 2017) to make the nearest neighbor search in DAE faster. We used clas- sification models used by Hendrycks et al. (2020) !. For segmentation and detection tasks, we used DeepLab-v3 and Faster-RCn RCNN from torchvision', whose backbone networks are ResNet- 50 (He al., 2016) pretrained on ImageNet (Russakovsky et al., 2015).
DAE is pretrained on each dataset for the classification task and on ImageNet for other tasks. We set a dictionary size to 512, i.e., K = 9, following Razavi et al. (2019). We set the parameters of Beta distribution (3o, $1) for mixing in Equation (2) to (1.0, 0.5), and the coefficient for the consistency loss 7 to 1.0.
DJMix is task-agnostic method and can improve robustness by itself. Additionally, DJMix x can be incorporated with task-specific data augmentation. We introduce a DJMix variant that applies random data augmentation (DJMix+RA), consisting of AugMix's augmentation operations. We describe more details of RA in Appendix B.5.
DATASETS
For classification. we used three datasets, CIFARIC, CIFAR100 (Krizhevsky, 2009), and ImageNet (Russakovsky et al., 2015), consisting of 10, 100, and 1,000 categories, respectively. We trained CNN models on clean training sets and evaluated the models using the accompanying clean test sets. We also evaluated the models on corrupted test sets (CIFAR10-OG CIFAR100- and ImageNet-OG proposed by Hendrycks & Dietterich (2019), which are created to measure behavior of CNN models with 15 common corruptions.
To benchmark the robustness in segmentation and detection tasks, we created Segmentation and Detection-C datasets from PASCAL VOC-2012 (Everingham et al., 2015). Namely, degenerated images of the test set of PASCAL VOC-201 for segmentation and the validation set of PASCAL VOC-2012 for detection using 10 degeneration operations used in ImageNet-Cr pooot_noisee
impulse_noiste snow, frost, fog, brightness, contrast, pixelate, and jpeg_compressiion. We omitted five blur operations, namely de defocus_blure glass_blur, mot ion_bbuuu zoom_blur, and caussian_blus cause the expected outputs for segmentation and detection under corruption are not trivial. Ex- amples of Segmentation aion-C are presented in Figure 3, Following the convention. we trained models on the augmented dataset of PASCAL-VOC (Hariharan al., 2011) for segmentation and the union of train and validation datasets of VOC-2007 and VOC-201 12 for detection. Similar to
caussian_noisse
be-

---

Under review as a conference paper at ICLR 2021
original
paussian
Impuise
hrightness
jing compression
Figure 3: Examples of the Segmentat on- C dataset with severest noise of each corruption type. We created this dataset to benchmark the robustness of segmentation networks by degenerating validation images with 10 operations, such as Gaussian noise. Similarly, we created Detection-C detection counterpart of regmentationn
Detection-- C, Michaelis et al. (2019) also created corrupted test images of detection datasets for autonomous driving.
METRICS
For the classification task, we use the error rates as the metric the original test sets. On the corrupted data, we measure the corrupted error rate Ec,s, where is the corruption type, e,g. Gaussian noise, and is the severity level, and report the statistics. Precisely, aged scores Ec, for CIICRII------ O and CIFAR100-GG c, and Corruption Error E., / E c,s
the aver-
FAlexNet
for ImageNet-C, following Hendrycks & Dietterich (2019), where AllexNet is the error rate of AlexNet (Krizhevsky et al., 2012). For the segmentation task, we report the mean intersection over union (mloU) on the clean data. On Segmentat ion-C, averaged mloU E. le. Similarly, for the detection task, we report mean average precision (mAP) on the clean data and averaged corrupted mAP IE, Ac,s of corrupted mAP Ag,o Detection-C.
corrupted mloU le,- and report
4.1 CLASSIFICATION
We trained models CIFAR10, CIFAR100, and ImageNet. For CIFAR10 and CIFAR100, used DenseNet-BR (k 12, d 100) (Huang et al., 2017), WideResNet-28 (Zagoryyko & Komodakis, 2016), and desNeXt-29 (Xie et al.. 2017). For ImageNet, we used Re ResNet (He et al., 2016).
The results on CIFAR10 and CIFAR100 are presented in Table 1 with comparison with the base- line, xup (Zhang al., 2018), AugMi x, and AugMix x without geometric transformations (GT). Table 1 shows that AugMix without GTs degenerates performance both on clean and corrupted data from AugMix, indicating that robustness of AugMix heavily depends on GTs. DJMix x shows balanced performance between clean and corrupted data, without such GTs. DJMix with RA further decreases сттог rates on corrupted datasets, as well as on clean datasets. We present the results on Tma ImageNet in Table Again, DJMi decreases Corruption Errors, particularly when strong data augmentation is introduced (DJMi +
4.2 SEMANTIC SEGMENTATION
We trained DeepTab-v3 (Chen al., 2017) on PASCAL-VOC. The logits before upsampling are used for the consistency loss.
Table shows the mloU comparison of baseline, AugMix without GTs (AugMix") DJMix, x, and DJMix with random augmentation without GTs (DJMix+RA') We omitted GTs, because defining the targets for mixed images that differently applied GTs are not trivial for segmentation and detec- tion tasks. AugMix w/o GT uses pairs of original and augmented images, i.e.. the width is set to 2. As can be seen, DJMix improves the robustness, especially when combined with extra data augmen- tation. In cases, such Gaussian noise, shot noise, and impulse noise, DJMi x RA markedly

---

Under review as a conference paper at ICLR 2021
Table Test error rates (J) comparison on clean test sets (CIFARI0 and CIFAR100) and corrupted sets (CIFARIO- 0- and CIFAR100-C), AugMix w/o GT is a variant of AugMix without using GTs. DJMix+RA is a variant of DJMix x with random data augmentation.
Baseline
mixup
AugMix
AugMix w/o
DJMix
DJMix+RA
CIFAR10
MideResNet
4.65
4.45
5.07
6.14
5.02
4.74
DenseNet
5.12
5.03
5,55
6,54
5.45
4,98
ResNeXt
4.13
3.66
4.17
5.42
4.82
4,31
CIPAR10-C
WideResNet
29.6
23.4
11.1
15.7
11.5
DenaeÑet
32.7
27.3
12.1
19.0
17.6
12.9
ResNext
30.6
26.1
10.7
18.4
14.4
11.1
00
Wi WideResNet
24.3
24.2
25.6
27.6
25.7
24,4
DenseNet
25.4
25.4
26.3
27.8
ResteXt
22.0
21.0
22.3
24.5
25.1
23.2
CIFARL00-C
WideResNett
56.4
51.6
36.4
46.0
44.3
38.3
DenseÑet
59.3
54.4
37.7
47.3
47.2
39.9
ResNeXL
56.2
50.8
34.1
42.1
35.9
Table 2:
Test error rates (4) on ImageNet and Corruption Error ()) on ImageNet using Re Net- DJMix works ImageNet and improves the robustness.
Cloan
Gauss.
Shot
Imguise
Defacus
Gloss
Mocion
7oom
Snow
Frost
Fog
Bright
Contrist
Pixel
JPRG
Average
Reseline
23.2
76.0
27.7
78.8
78.1
88.9
80.7
81.0
80.4
78.4
70.7
61.7
73.9
72.5
76.3
76.8
DJKLX
23.6
12.5
74.
74.7
76.1
86.2
79.9
30.8
77.0
76.0
68.6
60.9
70.2
71.1
76.1
74.6
DJMiXARA RA
23.8
66.8
07.5
68.9
71.
85.1
738
73.1
73.2
72.1
62.4
89.2
078
70.6
73.9
70.4
enhances the performance from DJMix and AugMi w/o GT, which implies the importance of the combination of spsc-fpecific and task-agnostic augmentation practice.
4.3 DETECTION
We trained Faster-RCNN (Ren et al., 2015) on FASCAL-VOC. The consistency loss between the output logits of the backbone network is used for training. Table 4 shows that DJMix yields better performance on almost all corruption types. As semantic segmentation, we compare base- line, AugMix* DJMix, and SMMx++RA* Similarly semantic segmentation, DJMix markedly improves the robustness in the detection task.
ANALYSIS
5.1 ABLATION STUDIES
DESIGN OF TASK Loss: The task loss of DJMix presented in Equation (1) is (()) y), but ((fo(z)) y), as AugMix, is also possible choice. We compare these choices in Table (a. b). ((fo()) y) improves the robustness compared with {(fo(x)))) y)
CHOICE OF CONSISTENCY LOss: DJMix uses IS divergence as the consistency loss, but other divergences can also be used the loss function. Here, we compare the performance when JS divergence replaced with KL divergence and L2 distance. As can seen from Table 5 (a, c), JS and KL show similar performance, whereas L2 shows performance degeneration corrupted data.
EFFECT OF DISCRETIZATION: We verify the effect of the discretization of DJMix x using DAE by substituting the standard autoencoder for DAE. Namely, removed vector quantization modules of DAE and pretrained the AE on the training data to minimize the reconstruction error as DAE. Table 5 (a, d) shows that discretization improves CNNs' robustness expected from the Information Bottleneck perspective presented in Section 3.3.
EFFECT OF MIXING: Table 5 (c) shows test emro rates of DJMix without mixing, f.e., 8 0 in Equation (2) where only discretized images are used. The results show that mixing is indispensable retain the performance clean data. We present further experiments betas in Appendix B.

---

Under review as a conference paper at ICLR 2021
Table 3: DJMiX improves the performance of semantic segmentation when input images rupted. We present mloU (1) on PASCAL-VOC (Clean) and Segmentation: C (the rest).
indicates that augmentation without geometrie transformations is used.
Clean
Gauss
Shot
Impulsc
Snow
Frost
Fog
Bright
Contrast
Pixel
JPEG
Average
Baseline AugMix"
73.3
29.5
26.5
22.2
36.4
58.8
65.4
39.8
58.6
58.8
42.7
74.7
39.0
40.3
37.6
21.8
38.6
63.1
68.3
46.7
60.9
62.9
47.9
DJMix
73.5
34.7
35.3
30.3
23.5
38.8
59.2
66.5
40.2
62.3
63.3
45.4
DJMiX RA"
74.4
46.9
47.7
45.3
22.6
41.9
63.6
69.7
46.9
64.6
66.3
51.6
Table 4: DJMix x improves the performance of detection when input images are corrupted. We present mAP (f) on PASCAL-VOC (Clean) and Detection-e (the rest). indicates that augmen- tation without geometric transformations is used.
Clean
Gauss
Shot
Impulse
Snow
Frost
Fog
Bright
Contrast
Pixel
JPEG
Average
Baseline 76,5
37.9
39.7
33.8
46.4
51.7
64.6
70.1
47.3
47.4
51.1
49.0
AugMix"
76.8
38.1
40.0
35.8
45.6
51.1
64.9
70.4
48.0
47.2
52.1
49.3
DJMIX
76.8
40.8
43.1
37.5
45.8
51.9
64.8
71.1
47.6
46.2
47.6
49.6
DJMI*+RA'
76.2
45.6
48.5
44.7
46.8
52.8
65.3
72.2
48.6
47.4
54.7
52.7
5.2
COMPUTATIONAL OVERHEAD OF DJMIX
We find that the computational overhead by the DAE is negligible. However, the number of for- ward passes affects the training time. For instance, the standard training CIFAR10 using WideResNet for 200 epoch requires approximately hour in our environment. DJMix with two forward passes per update takes about hours, and AugMix with three forward passes per update takes about 3 hours. Importantly, DJMix does not modify the components of CNNs as AugMix; therefore, these methods do not affect the test-time speed, which preferable for the real-world applications.
CONCLUSION
In this paper, we have proposed DJMix, a novel task-agnostic approach to make CNN models robust test-time corruption. To achieve task-agnostic robustness, we have used an autoencoder with discretization bottleneck. Unlike previous approaches, the image modification of DJMix does not affect the appearance of images, which is useful for non classification tasks. Experiments have shown that DJMix improves the robustness of CNN models to input noise in semantic segmentation and detection, in addition to classification. We have found that combining task-specific and task- agnostic augmentation methods further improves performance noisy images. We hope that data augmentation for robustness, including DJMi x, bridges research and the real-world practice of deep learning.
Table 5: Test error rates (average / standard deviation, J) CIFAR10 O and CIFAR1 0-C with various ablation settings.
CIFAR10
CIFARIC-C
DUMEX
4
15.7 +
((fo(z).))) y)
DJMLx KL
4.88 4: 0.13
15.3 0.2
DJMLX L2
4.70 - 0.25
20.3 0.5
(d) DJMiX wo discretization
4.44 0.09
28.9 0.4
(e) DUMLX wlo mixing
7.71 :: 0.19
15.3 + 0.0
Baseline
4.65 + 0.12
29.6 0.3
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In ICLR, 2017.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv, 2017.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
Terrance DeVries and Graham W. Taylor. Improved Regularization of Convolutional Neural Net- works with Cutout. arXiv.
M Everingham, M A Eslami, L VanGool, C K Williams, J Winn, and A Zisserman. The Pascal Visual Object Classes Challenge: A Retrospective. In ICCV, 2015.
Ian Fisher and Alexander A. Alemi. Ceb improves model robustness. In ICLR, 2020.
Robert Geirhos, Heiko H. Schütt, Carlos R. Medina Temme, Matthias Bethge, Jonas Rauber, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In NeurlPS, 2018.
Robert Geirhos, Claudio Michaelis, Felix A. Wichmann, Patricia Rubisch, Matthias Bethge, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In ICLR, 2019.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and Harnessing Adversarial Examples. arXiv, 2014.
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoc Le, and Jonathon Shlens. Using Videos to Evaluate Image Model Robustness. arXiv, 2019.
Hariharan, P Arbeláez, I. Bourdev, S Maji, and J Malik. Semantic contours from inverse detectors. In ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: Simple Data Processing Method Improve Robustness and Uncertainty. In ICLR,, 2020.
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely Connected Convolutional Networks. In CVPR, 2017.
Andrew llyss, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NewrIPS, 2019.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv, 2017.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. arXiv, 2017.
Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. In NIPS, 2012.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In ICLR, 2016.

---

Under review as a conference paper at ICLR 2021
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR, 2018.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness object detec- tion: Autonomous driving when winter is coming. arXiv, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurlPS, 2019.
Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NIPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In The 37th Annual Allerton Conference on Communication, Control, and Computing, 2000.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class Learning for Image Classi- fication. ICLR, 2018.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learn- ing. In NIPS, 2017.
Igor Vasiljevie, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the Impact of Blur on Recognition by Convolutional Networks. arXiv, 2016. URL httt:://axxiv.or//aas/.. 1611. 05760.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In CVPR, June 2020.
Saining Xie, Ross Girshick, and Piotr Doll. Aggregated Residual Transformations for Deep Neural Networks. In CVPR, 2017.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness computer vision. In NeurlPS, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empir- ical Risk Minimization. In ICLRR 2018.
Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random erasing data augmen- tation. In AAAL, 2020.
10

---

Under review as a conference paper at ICLR 2021
Image 2C
Reconstructed Image
CNN fe
Image :C
Image 2C
Augmenter
CNN fo
Taok Loss
Label !/
Random
Discretizing.oo AE
Consistency
Augmentation
Consistency
Label
DJMiX
AugMix
Task Loss
Figure 4: Schematic comparison of DJMi (ours) and AugMix x of mixing examples (Hendrycks et al., 2020). Both methods use mixing strategies, but the details are different. Notably, a recon- structed image of DJMix @ is almost indistinguishable from the original image æ.
A ADDITIONAL ABLATION STUDIES
A.1 THE EFFECT OF BETA DISTRIBUTION PARAMETERS
Main experiments used 31) (1.0, 0.5) of Beta distribution for mixing &b fax (1 {))( where 3 ~ Beta B1). Figure 5 shows test error rates on different combinations of the parameters using Wi Net. Larger 3n and smaller 31 yield & close to æ, and vice versa. which is reflected in the results on CIFAR1 O, i.e., clean data. We used (6u, B1) = (1.0, 0.5), which balances performance on clean and corrupted data.
Error Rates on CIFAR-10
Error Rates on CIFAR- -
1.0
Error Rates
0.5
0.5
10
0.1
0.1
0.1 0.5
1.0
0.1
0.5
1.0
Figure 5: Test error rates on different combinations (Bo, B1) of Beta distribution used for mixing. We used (3o. 31) (1.0, 0.5) in the main experiments.
B EXPERIMENTAL SETTINGS DETAILS
This section describes additional experimental settings.
B. DISCRETIZING AUTOENCODER
We trained the autoencoder for 200 epochs to minimize the reconstruction error, and its codebook is updated by exponential moving average. The hyperparameters are identical to Razavi et al. (2019).
B.2 CLASSIFICATION
We trained models CIFAR10, CIFAR100, 00, and Ima Imag For CIFAR10 and CIFAR1 C0, we used DenseNet BC (k 12, d = 100) (Huang et al., 2017), WideResNet 28 (Zagoruyko & Komodakis, 2016) and ResNeXt- -9 (Xie et al., 2017). We trained these networks for 200 epochs using stochastic gradient descent with a momentum 0.9 and setting an initial learning rate to 0.1 that decays cosine annealing with warm restart (Loshchilov & Hutter, 2016). We set weight decay to x 10 and a batch size to 128. We used data augmentation of random horizontal flipping, random cropping, and random erasing (Zhong et al., 2020) by default. For TmageNet,
11

---

Under review as a conference paper at ICLR 2021
used ResNet-50 (He al., 2016) and trained it for 100 epochs using SGD with momentum of 0.9 and a weight decay of 1 x 10- We set the batch size to 1,024 and an initial learning rate to 0.4 that decays at 30, 60, and 90th epochs. We used random cropping and horizontal flipping the base data augmentation.
When training ResNet-50 ImageNet, used automatic mixed precision (AMP) imple- mented in PyTorch v1. 6 to save the GPU memory consumption. We also used AMP for seman- tic segmentation and detection tasks.
B.3 SEMANTIC SEGMENTATION
We trained DeepLab-v3 (Chen et al., 2017) for 30 epochs with batch size of 32 and a learning rate of 1.0x 10-3 We used SGD with a momentum 0.9 and set its initial learning rate to 0.02. The learning rate is multiplied by factor of (1 iteration y0.9 Chen et al. (2017). See https: // github. com/ http:///trcciiiio//ttteeeeeeeeee
for fur- www.tmaaaee/er.aeaam /
ther details.
B.4 DETECTION
We trained Fastez-RCNN (Ren et al., 2015) for 26 epochs with a batch size of 32 and a learning rate 1.0 10-3 The learning rate divided by 10 at 16th and 22nd epochs, while the first 1,000 iterations are the warmup period. See https://oa //gpttuu// wttp/otttpp//eeeee wwwp//pyoorr//viiio//rreeo master www.eeereeeeeeeeeeeee
on for further details.
B.5 RANDOM AUGMENTATION
We used random augmentation (RA) as task-specific data augmentation, which is orthogonal to DJMix x. Basically, we followed the data augmentation module of AugMix, x, and the only difference the width. Whereas AugMix sets the width to 3, DJMi uses the width of 1, i.e. only single stream of operations is applied to each input image. Each image augmented by RA is used as an input æ to DJMix.
12


Reviews:
[{"review_id": "xvMuE0vZA7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the authors propose to linearly combine the following: the original image, and a discretized version thereof, to form an augmented image. Using the established pairing of a distance-based loss (for pushing logits from augmented/unaugmented data to be similar) and a classification loss, they show improved performance on a range of benchmarks that aim to test robustness to natural noise. \n\nOverall, linearly combining an image with a discretized representation of itself is a neat trick to produce an augmentation where the contours and lines of an object are identical. There is a reasonable analysis of the choices made, and the method is tested in multiple image processing domains — showing broadly positive results. The authors frame the aim of the work as needing 'robust methods that are agnostic to test-time noise a-priori'. While this framing is arguably unmet, the motivation is clear and interesting. \n\nTaken as a whole, the lack of space dedicated to analyzing the results is detrimental to the work. There are 5 separate tables in the main work, each of which only receives minimal discussion in the text. While simplistic conclusions are drawn from the tables (\"table 4 shows ... better performance on almost all corruption type\") examples that don't adhere to the general trend are ignored — why is the 'pixel' case in table 4 better for the baseline, for instance? Also, there is no indication of any sort of variance of the results — while this can be resource-intensive, it would be welcomed at least for CIFAR datasets. I would recommend more discussion of results and a more thorough examination of the trends in the results. To make space for this, I would recommend reducing §4. The methods are fairly well-explained before this, and multiple pages are spent on details that could safely be moved to the appendix. \n\nThere is also a lack of theoretical analysis of this method. For instance, the task loss is different to the augmix baseline — but this is not really explored except by experiment. Are there reasons why the different loss might be theorized to improve results? \n\nWhile Table 3 and 4 show results against an augmix without geometric augmentations (which is reasonable), I would recommend some discussion of how you COULD use geometric transforms for these methods. For instance, could you do some sort of mapping between pixels for the consistency loss (given by the generating transform of the augmentation)? \n\nIn general, the robustness of the proposed method is slightly overstated. Simply performing better on some of the corrupted datasets (and not all of them) is not enough for the sort of guarantees that would matter for truly safety-critical operations. This is especially true when the corruptions are applied one-at-a-time. I feel that the paper should at least acknowledge work on provable guarantees of robustness and explain why that type of analysis is irrelevant here, or talk about failings of previous models that this work highlights. \n\n*minor points:*\n* Throughout the manuscript, the authors refer to the fact that the augmented images are 'nearly impossible to distinguish' from the un-augmented images. Does this hold for humans and machines both? How well can you train a simple classifier to spot this? \n* I recommend a simplified explanation of the effect on segmentation — something like \"because the proposed augmentation does not 'move' the image at all, the edges are in the same place, leaving the labels untouched. \n* 'we omitted five operations' — please list all of the omitted operations in the main text. \n* 'ignore malicious features' — I would choose a different word than 'malicious'. \n* 'for the real-world application of CNNs...' I would contend that CNNs are already very-much used in the real world! \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tnjD9_L9VGG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes DJMix, a data augmentation method to improve the robustness by mixing each training image and its discretized one, and analyzes DJMix theoretically from the Information Bottleneck perspective. In addition, it creates datasets, Segmentation-C and Detection-C, to benchmark the robustness of CNN models on semantic segmentation and detection tasks. \n\nAlthough it achieves good results in some settings, e.g. CIFAR 10 in Table 1, when combined with Random Augmentation (RA), it looks like the overall results are just comparable even slightly worse than AugMix. In addition, the paper may also need to present results with RA only as good results are almost all achieved by DJMix + RA.  Knowing the sole effect of RA is also important. In addition, the results on ImageNet in Table 2 don't compare with other methods, e.g. AugMix, making it weaker. Results of AugMix in Table 3 and 4 don't use geometric transformations doesn't make sense to me as it can achieve good results according to Table 1. Comparisons with other methods are also not enough in Semantic segmentation and detection tasks. \n\nGiven the aforementioned evidence, I think it should add more experiments to support its argument. Therefore, I prefer to reject this paper until it has more solid experiments. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "t1ZTKPi3jmC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThis paper proposes an image augmentation strategy DJMix for helping the network robustness in image recognition tasks over the corrupted images. The experiments show it performs better than AugMix on segmentation and detection over pascal dataset. \n\n#### Novelty\n\nThe method is simple while effective as demonstrated with experiments, though it is similiar with Augmix. The idea could related to some robust training methods proposed. for example, Improving Adversarial Robustness by Data-Specific Discretization, which should also be discussed. \n\n#### Writing\n\nWriting is easy to follow， but the figure is comparably small (For example, images in Fig.1, 3), the author might need to adjust the size a bit. \n\n\n#### Experiments\n\nThe classification results seem to be limited，In classification, it seems the results are comparably worse than mixup (CIFAR10) and for segmentation, the baseline results is only 76.8 which is 12 percent lower than current SoTA algorithms. It would be more strong by comparing with these networks.  \n\nFor more experiments, I recommended experiments over larger dataset such as Coco detection and segmentation tasks, in addition, it could be better comparing with other augmentation strategy such as AutoAug and neural architecture searched. \n\npage 5: pretrinaed -> \"pretrained\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "O1SCiRwlry3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary \nThis paper proposes an alternative data augmentation method for robust image classification. Training on augmented data has been shown to improve robustness. The proposed method mixes discretized images with real images and uses a consistency loss to enforce smoothness among those predictions. Experimental results include classification, detection and segmentation. Section 5 includes ablation studies on the choice of consistency loss and choice of discretization.  \n\n# Strong and Weak points\n\n## Strong points\n\n  * The empirical results include numbers of both classification, detection and segmentation. Thus showing the benefits of the proposed methods on a range of application areas. \n  * Figure (3) shows quantifiable increase in robustness to perturbations based on fixed Fourier coefficients. Even though the phenomenon, as first described in Yin 2019, is not fully understood, this is an important step to make CNNs more robust. \n\n## Weak points\n\n  * In introduction, the argument is made that other methods (like AugMix and other augmentations) require hand crafted image transformations, such as rotation and solarizing. This paper proposes to replace the hand crafted image transformations with another neural network. However, it remains unclear how the known failure cases of neural networks (as described in the second paragraph of the introduction) do not also affect the auto encoder used for the image transformations.\n  * As seen in Figure 2, a critical difference between DJMix and AugMix is the design of the Task loss. In AugMix, the task loss penalizes the prediction on clean data, in DJMix, the task loss penalizes the prediction on augmented data. However, I miss the reasoning, either theoretical or empirical, to justify this choice. How do the methods differ when using clean or augmented data for the task loss?\n  * The baseline results do not match results published in literature. Table 1 notes that AugMix scores 36.4, for a WideResNet on CIFAR 100. However, the original paper, [1] reports 35.9 on the same experiment. I include the link to the paper below. Perhaps I am mistaking something here, but it’s curious that these results are not the same. \nIn either case of 35.9 or 36.4 being the correct number, both results are lower (better) than proposed method scores on the experiment, which scores the 38.3. I miss the reason why the proposed method scores a higher (worse) result on this experiment?\n  * Table 2 makes a comparison against a weak baseline. The results are 76.8 for the baseline, and 70.4 for the proposed method. (Lower is better, so this would exemplify an 6.4 points improvement.). However, despite this paper citing AugMix multiple times, this table fails to include the results from the AugMix paper ([1]) on the same experiment. For most of the augmentations, AugMix [1] scores lower (better) than the baseline reported in this table (see Table 2 in [1]). Having omitted these comparisons, the evidence we can draw from Table 2 is limited. \n  * Comparisons to traditional discretization methods are omitted. Overall, this method proposes to augment images using discretization. For the discretization, the “Vector-Quantized Variational AutoEncoder” is used. However, I miss a few intermediate steps. For example, a traditional auto encoder, as proposed in 2006 [2]. Moreover, how about traditional discretization techniques like JPEG compression? Given that JPEG compression has been around since 1992 [3], a comparison with this traditional method would increase the argument for the proposed method. \n\n[1] https://arxiv.org/pdf/1912.02781.pdf\n[2] Hinton and Salakhutdinov, 2006. “Reducing the dimensionality of data with neural networks”. Science.\n[3] https://en.wikipedia.org/wiki/JPEG \n\n# Statement\n\nRecommendation: Reject\nReasons\n  * The results miss comparisons to traditional techniques. For discretization, a “Vector-Quantized Variational AutoEncoder” is used, but the argument for not comparing to a traditional auto encoder, or even to a traditional compression technique like JPEG misses from the main text. \n  * The empirical results are not comparable to other reports in literature. Table 1 compares on CIFAR-C with AugMix, but does not match results reported in that paper. Table 2 compares on ImageNet-C against a weak baseline, and omits the results on ImageNet-C for AugMix [1] and other publicly available methods [2].\n\n\n[1] https://arxiv.org/pdf/1912.02781.pdf\n[2] https://paperswithcode.com/sota/domain-generalization-on-imagenet-c \n\n\n# Clarifying questions\n\n  * The first paragraph of the introduction speaks of noisy images in “the real world”, the first paragraph of the Related Work speaks of “the real-world application of CNNs”, and the conclusion speaks of “real-world practice of deep learning”. However, the evaluations restrict themselves to ImageNet-C, which uses synthetic perturbations. There exist many more evaluations of robustness on the ImageNet label space: ImageNet-A [1], ImageNet-V2 [2], ObjectNet [3], and ImageNet-Video-Robust [4] . What is the reason for not including these evaluations when claiming “real world” robustness?\nFinally, the text cites the texture/shape evaluation of Geirhos 2019 twice in the main text, but also that evaluation misses in the results section [5].\n  * How does the Theorem 1 relate to your application of the auto encoder? From equation (2), I understand that $\\hat{x}$ is a convex combination of the discretized image and the original image. The theorem states that the MI between input, x, and latent code, z, is upper bounded by K, but what does that tell us about $\\hat{x}$?\n  * The ablations in Table 5 explore the design choices for the proposed methods. However, in two cases, the ablation experiment scores better than the proposed method  (“DJMix w/o mixing”, 15.3; “DJMix w/ KL”, 15.3). What arguments are there for the design choices made in light of these numbers?\n\n[1] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv, abs/1907.07174, 2019\n[2] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.\n[3]  Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, 2019\n[4] Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. A systematic framework for natural perturbations from videos. arXiv, abs/1906.02168, 2019\n[5] Robert Geirhos, Claudio Michaelis, Felix A. Wichmann, Patricia Rubisch, Matthias Bethge, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019\n\n\n# Minor feedback\n\nThis minor feedback is not part of my assessment.\n\n  * “Jansen–Shannon divergence” -> “Jensen–Shannon divergence”\n  * The final paragraph of Section 3.1 and the final paragraph of Section 3.2 state the same text. Is this a mistake?\n  * “models learn vulnerable representation” -> “models learn vulnerable representations”\n  * “Pretrinaed” -> “pretrained”\n  * “Degenerated” -> “perturbed”? (Section 4.0.2)\n  * (Section 4.0.2) Reference to “Michaelis et al.” misses the year, 2019.\n  * “Nonclassification” -> “non-classification” or “non classification”\n  * The loss function in equation (1) is the same as the loss function in equation (1) of the AugMix paper. Even the choice for the JSD is the same. However, this connection is not mentioned in the main text. Given that you evaluate extensively against the AugMix results, please credit them appropriately for this design of the loss function. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to improve the robustness of computer vision models through a new augmentation strategy. There are two primary contributions of the work, first the use of a bottleneck autoencoder to generate discretized variants of the clean image, and second a slight variant of the task loss, where the task loss is evaluated on the augmented image vs the clean image as is done in prior work. Reviewers argued that the method did not meaningfully improve upon prior work, the method alone underperforms AugMix on existing benchmarks, and when combined with some additional augmentations from AugMix the gains were marginal. Additionally, when there were gains in robustness it was unclear as to the source. The work would be improved with additional experimental evidence that the claimed benefit of the information bottleneck is substantial for improving robustness (for example, when DJMix+RA outperforms AugMix, is this due to the use the of autoencoder or is it primarily due to the new task loss?). I recommend the authors incorporate additional reviewer feedback and resubmit.
Decision: Reject