Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
Authors: Alexander Robey
Hamed Hassani
George J. Pappas
Venue:  (2021)
Keywords: robustness
out-of-distribution generalization
natural variation
deep learning

Abstract: While deep learning (DL) has resulted in major breakthroughs in many applications, the frameworks commonly used in DL remain fragile to seemingly innocuous changes in the data. In response, adversarial training has emerged as a principled approach for improving the robustness of DL against norm-bounded perturbations.  Despite this progress, DL is also known to be fragile to unbounded shifts in the data distribution due to many forms of natural variation, including changes in weather or lighting in images.  However, there are remarkably few techniques that can address robustness to natural, out-of-distribution shifts in the data distribution in a general context.  To address this gap, we propose a paradigm shift from perturbation-based adversarial robustness to model-based robust deep learning.  Critical to our paradigm is to obtain models of natural variation, which vary data over a range of natural conditions.  Then by exploiting these models, we develop three novel model-based robust training algorithms that improve the robustness of DL with respect to natural variation. Our extensive experiments show that across a variety of natural conditions in twelve distinct datasets, classifiers trained with our algorithms significantly outperform classifiers trained via ERM, adversarial training, and domain adaptation techniques.  Specifically, when training on ImageNet and testing on various subsets of ImageNet-c, our algorithms improve over baseline methods by up to 30 percentage points in top-1 accuracy.  Further, we show that our methods provide robustness (1) against natural, out-of-distribution data, (2) against multiple simultaneous distributional shifts, and (3) to domains entirely unseen during training.

Under review as a conference paper at ICLR 2021
MODEL-BASEI ROBUST DEEP LEARNING: GENERAL- IZING TO NATURAL, OUT-OF-DISTRIBUTION DATA
Anonymous authors Paper under double-blind review
ABSTRACT
While deep learning (DL) has resulted in major breakthroughs in many appli- cations, the frameworks commonly used in DL remain fragile seemingly in- nocuous changes in the data. In response, adversarial training has emerged as a principled approach for improving the robustness of DL against norm-bounded perturbations. Despite this progress, DL is also known to be fragile to unbounded shifts in the data distribution due to many forms of natural variation, including changes in weather or lighting in images. However, there are remarkably few techniques that can address robustness to natural, out-of-distributinn shifts in the data distribution in general context. To address this gap, we propose a paradign shift from perturbation- based adversarial robustness to model-based robust deep learning. Critical our paradigm is to obtain models of natural variation, which vary data over a range of natural conditions. Then by exploiting these models, we develop three novel model-basee robust training algorithms that improve the ro- bustness of DL with respect to natural variation. Our extensive experiments show that across a variety of natural conditions in twelve distinct datasets, classifiers trained with our algorithms significantly outperform classifiers trained via ERM, adversarial training, and domain adaptation techniques. Specifically, when train- ing on ImageNet and testing various subsets of ImageNet-c, our algorithms improve over baseline methods by up to 30 percentage points in top-1 accuracy, Further, we show that our methods provide robustness (1) against natural, out-of- distribution data, (2) against multiple simultaneous distributional shifts, and (3) to domains entirely unseen during training.
INTRODUCTION
The last decade has seen remarkable progress in deep learning (DL), which has prompted wide- scale integration of DL frameworks into myriad application domains (LeCun et al. 2015). In many of these applications. and in particular in safety-critical domains, it is essential that the DL systems are robust and trustworthy (Dreossi et al. 2019). However, it is now well-known that DL is fragile to seemingly innocuous changes the input data (Szegedy al. 2013). Indeed, well-documented examples of fragility to carefully-designe
noise can be found in a variety of contexts, including image classification (Madry et al. 2017). clinical trials (Papangelou et al. 2018), and robotics (Melis et al. (017). Accordingly, number of adversarial training algorithms (Goodfellow et al. 1bb Wong Kolter 2017) as well as certifiable defenses (Raghunathan al. 2018 Fazlyab et al, 2019a) have recently been proposed, which have provided rigorous framework for improving the robustness of DL against norm- norn-bounded perturbations (Fazlyab al. 2019b Dobriban al. 2020
Despite this encouraging progress, very recent papers have unanimously shown that DL is also frag- ile unbounded shifts in the data-distr due to a wide range of natural phenomena (Djolonga et al. 2020 Taori et al. 2020 Hendrycks et al., 2020 Hendrycks & Dietterich] 2019). For example, in image classification, such shifts include changes due to lighting, blurring, or weather conditions Pei et al. 2017| Chernikova et al. 2019). However, there are remarkably few general, principled techniques that have been shown to provide robustness against these forms out-of-distribution, natural variation (Hendrycks et al. 2019a). Furthermore, these unseen distributional shifts arguably more common in safety-critica domains, the task of designing algorithms that generalize to natural, out-o--distributiin data is important and novel challenge for the DL community.

---

Under review as a conference paper at ICLR 2021
Perturbation-basee adversarial example. In perturbation-aase robustness setting, an input datum (left) is pereeptually indistinguishable from a corre- sponding adversarial example (right).
(b) Natural variation. this paper, we study ro- bustness with respect to natural variation. For exam- ple, differences in weather conditions such as snow illustrate one form of natural variation.
Figure I: A new notion of robustness. Past work has focused on cnturrattion-basele adversarial examples, such as Figure Ta In this paper, we focus on robustness with respect natural variation, shown in Figure 1b. which often does not obey perceptual or norm-bounded constraints.
In this paper, we propose a paradigm shift from perturbation- adversarial robustness to model- based robust deep leurning. Our goal is to provide principled, general algorithms that can be used to train neural networks to be robust against natural, cut-of-distribution shifts in data. Our experiments show that across variety of challenging, naturally-occuma conditions, such variation in light- ing, haze, rain, and snow, and across various datasets, including SVHN, GTSRB, CURE-TSR, and ImageNet, classifiers trained with model-based algorithms significantly outperform standard DL. baselines, adversarially-traan
classifiers, and, when applicable, domain adaptation methods.
Contributions. The contributions of this paper can be summarized as follows:
Paradigm shift. We propose paradigm shift from perturbation-base robustness to robust deep learning, where models of natural variation express changes due natural conditions. Optimization based formulation. We formulate a novel model-based robust training problem by constructing general robust optimization procedure to search for challenging natural variation. Models of natural variation. For many challenging forms of natural variation, we use deep generative models to learn models of natural variation that are consistent with realistic conditions. Novel algorithms. We propose family of novel robust training algorithms that exploit models of natural variation to improve the robustness of DL against worst-case natural variation.
-uttr--distribution robustness. We show that our algorithms are the first to consistently provide robustness against natural, out-of-distribution shifts that frequently occur in real-world environ- ments, including snow, rain, fog, and brightness on SVHN, GTSRB, CURE-TSR, and ImageNet.
ImageNet-c robustness. We show that our algorithms can significantly improve the robustness of classifiers trained on ImageNet and tested on ImageNet-c by as much as 30 percentage points.
Robustness to simultaneous distributional shifts. We show that our methods are composable and can improve robustness to multiple simultaneous sources of natural variation. To evaluate this feature, we curate several new datasets, each of which has two simultancous distributional shifts. Robustness to unseen domains. We show that models of natural variation can be reused on datasets that are entirely unseen during training to improve -ut-t--distiiiution generalization.
PERTURBATION-BASEE ROBUSTNESS: APPROACHES AND LIMITATIONS
In this paper, we consider a standard classification task in which training data (æ,, v) D is distributed according to a joint distribution D over instances x E Rd and labels E [k] == (0, "o.1.....k) .).. this setting, given finite training sample drawn i..d. from D, the goal of the learning problem is to obtain classifier fm. parameterized by weights w E R² such that f.. can correctly predict the labels y corresponding to new instances 2 drawn i.i.d. from D. In practice, one can learn fur by approximately solving the non-convex empirical risk-minimization (ERM) problem arg min, [l yi w)] where is suitable loss-function. However, neural networks trained using ERM are known to be susceptible to adversarial attacks. This means that given datum with corresponding label y, one can find another datum =d such that x is close to ndv in a given Eu- clidean norm and zadv is predicted by the learned classifier as belonging If such datum adv exists, it is called an adversarial example. This is illustrated Figure la although these pandas look identical, they were classified differently in (Goodfellow et al. 2014b
different class
y/
2

---

Under review as a conference paper at ICLR 2021
Input image x
Output image x
Image
Nuisance
Model of natural variation G(x, 8)
Generated images
parameter
(h) Input image x and corresponding generated images for a learned model of natural variation on ImageNet.
(a) Models take the form G(æ, 8). where is muisance parameter that describes how the output image x' == G(x, 6) is varied.
Figure this paper, we introduce models of natural variation to describe natural transformations.
The dominant paradigm toward improving robustness against adversarial examples relies on a ro- bust optimization perspective wherein neural networks are trained to correctly classify worst-case perturbations of data (Madry et al. 2017 Wong Kolter 2017). This can formulated as follows:
arg min (x,g)>
max {(= + y; w) SEA
We can think of (1) as comprising two coupled optimization problems: an inner maximization prob- lem which we seek a challenging perturbation and an outer minimization problem which we seek weights that lead to strong classification performance.
robustness. Despite remarkable progress toward improving the robustness of DL against norm-bounded perturbations, there are significant limitations to adversarial training. Notably, DL is known to be fragile to many forms of natural variation, which cannot be described by small perturbations
Limitations of perturbation
8. In image classification, such natural variation includes changes in weather or background color (Eykholt et al.) 2018 Hendryeks et al.) 2019b; Hosseini & Poovendran 2018), spatial transformations such rotation or scaling (Xiao et al.) 2018b) Kari- anakis et al.] 2016p. and sensor-based attacks (Kurakin et al.] 2016). Because such transformations frequently arise in safety-critical domains, it critically important for the DL community develop algorithms that are robust against out-of- di-diiitiitttinn natural variation in data. In this paper, we specifically address this challenge by proposing principled, optimization based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation.
3 A NEW ROBUSTNESS PARADIGM: MODEL-BASEL ROBUST DEEP LEARNING
Underlying the task of improving the robustness of neural networks against natural, out-of- distribution data two fundamental challenges. Firstly, unlike in the adversarial robustness com- munity, in real-world, safety-critical environments, data can vary in unknown and highly nonlinear ways. Thus, the first step toward building robust training procedure must be design mechanism that accurately describes how data varies in such environments. Next, assuming suitable model of natural variation, the second challenge is formulate training procedure that exploits this model toward improving robustness. In this section, we present novel solutions to each of these challenges.
3.1 MODELS OF NATURAL VARIATION
In order to effectively model sources of natural variation in domain-agnostic setting, we will ab- stractly define models of natural variation. Concretely, a model of natural variation G(z, 6) is map that describes how an input datum can be naturally varied by a nuisance parameter 8 resulting in new datum x' = G(r, 5). Ideally, for fixed datum varying the nuisance parameter å should vary the severity of the natural conditions in the generated datum x'. An example of such a model the left (in this case, in sunny weather) can be naturally varied by d and consequently transformed into the image x' on the right (in snowy weather). In the remainder of this subsection, consider cases in which (1) model G is known priori, and (2) model G is unknown and therefore must be learned offline from data. In this second case in which models of natural variation must be learned, we propose method for obtaining these models.
shown in Figure 2 where image
Known models of natural variation. In many problems, a model G(r, 6) is known a priori due to intrinsic geometric structure. For example, there underlying structure that describes how data can
3

---

Under review as a conference paper at ICLR 2021
be rotated, translated, or scaled; models for rotating an image can be characterized by G(z,, 5) R())x where R()) is a rotation matrix and ó E A := [0, 2T). In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al. 2017 Kamath et al. 2020).
Learning models of natural variation from data. In many situations, models natural variation are not known a priori are too costly to obtain. For example, consider Figure in which model G(r; 5) takes an image x of a street sign in sunny weather and maps it to an image x' := -Gz 6) in snowy weather. Even though there is a relationship between the two images, obtaining model G relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for learning the model G from data. To do so, we assume that we have access to two unpaired domains A and B that are drawn from common distribution. Domain A contains the original data, such as the images with sunny weather, and domain contains naturally transformed data, such as images with snow. Ideally, model of natural variation should learn to map images from domain A to corresponding images with different levels of natural variation captured by the images of domain B. In our experiments section, we rely on the MUNIT framework (Huang et al. 2018), which combines two autoencoders and two generative adversarial networks GGodffello et al. 2014a), to learn mappings between domains A and B. Furthermore, we note that many choices unpaired, unconditional image-to tore-tto-nngg to-image translation networks satisfy our criteria for G, and in future work plan to investigate the efficacy of these architectures. In Appendix A describe parallel experiments that we carried out with two other architectural choices for G, and we fully characterize the MUNIT architecture used in our experiments.
3.2 MODEL-BASED ROBUST TRAINING FORMULATION
The model-based robust training paradigm that we propose retains the basic elements of adversarial training described in Section g Our point of departure from the classical adversarial training formu- lation is in the choice of the so-called adversarial perturbation. In this paper, we assume that data can be transformed according to a model of natural variation G(xr, 5) by choosing different values of 8 from given nuisance space A. The goal of the model-based approach is to train classifier that achieves high accuracy both on a test set drawn i..d. from D and on more-challenging test data that has been subjected to the source of natural variation that G' models. This perspective can be captured by the following robust optimization problem:
min E (a max ((G(z, 0), v))
(2)
SEA
In the inner maximization problem of this formulation, given an instance-label pair (xr, y). we seek vector 8* E A that produces corresponding instance x' G(r, 5") which gives rise to high loss values (1, 8²),, y/; w) under the current weight w. One can think of this vector 8* as characterizing the worst-case nuisance that can be generated by the model G(xr,*)) for the original instance x. After solving this inner problem, we solve the outer minimization problem by finding weights w that minimize the risk against the challenging instance G(z, 8*). By training the network correctly classify this worst-case data, the goal is to become invariant to the model G(I, 5) for any ò E A.
4 MODEL-BBASD TRAINING ALGORITHM
We now assume that we have access suitable model of natural variation G(x, 8) and shift our attention toward exploiting G in the development of novel robust training algorithms. In the empir- ical version of @ rather than assuming access to the full joint distribution D. we assume that we given given set of i...d samples D,, ((z), Uy B,)b drawn from D. Thus we have:
w² E arg min WERK
max (G (),,)).3337 w) SEA
Note that when w parameterizes neural network, is a cinconvex-nonccncave min-max prob- lem, which is difficult to solve exactly. Thus, we resort to approximate optimization techniques for solving (3) Specifically, we propose three algorithmic variants: Modei-based Adversarial Train- ing (MAT), Model-based Robust Training (MRT), and Model-based Data Augmentation (MDA). Pseudocode for MAT is given in Algorithm I pseudocode for MRT and MDA as well as further discussion of these algorithms is given in Appendix B At high level, each of these methods alter- nates between solving the outer minimization problem and the inner maximization problem. To this
4

---

Under review as a conference paper at ICLR 2021
Algorithm Model-based Adversarial Training (MAT)
Input: Data sample D, = 85)3,, number steps k. step size a Output: Learned weight to
1: repeat
2: for minibatch Bm == {(I1, (111,,1)... (I2, W2). (Imi (mm) C. Dr, do
3: Initialize == (61, ......, . 6m) f (0g. 0g......0. .0......))).. 0g)
4: for k steps do
5:
g 5 V Σ (Z) (0j,995. w)
6: end for
- a + agl
II denotes projection onto the set A
7:
8:
+ V - [[] Uy + X Kay My: Uy: w)]
9:
w + Update(g, w)
# Update can be SGD, Adam, Adadelta, etc.
end for 11: until convergence
end, each of these algorithms uses SGD to solve the outer problem; the methods differ in how they seek solutions to the inner problem, and in what follows, describe each of these procedures in more detail. In each algorithm, given a datum (I, y), the solution 8" to the inner problem is used to create new datum (G(z, 8*), y) that added the training set before solving the outer problem.
Model-based Adversarial Training. In MAT, we seek an exact solution to the inner problem by performing k steps of gradient ascent in the objective ((G(,, 8), yj w). The resulting nuisance parameter 8* is one that causes ((G(x, 8²), yr w) to have high loss under the current weight w.
A for [k). We then select the i E [k] such that ((G(z, 6;-) ), y; w) is maximized. In this way, rather than exactly solving the inner problem, MRT uses sampling-bass based approach to finding challenging data G(r, 6,-).
Model-based Robust Training. In MRT, we first randomly sample
Model-based Data Augmentation. In MDA, rather than explicitly trying to solve the inner prob- lem, we seek a diversity of naturally-varying data rather than the "worst-case." In this way, MDA samples 8; E A for i E [k] and then appends (G(x, 6t), k) to the training dataset.
EXPERIMENTS
We present experiments in five different settings twelve distinct datasets demonstrate the broad applicability of MBRDL. First, in Sections 5 5.2, we show that our algorithms are the first consistently provide out-of-distribution robustness across range of challenging corruptions, in- cluding shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3 curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4 we show that models of natural varia- tion trained fixed dataset can be reused to provide robustness datasets entirely unseen during training. Finally, in Section 5.5 we assume access to unlabeled data corresponding to fixed domain shift, and compare algorithms to suitable basclines, including domain adaptation methods. Throughout these experiments, we use the notation "source (A-B)) to denote distributional shift from domain A to domain B. For example, "contrast (low (low~<iggh)" will denote shift from low-contrast to high-contrast. Images from domains A and for each of the shifts used in this paper available in Appendix A We note that experiments contain domains with both natural and variation; details concerning how we extracted non-artificial variation can be Architecture and hyperparameter details are given in Appendix
found in Appendix D
5.1 OUT-OF-DISTRIBUTION ROBUSTNESS
In many applications, one might have data corresponding low levels of natural variation, such as dusting of snow in images of street signs. However, it is often difficult to collect data corresponding to high levels of natural variation, such images taken during blizzard. In such cases, we show that our algorithms can be used to provide significant di-triiibbtioon robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of
5

---

Under review as a conference paper at ICLR 2021
Table 1: Out-of-distribution robustness. In each experiment, we train model of natural variation to map from challenge-level 0 to challenge-level 2 data from different subsets of CURE-TSR. We then perform model-based training using challenge-level 0 data and test on challenge-levels 3-5.
Test accuracy (top- levels 3. 4, and 5
CURE-TSR subset
ERM Aug
PGD Aug
MRT
4
4
Snow
86.5
74.8
60.9
82.9
77.3
61.8
88.0
77.8
70.7
Haze
55.2
54.0
47.5
83.8
63.1
53.4
83.9
79.1
70.1
Decolorization
87.9
85.1
78.8
84.7
75.2
64.9
90.5
89.6
89.4
Rain
72.7
71.7
66.9
68.9
66.4
60.5
80.7
78.7
74.8
Table 2: ImageNet to ImageNet-c robustness. In each experiment, we train a model of natural variation to map from classes 0-9 of ImageNet to the same classes from subset of ImageNet-c. Next, we use this model to perform model- training on classes 10-59 of ImageNet, and we test each network on classes 10-59 from the same subset ImageNet-c on which the model was trained.
Model dataset (classes 0-9)
Training dataset
Test dataset
Test accuracy (opp-//oo-S)
(classes 10-59)
(classes 10-59)
ERM
AugMix
MDA
Snow
Snow
20.9
49.9
1.10
8.3
31.1
61.2
Contrast
Contrast
41.1
73.4
0.72
6.76
50.0
79.5
ImageNet
Brightness
Brightness
26.9
59.2
0.56
5,20
53.0
81.7
Frost
Frost
16.3
39.0
29.5
58.4
36.0
67.2
natural variation. To do so, we use data from the CURE-TSR dataset (Temel et al, 2019). which contains images of street signs divided into subsets according various sources of natural variation and corresponding severity levels. For example, for images the "snow" subset, level corresponds snow, whereas level 5 corresponds unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSE TSR. We then train classifiers using MDA with labeled level data. We also train classifiers using ERM and PGD using the labeled data from levels 0 and 2. We then test all classifiers on data from levels 3, 4, and 5. Note that while this is an unfair comparison for our methods, given that the model-based algorithms are not given access to labeled level 2 data, our algorithms still outperform the baselines by as much as 20 percentage points on level data.
full blizzard. Thus, for each row of Table D we use
5.2 MODEL-BASED ROBUSTNESS ON THE SHIFT FROM IMAGENET TO IMAGENET-C
To demonstrate the scalability of our approach, we perform experiments on ImageNet DDng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks & Dietterich 2019 ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to corresponding classes ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is recently proposed method that adds known transformations to the data (Hendrycks et al. 2019a).
5.3 ROBUSTNESS TO SIMULTANEOUS DISTRIBUTIONAL SHIFTS
In practice, is common to encounter multiple simultancous distributional shifts. For example, in image classification, there may be shifts in both brightness and contrast; yet while there may be examples corresponding to shifts either brightness or contrast in the training data, there may not be any examples of both shifts occurring simultaneously. To address this robustness challenge,

---

Under review as a conference paper at ICLR 2021
Table 3: Composing models of natural variation. We consider shifts in distinct and simul- taneous sources of natural variation. To perform model-base training, we compose two models of natural variation trained separately on each of the two sources of natural variation.
Dataset
Challenge 1 (dom. -oom. B₂)
Challenge 2
Test acc. (top-1)
(dom. Au-ddmm Bu)
ERM
MDA
SVHN
Brightness -hig))
Contrast (low~>high)
54.9
67.2
ImageNet
IN-c brightness >high
IN-c contrast (high >low)
13.6
49,9
ImageNet
IN-c brightness
IN-c snow (no-tyes)
53.3
58.3
ImageNet
NN-c brightness (low->high
IN-c fog (no-tyes)
50.3
58,8
ImageNet
IN-c contrast (high~llow)
IN-c fog (no -ye-)yys)
8.40
23.2
Table 4: Transferability of model-based robustness. In each experiment, we train model of natural variation on given training dataset D1. Then, we use this model perform model-based training on a new dataset Dj entirely unseen during the training of the model.
Training dataset Dj
Challenge (dom. dom. B)
Test accuracy (top-1)
dataset D2
ERM
PGD
MRT
MDA
MAT
Fashion-
69.3
67.7
81.4
80.1
76.1
MNIST
Background
Q-MNIST
87.0
79.9
98.0
98.0
98.0 0
MNIST
color
E-MNIST
63.5
49.3
86.1
85.9
84.1
(blue->red)
K-MNIST
47.9
47.7
89.1
89.3
86.8
USPS
89.9
87.4
93.3
93.4
91.9
GTSRB
CURE
Brightness thigh~>lowg
47.6
43.6
73.0
72.4
67.8
ImageNet
Snow (no-tyes)
52.0
53.0
59.4
62.2
59.4
CURE
ImageNet-c
Brightness (low (low->high)
41.5
40.2
46.6
46.7
47.5
t we learn two models of natural variation G₁ and G2 using unlabeled training data corresponding to separate shifts, which map domains Ar B₁ (e.g. low- high- brightness) and A2 B2 (e.g. low- to high-contrast). We then compose these models to form a new model G(æ:, 6) G (z, 8) which be used to provide robustness against both shifts simultaneously. We then train classifiers on labeled data from A, and test on data from B nB2- To create the data from B, B2 for the ImageNet experiments, apply pairs of transformations that were originally used to create the ImageNet-c datasets; more details are in Appendix D
for each row of Table
5.4 TRANSFERABILIT OF MODEL- BASED ROBUSTNESS
Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are entirely unseen while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against common source of natural variation. For example, one might have access to two domains corresponding to the shift from images of European street signs taken during the day to images taken at night. However, one might wish to provide robustness against the same shift from daytime to nighttime of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table shows several experiments of this stripe in which model G is learned on one dataset D2 and then applied on another D2; we improve robustness on unseen domains by up to 40 percentage points.
new dataset
5.5 MODEL-BASED ROBUST DEEP LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION
While approach does not require labeled data from domain B, when such data is available, it is of interest to evaluate how our approach compares to relevant methods such as domain adaptation. In Table 5 for each shift from domain A to B, assume access to labeled data from domain A

---

Under review as a conference paper at ICLR 2021
Table 5: In each experiment, we assume access unlabeled data from domain B, which we use to train a model of natural variation. We compare to suitable baselines, including domain adaptation.
Challenge
Test accuracy (top-1)
Dataset
(dom. A d-m B)
ERM
PGD
ADDA
MRT
MDA
MAT
SVHN
Brightness (low- high)
30.5
36.2
60.1
70.9
69.5
52.2
SVHN
Contrast (low-+hight hhigh)
55.9
57.9
54.6
74.3
74.1
55,2
GTSRB
Brightness +high)
40.3
34.7
27.6
50.4
48.3
64.8
GTSRB
Contrast (low~>high)
44.5
41.9
14.7
68.4
69.4
55.1
CURE
Snow (no->yes)
52.0
53.0
16.1
74.0
74.5
72.3
CURE
Haze (no->yes)
57.2
50.9
49.2
72.5
70.0
74.6
CURE
Rain (no (no->yes)
62.6
62.3
16.5
75.2
73.7
75.3
and unlabeled data from domain B. In each row, use unlabeled data from both domains to train a model of natural variation. We then train classifiers using our algorithms, as well with ERM and PGD, using data from domain A and test data from the test set for domain B. Furthermore, we compare to ADDA, which a well-known domain adaptation method TTznng et al. 2017). In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points. Note that while this is one of the most commonly studied settings in domain adaptation, represents only one particular setting which the MBRDL paradigm can be applied.
RELATED WORK
Aside from the algorithms we introduced in Section 4 we are not aware of any other algorithms that can be used to address out-of-distribution robustness across the diverse array of tasks presented in the previous section. However, several lines of research have sought to address this problem in constrained settings or under highly restrictive assumptions. In the domain adaptation literature, various methods have been proposed which rely on the restrictive assumption that unlabeled data corresponding to fixed distributional shift is available during training (Tzeng et al. 2017: Ajakan al. 4. Ganin Lempitsky 2015). Unlike these approaches, our solution does not assume ac- cess to unlabeled data from fixed shift and can be applied to datasets that are entirely unseen during training. Furthermore, several works have used generative models to create adversarial perturbations
(iioo al. 2018a| Lee et al. 2017: Wang & Yu 2019 Samangouei et al. 2018 Jalal et al. 2017) or perceptually - images subject to relatively simple corruptions in specific application domains
(Dunn al. 2019 Song all 2018 Vandenhende et al. 2019 Arruda al. 2019). On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.
Two concurrent works formulate robust training procedures assuming that data is corrupted accord- ing to a fixed generative architecture. Gowal et al. (2020) exploit properties specific to the StyleGAN architecture formulate a training algorithm that provides robustness against color-based shifts on MNIST and CelebA. In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurrin shifts across twelve distinct datasets. [Wong & Kolter (2020) use conditional VAES to learn perturbation sets corresponding to simple corruptions from pairs of images. In our framework we improve robustness against more challenging, natural shifts by learn- ing from unpaired datasets and we do not rely on conditioon to generate realistic images.
7 CONCLUSION
In this paper, we proposed a novel model-based robust training paradigm for deep learning that provides robustness with respect to models of natural variation. Our notion of robustness offers departure from adversarial training with respect to norm-bounded data perturbations. In our exper- iments, we show that our paradigm challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and domains that are entirely unseen while training the model, and shows significant dutttf-distributioo robustness as datasets become more challenging.
provide significant diitr---iitribttionn robustness many
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarial neural networks. arXiv preprint arXiv:1412.4446, 2014.
Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Augmented cyclegan: Learning many-to-many mappings from unpaired data. arXiv preprint 180i: 10151, 2018.
Vinicius F Arruda, Thiago Paixão, Rodrigo F Berriel, Alberto De Souza, Claudine Badue, Nicu Sebe, and Thiago Oliveira-Santos Santos. Cross-domain car detection using unsupervised image-to- image translation: From day to night. 2019 International Joint Conference on Neural Nenworks (LJCNN) pp. 1-8. IEEE, 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: 1.K2:1800000420 2018.
Alesia Chernikova, Alina Oprea, Cristina Nita-Rotaru, and BaekGyu Kim. Are self-driving cars secure? evasion attacks against deep neural networks for steering angle prediction. In 2019 IEEE Security and Privacy Workshops (SPW). pp. 132-137. IEEE, 2019.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-imant translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8188 8197, 2020.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint /rXiv:/812.01718,, 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (LJCNN), pp. 2921-2926. IEEE, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' A'mour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. arXiv preprint arXi::2007.08558, 2020.
Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey. Provable tradeoffs in adver- sarially robust classification. arXiv preprint (ttt:2000605166// 2020.
Tommaso Dreossi, Alexandre Donzé, and Sanjit Seshia. Compositional falsification of cyber- physical systems with machine learning components. Journal of Automated Reasoning, 63(4): 1031-1053, 2019.
Isaac Dunn, Tom Melham, and Daniel Kroening. Generating realistic unrestricted adversarial inputs using dual-objective gan training. arXiv preprint arXiv: 2019.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. arXiv preprint 1rXiv:1712.00279, 2017.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust whrli attacks deep learning visual classification. Proceedings of the IEEE Conference on Computer Vision and Pattrnn Recognition, pp. 1625-1634, 2018

---

Under review as a conference paper at ICLR 2021
Mahyar Fazlyab, Manfred Morari, and George Pappas. Safety verification and robustness anal- ysis of neural networks via quadratic constraints and semidefinite programming. arXiv preprint arXiv:1903.01287, 2019a.
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. In Advances in Neural Information Processing Systems. pp. 11423-11434, 2019b.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189 PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversario training of neural net- works. The Journal of Machine Learning Research, 1 17(1):2096220) 2030, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014a.
Jan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014b.
Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Pushmeet Kohli. Achieving robustness in the wild via adversarial mixing with disen- tangled representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1211-1220, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 1912.0271,, 2019a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv: 0000:0907700100 2019b.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006. 16241, 2020.
Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1614-1619, 2018.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 172-189, 2018.
Jonathan J. Hull. A database for handwritten recognition research. IEEE Transactions on paltern analysis and machine intelligence, 16(5):550-554, 1994.
Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint 192.9:17 2017.
Sandesh Kamath, Amit Deshpande, and KV Subrahmanyam. Invariance vs. robustness of neural
networks. arXiv preprint arXiv:2002. 11318, 2020.
Nikolaos Karianakis, Jingming Dong, and Stefano Soatto. An empirical evaluation of current volutional architectures' ability to manage nuisance location and scale variability. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4442-4451, 2016.
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 6444 6454, 2018.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1607.v:607.02220 2016.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs (Online). Available: htlp://yann. lecun. com/exlb/mmist, 2, 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553);436-444, 2015.
Hsin-Y Ying Lee, Hung-Yo Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuun Yang. Diverse image-to-imans translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pp. 35-51, 2018.
Hyeungill Lee, Sungyeob Han, and Jungwoo Lee. Generative adversarial trainer: Defense to adver- sarial perturbations with gan. arXiv preprint arXiv: /00. 1705.03387, 2017.
Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multi- variate anomaly detection for time series data with generative adversarial networks. Interna- tional Conference on Artificial Neural Networks, pp. 703-716. Springer, 2019.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0660 2017.
Qi Mao, Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming- Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4439-437, 2019.
Marco Melis, Ambra Demontis, Battista Biggio, Gavin Brown, Giorgio Fumera, and Fabio Roli. Is deep learning safe for robot vision? adversarial examples against the icub humanoid. In Proceed- ings of the IEEE International Conference on Computer Vision Workshops, pp. 751-759,
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Konstantinos Papangelou, Konstantinos
Sechidis, James Weatherall, and Gavin Brown. Toward an understanding of adversarial examples in clinical trials. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 35-51. Springer, 2018.
Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium on Operating Systems Principles, pp. 1-18, 2017.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. arXiv preprint arXiv:1801.09330 2018.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gas gan:: Protecting classifiers against adversarial attacks using generative models. arXiv preprint 188.v:8805.06605, 2018.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, pp. 8312-8323, 2018.
11

---

Under review as a conference paper at ICLR 2021
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. IEEE International Joint Conference on Neural Networks, pp. 1453-1460, 2011.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprini arXiv:1312.6199 2013.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint arXiv:2007.00644 2020.
Dogancan Temel, Min-Hung Chen, and Ghassan AlRegib. Traffic sign detection under challeng- ing conditions: A deeper look into performance variations and spectral characteristics. IEEE Transactions Intelligent Transportation Systems, 2019.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Simon Vandenhende, Bert De Brabandere, Davy Neven, and Luc Van Gool. A three-player gan: gen- erating hard samples to improve classification networks. In 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1-6. IEEE, 2019.
Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversarial net- works. arXiv preprint arXiv: 999.0099911 2019.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312: 135-153, 2018.
Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. arXiv preprint arXiv: 1909, 1/266, 2019.
Eric Wong and J Zico Kolter. Provable Defenses Against Adversarial Examples Via the Convex Outer Adversarial Polytope. arXiv preprint arXiv: 2017.
Eric Wong and J Zico Kolter. Learning perturbation sets for robust machine learning. arXiv preprint arXiv:2007.08450, 2020.
Chaowei Xiao, Bo Li, Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adver- sarial examples with adversarial networks. arXiv preprint arXiv: 1801, 1801,02610. 2018a.
Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially trans- formed adversarial examples. arXiv preprint arXiv:1801.0266 2018b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. In Advances in Neural Information Processing Systems, pp. 13443-13452, 2019.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In Proceedings of the IEFE international conference on computer vision, pp. 2849-2857, 2017.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.570. 2012.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-imann translation. In Advances in neural information processing systems, pp. 465-476, 2017b.
12


Reviews:
[{"review_id": "uN7_9dG4MVD", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper looks at the idea of building models of natural variation of an input and then using these models to develop robust training algorithms that are less susceptible to outliers ( testable for worst-cases via adversarial attacks ). \n\nStrength:\n\n+ The paper addresses a very important topic of adversarial robustness of DNN models and is accompanied by diligent evaluation over different datasets. \n\nWeakness:\n\n- The paper adopts an approach which is very reminiscent of Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1712.09196, 2017. The key difference seems to be adoption of the idea of using auxiliary transformations (called natural perturbations) which is also very well-studied in literature, for e.g. see https://openaccess.thecvf.com/content_CVPR_2020/papers/Zhang_Auxiliary_Training_Towards_Accurate_and_Robust_Models_CVPR_2020_paper.pdf  So, the approach presented here is quite incremental for a premier venue such as ICLR.\n\n- The attack in Jalal et. al. paper can be launched against this approach once the attacker also access to this generative/natural model (which would be easy to build for an attacker)\n\n- The reviewer will strongly recommend reviewing the advices in https://arxiv.org/abs/1902.06705 for writing papers on defense and how to self-evaluate its robustness by suitably designing the attacker. If the defense approach uses some background/auxiliary knowledge, one must consider the attacker with this knowledge if it is accessible to the attacker. \n\nQuestions to authors:\n\n- Can authors explain why an attacker can't build similar natural model to defeat the proposed defense? \n- Any clarification on incremental novelty from  Jalal et. al. and Zhang et. al. (and references therein) would be also useful. \n\n\nAfter author's rebuttal:\n\n\"The other paper that the review points to ([4]) addresses a similar setting to our paper, but the approach is completely different. ... Moreover, [4] was published within three months of the ICLR submission deadline, meaning that it is essentially concurrent with our work.\"\n\nYes, the reviewer concurs that a work published so close to deadline should be treated as concurrent and will raise the score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "oezlT96FUV8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extends current adversarial learning approaches beyond imperceptible L_p norm perturbations. The proposed approach can handle many models of natural variation, such as a change in brightness. The main idea behind the approach is to use unsupervised approaches such as GANs to model the natural variation. Given this model of natural variation, the paper replaces the adversarial learning objective of finding the worst example in an L_p norm ball around a data point to finding the worst example based on the model of natural variation. This is expensive, so the paper also proposes more computationally efficient approaches based on data augmentation. The experimental results demonstrate that the proposed approach performs well on a variety of tasks.\n\nStrengths:\n\n1. The question of training robust neural networks is clearly timely and well-motivated. A valid raised criticism of the current adversarial examples literature is that it is too tied to imperceptible or L_p norm perturbations, and this paper tries to go beyond them.\n2. There is a bit of a lack of relevant methods to compare against, but the experiments still show that the proposed approach does well. I also appreciated the comparisons to domain adaption, and the multiple experimental setups generally.\n\nWeaknesses:\n\n1. Though the paper suggests that the approach is a “paradigm shift” in robust deep learning, it appears to be more or less an extension of the current adversarial examples literature. The adversarial training approach has been quite successful in defending against a fixed adversary model, from various L_p norms to geometric transformations. This paper is essentially defining a new adversary model, given by natural variations. In fact it is simpler than this, since the goal is to only do well on \"random variations\" in the predefined test set, i.e. there is no adversary which will try and attack the network by using the class of variations. There is a novelty here though that the variation model is learnt rather than pre-specified. But given that learning transformations such as change in brightness etc. is quite straightforward for unsupervised models such as GANs, this is not too much to ask for. Therefore, I find it a bit unsurprising that the proposed method works well, given the success of adversarial training and GANs.\n2. Related to the above, in my view one main drawback of the current adversarial learning literature is that it is not difficult to get robustness to one pre-specified model of perturbation/variation, but in reality the space of possible variations is quite large. It is good that the proposed approach does seem to work on two simultaneous shifts, but the approach still seems restricted and not a substantial change in the current understanding. Note that some of the compared approaches are different in this regard: AugMix does not tailor the model to a particular variation (it explicitly excludes operations which overlap with ImageNet-C corruptions) and domain adaptation can handle quite large changes in the data distribution (which are not just transformations of the original data points). \n3. Perhaps less importantly, in the experiments the proposed MDA approach based on data augmentation does almost as well as the other ones, and data augmentation using unsupervised models is already studied in the literature.\n\nOverall, this is not a bad paper and I am not completely opposed to acceptance, but I am not sure I can argue for it.\n\nOther less important comments:\n\n1. Since there is a lack of baselines to compare against, the paper could benefit from a few ablation studies. For example, is there any difference on using other unsupervised approaches? \n2. I think it would be better to have a concise list of a few major contributions in Section 1, rather than the current rather long list.\n3. It would be nice to have representative images of the models of variation, perhaps in an appendix.  \n\n------Updates after author response------\n\nI thank the authors for the response and it helps clarify some points. However, I am still not unconvinced that the paper is a significant departure from current work on adversarial robustness (see weakness 1 and 2 above). I think it would be much more interesting if the approach could yield robustness against a wider/different class of shifts compared to what it was adversarially trained against. Therefore, I am keeping my score at 5 and will not advocate for acceptance, though I am not completely opposed to it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ER1aRZ474Jg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper proposes a “paradigm shift” for augmenting datasets when training CNN-based image classifiers. On one side, traditional augmentations include blur, Gaussian noise, color distortions. On the other side, methods like adversarial training consider augmentations under norm bounds in the image space. The proposed method, instead, uses models of natural variation to augment the images. Increased out-of-domain accuracy is shown on ImageNet-C and several slices of the CURE-TSR dataset. \n\nCentral question: How to shift from perturbation-based robustness analysis to model-based robustness analysis?\n\n# Strong and Weak points\n\n## Strong points \n\n  * This paper introduces a “paradigm shift” from perturbation based to model-based augmentation, when training CNN based image classifiers. The existing perturbations only reach so far, and using models of natural variation provides a new avenue for augmenting data sets. \n  * The method is presented in clear pseudo code, and detailed in the 36 page counting appendix. \n\n## Weak points\n\n  * The paper claims a “paradigm shift”, but the proposed method of augmenting data has been used since 2012,  [1], and is used ever since [2]. Tables 2 and 3 provide key results for this method, but both were obtained with Model-based Data Augmentation, which is the common method before the \"paradigm shift\".\n  * Results outlined in Section 5.2 and Table 2 concerns scenarios where the test-time perturbation is known at training time. Table 2 then compares MDA to ERM using this a-priori knowledge, and shows improved accuracy. However, the comparison is unbalanced, as ERM does not benefit from the prior knowledge. If Table 2 would be about comparing methods under the assumption that the test-time perturbation would be known at training time, then the ERM should be trained with the same data augmentation.\n  * The paper misses comparisons against previous published work. Although Table 2 and 4 compare against another method, these results were not obtained by the respective authors, thus introducing a bias. The related work, Section 6, cites as many as 9 papers that addressed the same problem as this paper. The argument for the proposed method could be stronger when comparing to results from any of these papers. \nMoreover, Table 3 reports results on ImageNet-C, which multiple other research papers have published results on. See for example, [3] and [4]. Likewise, Table 4 reports results on MNIST-C, which have been reported in [5] and [6]. Omitting those comparisons hampers the argument for the proposed method. \n\n[1] Krizhevsky, 2012\n[2] Szegedy, 2015\n[3] https://paperswithcode.com/sota/domain-generalization-on-imagenet-c\n[4] https://arxiv.org/pdf/1903.12261.pdf \n[5] https://proceedings.icml.cc/paper/2020/file/20546457187cf3d52ea86538403e47cc-Paper.pdf \n[6] https://arxiv.org/pdf/1906.02337.pdf \n\n# Statement\n\nRecommendation: Reject\nReason:\nAlthough this paper provides four tables of extensive evaluation of the method, none of the numbers were obtained from previous literature. The related work section cites 9 papers on the same problem, but none were compared with in the results section. (See concrete details above)\n\n# Clarifying questions\n\n  * How to quantify the increase in computational complexity for the inner maximization in Equation (2)? As I understand, even evaluating one point requires a forward pass of the variation-model and the classifier-model. In case of iterative optimization, this would also require two backward passes(see line 5 in Algorithm (1)). \nIf the proposed method requires significantly more compute, the comparison should be made with a method that has access to the same amount of compute (for example a larger model, or longer trained model).\n  * Line 6 of Algorithm (1). I miss the definition of set $\\Delta$ and the projection operator in the text nor the appendix. Please explain?\n  * Section 5.2 speaks of ImageNet “classes”. I am confused if these “classes” are referring to the ImageNet classes (1000 in total) or the severity levels in ImageNet-C (75 in total). Could you please clarify?\n  Moreover, this argument implicitly assumes that the test-time perturbation is known at training-time. Could you name a scenario where this assumption is justified?\n  * To what extent is the accuracy on the clean data preserved by the proposed method? Table 3, for example, shows an increase in top-1 accuracy on ImageNet data under perturbation. However, I wonder if this model preserves accuracy on the clean data. We know from existing literature [1] that a trade off exists for adversarial training between accuracy on perturbed and clean data. Providing the clean accuracy would provide evidence that this method compares favourably on this trade off. \n\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" arXiv preprint arXiv:1901.08573 (2019).\n\n# Minor feedback\n\nThis minor feedback is not part of the assessment.\n\n  * “arguably more common” -> “arguably more important”\n  * “x with a corresponding label y” -> “x with a corresponding prediction y”\n  * Abbreviation “MBRDL” appears in Section 5 without any explanation or definition. \n  * Abbreviation “CURE-TSR” appears in Section 5.0 without any explanation or definition.\n  * The citation for CURE in Section 5.1 is wrong. The bibliography refers to “Temel, Dogancan, Min-Hung Chen, and Ghassan AlRegib. \"Traffic sign detection under challenging conditions: A deeper look into performance variations and spectral characteristics.\" IEEE Transactions on Intelligent Transportation Systems (2019).”, but the CURE TSR dataset was published in “Temel, Dogancan, et al. \"CURE-TSR: Challenging unreal and real environments for traffic sign recognition.\" arXiv preprint arXiv:1712.02463 (2017).”\n  * Caption of Table 4: “ Then, we use this model to perform model-based training on a new dataset D1” -> “ Then, we use this model to perform model-based testing on a new dataset D2”? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4kddaVVuOQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model-based framework for improving the robustness of image classifiers to average-case corruptions of varying severity. The proposed framework can be thought of as adversarial training where the perturbation is replaced by a function that transforms the image according to a specific corruption. A nuisance parameter controls the instantiation and severity of the corruption that is applied to the input. The paper compares baselines to different versions of this general model-based framework with experiments on several datasets.\n\nPros:\n- Focusing on average-case corruptions is an important and underexplored problem\n- Using image translation to learn the corruptions is an interesting proposal\n\nCons:\n- There isn't much novelty in the proposed technique. Adversarial training with 'natural' corruptions has been done several times before.\n- There isn't any discussion of how the nuisance parameter is used. I suspect it isn't used for ImageNet-C experiments, though I may be wrong. In this case, the method becomes data augmentation with image translation, which is not very novel either.\n- It sounds like the ImageNet-C experiments use the corruptions to train the image translation network. Even though these experiments use held-out images, the corruption types themselves are not being held out, which goes against the recommended methodology for that dataset.\n\n\nTypo: \"ImageNet-c\" should be \"ImageNet-C\"\n\n___________________________________________________________________________________\n\nUpdate after author feedback:\n\nI thank the authors for improving my understanding of the paper. I feel better about it after reading it again. One of the most interesting findings--that a translation network trained on one domain/set of classes generalizes to another--needs more discussion. Using this phenomenon to improve robustness is a good idea, and the MRT/MDA/MAT methods explored in this work are nice choices for this investigation. However, I agree with the other reviewers that it would be nice to include more baselines from other work where appropriate.\n\nThe results in Table 1 are also very interesting and deserve more discussion--perhaps an analysis of whether a translation network trained on weak corruptions can generalize to create something akin to the ground-truth stronger corruptions, as the results in the table for MRT imply.\n\nOverall, I think this paper has some strong experiments and investigates a good idea, but the claims of a \"paradigm shift\" are overly grandiose, and some of the most interesting experiments could use more analysis. Additionally, the writing clarity and presentation could be improved. My initial understanding of the paper was flawed in some places, so I'll raise my score to 5.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Quality:
The technical contribution of the paper seems reasonable and there were only minor points being highlighted by the reviewers.

# Clarity:
The paper would benefit from being more polished. During the rebuttal, the authors suggested that several reviewers misunderstood the paper. This alone should encourage the authors to improve clarity.

# Originality:
Several reviewers presented concerns about the claims of the authors and the existence of connections to existing literature. Nonetheless, the proposed approach seems novel to the best of the reviewers and my knowledge.

# Significance of this work: 
The topic of the manuscript is relevant and impactful. However, several reviewers suggested to include additional baselines in the experiments to validate the goodness of the proposed approach.

# Overall:
The paper presents an interesting idea, with a high potential impact. Despite the interesting topic and some interesting insights, all the reviewers agree that the manuscript is not ready for publication just yet. I want to encourage the authors to keep improve it and resubmit it at the next conference.
Decision: Reject