A Fine-Grained Analysis on Distribution Shift
Authors: Olivia Wiles
Sven Gowal
Florian Stimberg
Sylvestre-Alvise Rebuffi
Ira Ktena
Krishnamurthy Dj Dvijotham
Ali Taylan Cemgil
Venue: ICLR 2022 Oral (2022)
Keywords: robustness
distribution shifts

Abstract: Robustness to distribution shifts is critical for deploying machine learning models in the real world. Despite this necessity, there has been little work in defining the underlying mechanisms that cause these shifts and evaluating the robustness of algorithms across multiple, different distribution shifts. To this end, we introduce a framework that enables fine-grained analysis of various distribution shifts. We provide a holistic analysis of current state-of-the-art methods by evaluating 19 distinct methods grouped into five categories across both synthetic and real-world datasets.  Overall, we train more than 85K models. Our experimental framework can be easily extended to include new methods, shifts, and datasets. We find, unlike previous work (Gulrajani & Lopez-Paz, 2021), that progress has been made over a standard ERM baseline; in particular, pretraining and augmentations (learned or heuristic) offer large gains in many cases. However, the best methods are not consistent over different datasets and shifts. We will open source our experimental framework, allowing future work to evaluate new methods over multiple shifts to obtain a more complete picture of a method's effectiveness. 
Code is available at github.com/deepmind/distribution_shift_framework.


Published as a conference paper at ICLR 2022
A FINE-GRAINED ANALYSIS ON DISTRIBUTION SHIFT
Olivia Wiles Sven Gowal Florian Stimberg Sylvestre-Alvis Rebuffi
Ira Ktena Krishnamurthy (Dj) Dvijotham DeepMind, London, UK
Taylan Cemgil
fooviler, gowa.., stimberg, wttp://........re....seettt.eee. sylvestre, irakecna, tay Edeepnins.com dvi goo..googe
ABSTRACT
Robustness to distribution shifts is critical for deploying machine learning mod- els in the real world. Despite this necessity, there has been little work in defin- ing the underlying mechanisms that cause these shifts and evaluating the robust- ness of algorithms across multiple, different distribution shifts. To this end, we introduce a framework that enables fine-grained analysis of various distribution shifts. We provide a holistic analysis of current state-of-the-ae methods by eval- uating 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Overall, we train more than 85K models. Our experimental framework can be easily extended to include new methods, shifts, and datasets. We find, unlike previous work (Gulrajani & Lopez-Paz, 2021), that progress has been made over a standard ERM baseline; in particular, pretraining and augmen- tations (learned or heuristic) offer large gains in many cases. However, the best methods are not consistent over different datasets and shifts. Code is available at gi github. com/ http//eeepmmnn//e /idttiidd/ddteeeeeeee ribut oonsonnsfeeiie framework.
INTRODUCTION
If machine learning models are to be ubiquitous in critical applications such as driverless cars (Janai al., 2020), medical imaging (Erickson et al., 2017), and science (Jumper et al., 2021), it is pivotal to build models that are robust to distribution shifts. Otherwise, models may fail surprisingly in ways that derail trust in the system. For example, Koh et al. (2020); Perone et al. (2019); AlBadawy et al. (2018); Heaven (2020); Castro et al. (2020) find that model trained one set of hospitals may not generalise to the imaging conditions of another; Alcorn et al. (2019); Dai & Van Gool (2018) find that a model for driverless cars may not generalise new lighting conditions object poses; and Buolamwini & Gebru (2018) find that a model may perform worse on subsets of the distribution, such different ethnicities, if the training set has an imbalanced distribution. Thus, it is important to understand when we expect a model to generalise and when we do not. This would allow a practitioner have confidence in the system (e.g. if model demonstrated be robust to the imaging conditions of different hospitals, then it can be deployed in new hospitals with confidence).
While domain generalization is well studied area, Gulrajani Lopez-Paz (2021); Schott et al. (2021) have cast doubt on the efficacy of existing methods, raising the question: has any progress been made in domain generalization over standard expectation risk minimization (ERM) algo- rithm? Despite these discouraging results, there are many examples that machine learning models do generalise across datasets with different distributions. For example, CLIP (Radford et al., 2021), with well engineered prompts, generalizes to many standard image datasets. Taori et al. (2020) found that models trained on one image dataset generalise to another, albeit with some drop in performance; in particular, higher performing models generalise better. However, there is little un- derstanding and experimentation on when and why models generalise, especially in realistic settings inspired by real-world applications. This begs the following question:
Can we define the important distribution shifts to be robust to and then systematically evaluate the robustness of different methods?
To answer the above question, present grounded understanding of robustness to distribution shifts. We draw inspiration from disentanglement literature (see section 6), which aims to separate images into an independent set of factors of variation (or attributes). In brief, we assume the data

---

Published as a conference paper at ICLR 2022
is composed of some (possibly extremely large) set of attributes. We expect models, having some distribution of values for an attribute, to be able to learn invariance to that attribute and so to generalise to unscen examples of the attribute and different distributions over that attribute. Using a simple example to clarify the setup, assume our data has two attributes (shape and color) among others. Given data with some distribution over the set of possible colors (e.g. red and blue) and the task of predicting shape (c.g. circle or square), we want our model to generalise to unseen colors (e.g. green) or a different distribution of colors (e.g. there are very few red circles in the training set, but the samples at evaluation are uniformly sampled from the set of possible colors and shapes).
Using this framework, we evaluate models across three distribution shifts: spurious correlation, low- data drift, and unseen data shift (illustrated in figure 1) and two additional conditions (label noise and dataset size). We choose these settings as they arise in the real world and harm generalization performance. Moreover, in our framework, these distribution shifts are the fundamental blocks of building more complex distribution shifts. We additionally evaluate models when there varying amounts of label noise (as inspired by noise arising from human raters) and when the total size of the train set varies (to understand how models perform as the number of training examples changes). The unique ability of our framework to evaluate fine- tne-ggained performance of models across different distribution shifts and under different conditions is of critical importance to analyze methods under variety of real-world settings. This work makes the following contributions:
We propose framework to define when and why we expect methods to generalise. We use this framework to define three, real world inspired distribution shifts. We then use this framework to create a systematic evaluation setup across real and synthetic datasets for dif- ferent distribution shifts. Our evaluation framework is casily extendable to new distribution shifts, datasets, or methods to be evaluated.
We evaluate and compare 19 different methods (training more than 85K models) in these settings. These methods span the following common approaches: architecture choice, data augmentation, domain generalization, adaptive algorithms, and representation learn- ing. This allows for a direct comparison across different areas in machine learning.
We find that simple techniques, such data augmentation and pretraining are often effec- tive and that domain generalization algorithms do work for certain datasets and distribution shifts. However, there is no easy way to select the best approach a- a-priori and results are inconsistent over different datasets and attributes, demonstrating there is still much work to be done to improve robustness in worldwrrld settings.
FRAMEWORK TO EVALUATE GENERALIZATION
In this section we introduce our robustness framework for characterizing distribution shifts in principled manner. We then relate three common, real world inspired distribution shifts.
2.1 LATENT FACTORISATION
We assume a joint distribution p of inputs x and corresponding attributes y²,y²,... ........... (denoted as I with y* E A where A is finite set. One of these K attributes is a label of interest, denoted as y' (in a mammogram, the label could be cancer/benign and a nuisance attribute y with i * could be the identity of the hospital where the mammogram classifier f that minimizes the risk R. However, in real-world applications, we only have access to a finite set of inputs and attributes of size n. Hence, we minimize the empirical risk ft instead:
.y
taken). Our aim is build
R(f) (x [Cly', f(x))]
R(fip)
Σ (())::))<pp
Cly., f(x:)).
where a suitable loss function. Here, all nuisance attributes y* with k of / aare ignored and we work with samples obtained from the marginal p(y'. .z.. In practice, however, due to selection bias or other confounding factors in data collection, we are only able to train and test our models on data collected from two related but distinct distributions: Pirain Peest For example, Ptrain and Ptest may be concentrated on different subsets of hospitals and this discrepancy may result in a distribution shift; for example, hospitals may use different equipment, leading to different staining their cell
2

---

Published as a conference paper at ICLR 2022
(a) Puriin SC.
(b) Prain: LDD.
(c) Purinn UDS.
(d) Prest s y IID.
Figure I: Visualization of the joint distribution for the different shifts we consider on the DSPRITES example. The lighter the color, the more likely the given sample. figure la-le visualise different shifts over Pusin , 3²) discussed in 2.2: spurious correlation (SC), low-data drift (LDD), and un- seen data shift (UDS). figure 1d visualises the test set, where the attributes are uniformly distributed.
images. While we train f on data from Ptrain by minimizing R((f: Ptrain), ), we aim to learn a model that generalises well to data from Prest; that is, it should achieve small P(:s
While generalization in the above sense is desirable for machine learning models, not clear why model f trained on data from Ptrain should generalise to Ptest- It is worth noting that while Ptrain and Ptest can be different, they are both related to the true distribution p. We take inspiration from disentanglement literature to express this relationship. In particular, that can view data as being decomposed into an underlying set of factors of variations. We formalise various distribution shifts using a latent variable model for the true data generation process:
2 ~ p(2)
y¹ ~ p(y'12) i l.....
x a~ <>p())))
where z denotes latent factors. By simple refactorization, we can write
p x) p(y
= kkk p(xx))
Thus, the true distribution can be expressed as the product the marginal distribution of the at- tributes with conditional generative model. We assume that distribution shifts arise when new marginal distribution for the attributes chosen, such as kKy * Perain (yl:K) * Prest K ), but otherwise the conditional generative model is shared across all distributions, i.e., we have
Prest 2y x) = Prest (y f p(æ) 2)p(2l))) (2 )dz, and similarly for Ptrain-
To provide more context, as running example,
the color DSPRITES dataset (Matthey al., 2017); where in our notation y² defines the color with A = (red, green, blue), and y² defines the shape with A {ellipse, heart, square]. We can imagine that data collector (intentionally or im- plicitly) selects some marginal distribution over attributes Ptrain (y when training; for example they select mostly blue ellipses and red hearts, This induces a new joint distribution over latent fac- tors and attributes: Ptrain (2, f p(z|y1-K PPrriin (yl:K). Consequently, during training, we get images with a different joint distribution Perain (x, y 1::k applies when collecting data for the test distribution. We focus on common cases of distribution shifts visualized figure I; we discuss these in more detail in section 2.2.
p(æ (((rai) (z, y=: ). This similarly
The goal of enforcing robustness to distribution shifts is to maintain performance when the data generating distribution Pusin changes. In other words, we would like to minimize risk on Ptest given only access to Ptrain. We can achieve robustness in the following ways:
Weighted resampling. We can resample the training set using importance weights (y (()== (y (y3:K), This means that given the attributes, the i-th data point (v) 1:K ,x;) in the training set is used with probability W Wy )/ E/ W (v!::) rather than 1/n. We refer this empirical distribution Preweight This procedure requires access the true distribution of attributes p(y1:K), so to avoid bias and improve fairness, it is often assumed that all combinations of attributes happen uniformly at random.
Data Augmentation: Alternatively, we can learn a generative model p() from the training data that aims approximate J ((=))) )dz, as the true conditional
3

---

Published as a conference paper at ICLR 2022
DSPRITES.
MPI3D.
SHAPES3D.
SMALLNORB.
CAMELYON
IWILDCAM.
Figure 2: Dataset samples. Each row fixes attribute (e.g. color for DSPRITES, MPI3D, SHAAPSS3:; azimuth SMALLNORB; hospital for CAMELYON17; and location for IWILDCAM).
generator by our assumption the same over all (e.g. train and test) distributions. such a conditional generative model can be learned, we can sample new synthetic data at training time (e.g. according to the true distribution "K)) to correct for the dis- tribution shift. More precisely, we can generate data from the augmented distribution Paug = (1 ar) Preweeig + arp(æ) yy 1kk) and train a supervised classifier on this aug- mented dataset. Here, a E [0, 1] is the percentage of synthetic data used for training.
Representation Learning: An alternative factorization of data generating distribution (e.g. train) is Pusin (y 1:K x p ( 1: 22)22. We can learn an unsupervised representation that approximates p(2) æ)) using the training data only, and attach classi- fier to learn a task specific head that approximates Purin momn (y'(z). Again, by our assumption p(z[æ=) p(z]2=)((2). Given good guess of the prior, the learned representation would not be impacted by the specific attribute distribution and so generalise to Ptest:
2.2 DISTRIBUTION SHIFTS
While distribution shifts can happen in a continuum, we consider three types of shifts, inspired by real-world challenges. We discuss these shifts and two additional, real-world inspired conditions.
Test distribution Prest. We assume that the attributes distributed uniformly: Prest (y I/II, AA). This is desirable, as all attributes are represented and a-priori independent.
Shift Spurious correlation A Attributes are correlated under Ptrsin but not Prest Spurious correlation arises in the wild for number of reasons including capture bias, environmental factors, and geographical bias (Beery al., 2018: Torralba Efros, 2011). These spurious correlations lead to surprising results and poor generalization. Therefore, is important to be able to build our framework. spurious correlation arises when two attributes y², y are correlated at training time, but this is not true of Prest, for which attributes are independent: Perain(y" ".."..""
models that are robust to such challenges.
ly" ...g""!.! " - ... y)) This is especially problematic when one attribute y is y the label. Using the running DSPRITES example, shape and
..yy" "....""""""".......... .y²)
color may be correlated and the model may find it easier to predict color. If color is the label, the
model will generalise well. However, if the aim to predict shape, the model's reliance color will lead to poor generalization.
Shift 2: Low-data drift A Attribute values are unevenly distributed under Piraiu but not under Prest- Low-data drift arises in the wild (e.g. in (Buolamwini & Gebru, 2018) for different ethnici- ties) when data has not been collected uniformly across different attributes. When deploying models in the wild, it is important be able reason and have confidence that the final predictions will
4

---

Published as a conference paper at ICLR 2022
0.3
0.2
06
0.1
327
1333
1000
33300
10000
Th TLAKen
OOMA
(200
Architeeture
Heunntie
Domwan Gen
Addur
Machshechune
Hourtene
Demain
Figure Spurious Correlation. We use all correlated samples and vary the number of sam- ples N from the true, uncorrelated distribution. We plot the percentage change over the baseline ResNet, averaged over all seeds and datasets. Blue is better, red worse. CYCLEGAN performs consistently best while ImageNet augmentation and pretraining on ImageNet also consistently boosts performance.
Figure 4: ples from the high data regions and vary the number of samples N from the low-data region. We plot the percentage change over the baseline ResNet, averaged over all seeds and datasets. Blue is better, red worse. Pretraining geNet performs consistently best, while CYCLE- GAN, most domain generalization methods and ImageNet augmentation also provide some boost in performance.
Low-data drift. We use all sam-
Ima-
be consistent and fair across different attributes. In the framework above, low-data shifts arise when certain values in the set A of an attribute y are sampled with much smaller probability than in Prest: Ptrain (1/* v) < Ptest (1f" = v). Using the DSPRITES example, only a handful of red shapes may be seen at training time, yet in Prest all colors are sampled with equal probability.
Shift 3: Unseen data shift - Some attribute values are unseen under Ptrain but are under Ptest This is special case of shift low-data drift which we make explicit due its important real world applications. Unseen data shift arises when a model, trained in one setting is expected to work in another, disjoint setting. For example: a model trained to classify animals on images at certain times of day should generalise to other times of day. In our framework, unseen data shift arises when some values in the set A" an attribute y are unseen in Perain but are in Ptest:
Ptrain (y" = v) =
Prest (3² = v) > 0
Hulprraii (v/" = = >
(2)
This is a stronger constraint than in standard cuttof-ddisribbution generalization (see section 6). as multiple values for A must be seen under Persin- which allows the model learn invariance y²- In the DSPRITES example, the color red may be unseen at train time but all colors are in Preat-
Discussion. We choose these sets of shifts as they are the building blocks of more complex dis- tribution shifts. Consider the simplest case of two attributes: the label and nuisance attribute. If we consider the marginal distribution of the label, it decomposes into two terms: the conditional probability and the probability of a given attribute value: p(y') = =ye p(y )p()") The three shifts consider control these terms independently: unseen data shift and low-data drift control p(v") whereas spurious correlation controls p(3'ly"" The composition of these terms describes any distribution shift for these two variables.
2.3 CONDITIONS
Label noise. We investigate the change in performance due to noisy information. This can arise when there disagreements and errors among the labellers (e.g. in medical imaging (Castro al., 2020)). We model this as an observed attribute (e.g. the label) being corrupted by noise. j ~ c(y²), ) where y² E A is the true label, j² E A' the corrupted, observed one, and the corrupting function.
Dataset size. We investigate how performance changes with the size of the training dataset. This setting arises when is unrealistic expensive to collect additional data (e.g. in medical imaging or in camera trap imagery). Therefore, it is important to understand how performance degrades given fewer total samples. We do this by limiting the total number of samples from Ptrain-
5

---

Published as a conference paper at ICLR 2022
3 MODELS EVALUATED
We evaluate 19 algorithms to cover broad range of approaches that can be used to improve model robustness to distribution shifts and demonstrate how they relate to the three ways to achieve robust- ness, outlined in section 2. We believe this is the first paper to comprehensively evaluate a large set of different approaches in a variety of settings. These algorithms cover the following areas: architec- ture choice, data augmentation, domain adaptation, adaptive approaches and representation learning. Further discussion on how these models relate to our robustness framework is in appendix E.
Architecture choice. We evaluate the following standard vision models: ResNet18, ResNet50, ResNet101 (He et al., 2016), VIT (Dosovitskiy et al., 2021), and an MLP (Vapnik, 1992). We use weighted resampling Preweight to oversample from the parts of the distribution that have lower probability of being sampled from under Ptrain- Performance depends on how robust the learned representation to distribution shift.
Heuristic data augmentation. These approaches attempt to approximate the true underlying gen- erative model p(zly!. in order to improve robustness. We analyze the following augmentation methods: standard ImageNet augmentation (He et al., 2016), AugMix without JSD (Hendrycks et al., 2020), RandAugment (Cubuk et al., 2020), and AutoAugment (Cubuk et al., 2019). Perfor- mance depends on how well the heuristic augmentations approximate the true generative model.
Learned data augmentation. These approaches approximate the true underlying generative model p(zly!: by learning augmentations conditioned on the nuisance attribute. The learned augmenta- tions can be used to transform any image æ to have a new attribute, while keeping the other attributes fixed. We follow Goel et al. (2020), who use CYCLEGAN (Zhu et al., 2017), but we do not use their SGDRO objective in order to evaluate the performance of learned data augmentation alone. Perfor- mance depends how well the learned augmentations approximate the true generative model.
Domain generalization. These approaches aim to recover a representation 2 that is independent of the attribute: p(y"; =) p(y") ))(()) =) to allow generalization over that attribute. We evaluate IRM (Arjovsky et al., 2019), DeepCORAL. (Sun & Saenko, 2016), domain MixUp (Gulrajani & Lopez- Paz. 2021), DANN (Ganin al., 2016), and SagNet (Nam al., 2021). Performance depends on the invariance of the learned representation z.
Adaptive approaches. These works modify p Preweight dynamically, We evaluate JTT (Liu et al., 2021) and BN-Adapt (Schneider et al., 2020). These methods do not give performance guarantees.
Representation learning. These works aim learn robust representation of that describes the true prior. We evaluate using a 3-VAE (Higgins et al., 2017a) and pretraining on ImageNet (Deng et al., 2009). Performance depends on the quality of the learned representation for the specific task.
4 EXPERIMENTS
We first introduce the datasets and experimental setup. We evaluate the 19 different methods across these six datasets, three distribution shifts, varying label noise, and dataset size. We plot aggregate results in figures 3-7 and complete results in the appendix in figures 10-12. We discuss the results by distilling them into seven concrete takeaways in section 4.1 and four practical tips in section 4.2.
Datasets. We evaluate these approaches on six vision, classification datasets -DSPRITES (Matthey et al., 2017), MPI3D (Gondal et al., 2019), SMALLNORR (LeCun et al., 2004), SHAPES3D (Burgess & Kim, 2018), CAMELYON17 1 (Koh et al., 2020; Bandi et al., 2018), and IWILDCAM (Koh al., 2020; Beery et al., 2018). These datasets consist of multiple (potentially an arbitrarily large number) attributes. We select two attributes y² yd for each dataset and make one y' the label. We then use these two attributes to build the three shifts. Visualizations of samples from the datasets are given in figure 2 and further description in appendix D.I. We discuss precisely how we set up the shifts, choose the attributes, and additional conditions for these datasets in appendix D.2.
Model selection. When investigating heuristic data augmentation, domain generalization, learned augmentation, adaptive approaches, and representation learning, we use a ResNet18 for the simpler, synthetic datasets (DSPRITES, MP13D, SHAPES3D, and NORB) but ResNet50 for the

---

Published as a conference paper at ICLR 2022
Figure 5: Unseen data shift. We rank the methods (where best is 1, worst 19) for each dataset and seed and plot the rankings, with the overall median rank as the black bar. Pretrain- ing ImageNet and ImageNet augmentation perform consistently best. DANN, CycleGAN and other heuristic augmentations perform consis- tently well.
Nanking
B 3
Resterts
NosWet50
Restiers
proseeie
Ran Randdhuggn
gois
CORAL
Maop
Sameet
OyclesAAN
Pratralneo
subei
more complex, real world ones (CAMELYON17 and IWILDCAM). To perform model selection, we choose the best model according to the validation set which matches the distribution of the test set. In the unseen data shift setting for the CAMELYON17 7 and IWILDCAM, we use the given out-of- distribution validation set, which is different, distinct set in 'D that is independent of Dtrainn Drest- (We consider using the in-distribution validation set in appendix B.4.)
Hyper peeepaaaaeee choices. We perform a sweep over the hyperparameters (the precise sweeps are given in appendix F.8). We run each set of hyperparameters for five seeds for each setting. To choose the best model for each seed, we perform model selection over all hyperparameters using the top-1 accuracy on the validation set. In the low-data and spurious correlation settings, we choose a different set of samples from the low-data region with each seed. We report the mean and standard deviation over the five seeds.
4.1 TAKEAWAYS
Takeaway 1: While we can improve over ERM, no one method always performs best. The relative performance between methods varies across datasets and shifts. Under spurious correlation (figure 3), CYCLEGAN consistently performs best but in figure 4, under low-data drift, pretraining consistently performs best. Under unseen data shift (figure 5), pretraining is again one of the best performing models. However, if we drill down on the results in figure 10 (appendix B.1), we can see pretraining performs best on the synthetic datasets, but not on CAMELYON17 (where using augmentation or DANN is best) or !WILDCAM (where using VIT or an MLP is best).
Pretraining is a powerful tool across different shifts and datasets. While pre- training is not always helpful (e.g. in appendix B.1 on CAMELYON17 in figures 10-11, IWILDCAM in figures (0-11), it often provides a strong boost in performance. This presumably because the representation learned during pretraining is helpful for the downstream task. For example, the representation may have been trained to be invariant to certain useful properties (c.g. scale, shift, and color). If these properties useful on the downstream tasks, then the learned representation should improve generalization.
Takeaway 2:
Takeaway 3: Heuristic augmentation improves generalization if the augmentation describes an attribute. In all settings (figures 3-5), ImageNet augmentation generally improves performance. However, RandAugment, AugMix, and AutoAugment have more variable performance (as further shown in figures 10-12). These methods are compositions of different augmentations. We investigate the impact of each augmentation in Rand in appendix B.2 and find variable performance. Augmentations that approximate the true underlying generative model p( y 2: ) lead to the best results; otherwise, the model may waste capacity. For example, on CAMELYON17 (which consists of cell images), color jitter harms performance but on SHAPES3D and MPI3D it is essential.
Takeaway 4: Learned data augmentation is effective across different conditions and distribu- tion shifts. This approach is highly effective in the spurious correlation setting (figure 3). It can also help in the low-data and unseen data shift settings (figure 4,5) (though the gains for these two shifts are not as large as for pretraining). The effectiveness of this approach can be explained by the fact that if the augmentations are learned perfectly, then augmented samples by design are from the true underlying generative model and can cover missing parts of the distribution.
Takeaway 5: Domain generalization algorithms offer limited performance improvement. In some cases these methods (in particular DANN) do improve performance, most notably in the low- data drift and unseen data shift settings (figures 4-5). However, this depends on the dataset (see figures 10-12) and performance is rarely much better than using heuristic augmentation.
7

---

Published as a conference paper at ICLR 2022
-02
60
COROO
-04
COTAAL
DAN
Architecure
(Teunsac Aug
Domarn Den
Adagt (A)
Rep.
Architacture
slc Aug
Domain Can
Acapt
Bxp.
Condition 1: Noisy labels. We vary the amount of noise p in the labels. We plot the percentage change over the baseline ResNet, av- eraged over all seeds and datasets.
Figure 6:
Figure 7: Condition 2: Fixed data. We vary the total size of the dataset T. We plot the percentage change over the baseline ResNet, averaged over all seeds and datasets.
Takeaway 6: The best algorithms may differ under the precise conditions. When labels have varying noise in figure 6, relative performance is reasonably consistent. When the dataset size decreases in figure 7, heuristic augmentation methods perform poorly. However, using pretraining and learned augmentation consistently robust.
Takeaway 7:
The precise attributes consider directly impacts the results. For example, on DSPRITES, if we make color y' and shape y", we find that all methods generalise perfectly in the unseen data shift setting (as demonstrated in appendix B.3) unlike when shape is y (figure 10).
4.2 PRACTICAL TIPS
While there is no free lunch in terms of the method choose, recommend the following tips.
Tip heuristic augmentations approximate part of the true underlying generative model, use them. Under this constraint, heuristic augmentations can significantly improve performance; this should be a first point of call. How to heuristically choose these augmentations without exhaus- tively trying all possible combinations is an open research question.
Tip 2: heuristic augmentations do not help, learn the augmentation. the true underlying generative model cannot be readily approximated with heuristic techniques, but some subset of the generative model can be learned by conditioning on known attributes, this is a promising way to further improve performance. How to learn the underlying generative model directly from data and use this for augmentation is promising area to explore more thoroughly.
Tip Use pretraining. In general, pretraining was found be a useful way learn robust rep- resentation. While this was not true for all datasets (e.g. CAMELYON 17, IWILDCAM), performance could be dramatically improved by pretraining (DSPRITES, MPI3D, SMALLNORE, SHAPES3D). An area to be investigated is the utility of self-supervised pre-training.
Tip More complex approaches lead limited improvements. Domain generalization, adap- tive approaches and disentangling lead to limited improvements, if any, across the different datasets and shifts. Of these approaches, DANN performs generally best. How to make these approaches generically useful for robustness is still an open research question.
5 DISCUSSION
Our experiments demonstrate that one method performs best over all shifts and that performance dependent on the precise attribute being considered. This leads the following considerations.
There is no way to decide a-priori on the best method given only the dataset. It would be helpful for practitioners to be able to select the best approaches without requiring comprehensive evaluations and comparisons. Moreover, it is unclear how to pinpoint the precise distribution shift (and thereby methods to explore) in a given application. This should be an important future area of investigation.
We should focus on the cases where we have knowledge about the distribution shift. We found that the ability of given algorithm to generalize depends heavily the attribute and dataset being
8

---

Published as a conference paper at ICLR 2022
considered. Instead of trying make one algorithm for any possible shift, it makes to have adaptable algorithms which can use auxiliary information given. Moreover, algorithms should be evaluated in the context for which we will use them.
is pivotal to evaluate methods in a variety of conditions. Performance varies due to the number of examples, amount of noise, and size of the dataset. Thus it is important perform comprehensive evaluations when comparing different methods, as in our framework. This gives others a more realistic view of different models' relative performance in practice.
RELATED WORK
We briefly summarize benchmarks on distribution shift, leaving a complete review to appendix C.
Benchmarking robustness to out of distribution (OOD) generalization. While a multitude of methods exist that report improved OOD generalization, Gulrajani & Lopez-Paz (2021) found that in actuality no evaluated method performed significantly better than a strong ERM baseline on variety of datasets. However, Hendrycks et al. (2021) found that, when we focus on better aug- mentation, larger models and pretraining, we can get sizeable boost in performance. This can be seen on the Koh al. (2020) benchmark (the largest boosts come from larger models and better augmentation). Our work is complementary to these methods, as we look at range of approaches (pretraining. heuristic augmentation, learned augmentation, domain generalisation, adaptive, disen- tangled representations) on range of both synthetic and real-world datasets. Moreover, allow for a fine- -ee-grained analysis of methods over different distribution shifts.
Benchmarking spurious correlation and low-data drift. Studies on faimeess and bias (surveyed by Mehrabi et al. (2021)) have demonstrated the pernicious impact of low-data in face recognition (Buolamwini & Gebru, 2018), medical imaging (Castro et al., 2020), and conservation (Beery et al., 2018) and spurious correlation in classification (Geirhos al., 2019) and conservation (Beery al., 2020). Arjovsky et al. (2019) hypothesized that spurious correlation may be the underlying reason for poor generalization of models to unseen data. To our knowledge, there has been no large scale work focused on understanding the benefits of different methods across these distribution shifts systematically across multiple datasets and with fine-graind control on the amount of shift. Here we introduce a framework for creating these shifts in a controllable way to allow such challenges to be investigated robustly.
Benchmarking disentangled representations. A related area, disentangled representation learn- ing, aims to learn representation where the factors of variation in the data are separated. If this could be achieved, then models should be able to generalise effortlessly to unseen data as inves- tigated in multiple settings such reinforcement learning (Higgins et al., 2017b). Despite many years of work in disentangled representations (Higgins et al., 7a; Burgess et al., 2017; Kim & Mnih, 2018; Chen al., 2018), benchmark study by Locatello al. (2019) found that, without supervision or implicit model or data assumptions, one cannot reliably perform disentanglement; however, weak supervision appears sufficient to do (Locatello et al., 2020). Dittadi et al. (2021); Schott et al. (2021); Montero et al. (2020) further investigated whether representations (disentan- gled or not) can interpolate, extrapolate, or compose properties; they found that when considering complex combinations of properties and multiple datasets, representations do not do so reliably.
7
CONCLUSIONS
This work has put forward general, comprehensive framework reason about distribution shifts. We analyzed 19 different methods, spanning a range of techniques, over three distribution shifts spurious correlation, low-data drift, and unseen data shift, and two additional conditions - label noise and dataset size. We found that while results are not consistent across datasets and methods, number of methods do better than an ERM baseline in some settings. We then put forward a number of practical tips, promising directions, and open research questions. We hope that our framework and comprehensive benchmark spurs research on in this area and provides a useful tool for practitioners to evaluate which methods work best under which conditions and shifts.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors thank Irina Higgins and Timothy Mann for feedback and discussions while developing their work. They also thank Irina, Rosemary Ke, and Dilan Gorur for reviewing earlier drafts.
REFERENCES
Ehab A AlBadawy, Ashirbani Saha, and Maciej A Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutioaal training and testing. Medical physics, 2018.
Michael A Alcorn, Li Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: orXiv:/907.028900 2019.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status the patient level: the CAMELYONI7 challenge. IEEE Transactions on Medical Imaging, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference Computer Vision, 2018.
Sara Beery, Yang Liu, Dan Morris, Jim Piavis, Ashish Kapoor, Neel Joshi, Markus Meister, and Pietro Perona. Synthetic examples improve generalization for rare classes. In Proceedings of the IEEE Workshop on Applications of Computer Vision, 2020.
Joy Buolamwini and Timnit Gebru. Gender shades: cial gender classification. In Conference on fairness, accountability and transparency. 2018.
Intersectional accuracy disparities in commer-
Chris Burgess and Hyunjik Kim. 3D shapes dataset. http:///ghhhbboommmeeemmm////aas 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. In Workshop Learning Disentangled Representations at the s1 Conference on Neural Information Processing Systems, 2017.
Fabio M Carlucci, Paolo Russo, Tatiana Tommasi. and Barbara Caputo. Hallucinating agnostic images to generalize across domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Daniel C Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communications, 2020.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Woo Ha, Sunghun Kim, and Jaegul Choo. Star- Gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
10

---

Published as a conference paper at ICLR 2022
Ekin D Cubuk, Barret Zoph. Jonathon Shlens, and Quoe V Le. RandAugment: Practical auto- mated data augmentation with a reduced search space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020.
Dengxin Dai and Luc Van Gool. Dark model adaptation: Semantic image segmentation from day- time to nighttime. In International Conference on Intelligent Transportation Systems, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. In Proceedings of the International Conference on Learning Representations, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition scale. In Proceedings of the International Conference on Learning Representations, 2021.
Bradley J Erickson, Panagiotis Korfiati,, Zeynettin Akkus, and Timothy L Kline. Machine learning for medical imaging. Radiographics, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(1):2096-2030, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness. In Proceedings of the International Conference on Learning Representations, 2019.
Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the subgroup performance gap with data augmentation. arXiv preprint 2rXiv:2008.06775. 2020.
Muhammad Waleed Gondal, Manuel Wüthrich, Dorde Miladinović, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. arXiv preprint arXiv:1906.03220 2019.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Pushmeet Kohli. Achieving robustness in the wild via adversarial mixing with disen- tangled representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Keren Gu, Xander Masotto, Vandana Bachani, Balaji Lakshminarayanan, Jack Nikodem, and Dong Yin. An instance-dependent simulation framework for learning with label noise. arXiv preprint ar7.v:2 1413, 2021.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In Proceedings of the International Conference on Learning Representations, 2021.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, 2018

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Will Douglas Heaven. Google's medical ai was super accurate in lab. real life was different story., 2020.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Repre- sentations, 2019.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems, 2018.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre- pre-training can improve model robustness and uncertainty. In Proceedings of the International Conference on Machine Learning, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: simple data processing method improve robustness and uncertainty. In Advances in Neural Information Processing Systems, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. Proceedings of the International Conference on Computer Vision, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In Proceedings of the International Conference on Learning Representations, 2017a
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2017b.
Joel Janai, Fatma Güney,, Aseem Behl, Andreas Geiger, et al. Computer vision for autonomous vehicles: Problems, datasets and state of the art. Foundations and Trends in Computer Graphics and Vision, 2020.
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The International Conference on Artificial Intelligence and Statistics. PMLR, 2019.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Ziclinski, Martin Steinegger, Michalina Pacholska. Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy labeled data. In Proceedings of the International Conference on Learning Representations, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the International Conference Machine Learning, 2018
12

---

Published as a conference paper at ICLR 2022
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of wn-the-wild distribution shifts. arXiv preprint urXiv:0012.07421, 2020.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance pose and lighting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2004.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the International Conference on Computer Vision, 2017.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision, pp. 624-639 639, 2018.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just Train Twice: Improving group robustness without training group information. In Proceedings of the International Conference on Machine Learning. 2021.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In Proceedings of the International Conference on Machine Learming. 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervisee disentanglement without compromises. In Proceedings of the International Conference on Machine Learning, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the International Conference Machine Learning, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. Proceedings of the International Conference on Machine Learning, 2017.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset,
2017.
Ninarch Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 2021.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In Proceedings of the International Conference on Learning Representations, 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the International Conference on Machine Learning, 2010.
Hyeonseob Nam, HyunJae Lee, Jongehan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi--ourre domain adaptation. In Proceedings of the International Conference on Computer Vision, 2019.
13

---

Published as a conference paper at ICLR 2022
Christian S Perone, Pedro Ballester, Rodrigo C Barros, and Julien Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. Neurolmage, 2019. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Proceedings of the International Conference on Machine Learning. 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. Proceedings of the International Conference on Machine Learning. 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. Proceedings of the International Conference on Learning Representations, 2020.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Proceedings of the International Conference on Learning Representations, 2020.
Lukas Schott, Julius von Kügelgen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Berharr Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. In Proceedings of the International Con- ference on Learning Reprexentations, 2021.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? arXiv preprint
2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Proceedings of the European Conference on Computer Vision, 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint AXXv:20077000444 2020.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. IEEE, 2011.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in Neural Infor- mation Processing Systems, 1992.
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Yufei Wang, Haoliang Li, and Alex C Kot. Heterogencous domain generalization via domain mixup. In Proceedings of the IEEE International Conference Acoustics, Speech and Signal P Processing, 2020.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or Signal: The role of image backgrounds in object recognition. ArXiv preprint arXiv:2006. 09994, 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In Proceedings of the International Conference Learning Representations, 2020.
14

---

Published as a conference paper at ICLR 2022
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. AAAI Conference on Artificial Intelligence, 2020.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprini urXiv:2001.0067. 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. MixUp: Beyond empir- ical risk minimization. Proceedings of the International Conference on Learning Representa- lions, 2018.
Ling Zhang, Xiaosong Wang, Dong Yang, Thomas Sanford, Stephanie Harmon, Baris Turkbey, Hol- ger Roth, Andriy Myronenko, Daguang Xu, and Ziyue Xu. When unseen domain generalization is unnecessary? rethinking data augmentation. arXiv preprint 1906,03990003322 2019.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant rep- resentations for domain adaptation. In Proceedings of the International Conference on Machine Learning. 2019.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversare im- age generation for domain generalisation. In AAAI Conference Artificial Intelligence, 2020.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the International Conference on Computer Vision 2017.
15


Reviews:
[{"review_id": "d4k6B-0OJa1", "reviewer": "Reviewer_HwsU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n-----------------------------\n\n- Firstly, this paper is a job well done! It was written very clearly and most parts were very easy to follow. The motivations of the paper are very clear. The major strength of the paper lies in the extensive experimental evaluation provided across the main paper and the supplementary, which may be used as a standardized reference for many future works. \n\n- The efforts made towards stringing various works aimed at generalization, as well as various natures of distribution shifts, on a common thread is a commendable effort. \n\n- The figures and the graphical illustrations are very well made, and effectively distill the major inferences of the experiments.\n\n### Required Clarifications\n-----------------------------\n\n- My major question to the authors of the paper is how general the proposed framework? You seem to select different datasets, but take only two attributes on each datasets which raises questions if the findings of the study hold with larger scale real world datasets with multiple factors of variation, often heavily entangled. I understand that these settings are chosen to keep the experiments tractable, but it would be helpful to provide insight into whether the framework holds for Imagenet scale datasets with unknown (and possibly unobservable) factors of variation [1].\nFor example, the inferences and observations in sec 4.1 and sec 5 are conditioned on the assumption that the factors of variation are known, which gives us a cue to which model might work (CycleGAN for low data-drift, pretraining for unseen data shift etc). But can this knowledge be extrapolated to judge *in-the-wild* datasets? This brings me to the next point.\n\n- I could not pinpoint a single take-home message from the paper. The empirical study results have high variance, and it can only be inferred that no single method works for all the cases. But can the authors, equipped with the knowledge of these experiments, draw any *meta recommendation* of mapping between (factors of variation, distribution shift, modeling) choices? Of course, I fully agree that the current findings are still incredibly useful, but I was expecting to see a more optimistic take home message for the readers : ) .\n- I fail to see that significance of the latent factorization model proposed in sec 2.1. While the formulation of the data, discrete attributes and the labels seem clear, I do not see the significance of introducing the latent factor *z*. The explanation seems equally effective by using only notation of (x,y). Also looking at the models used to achieve robustness, none of weighted resampling or data augmentation require any latent factorization. Similarly, Imagenet pretraining also does not necessarily relate to latent factors in the attributes. Beta-VAE however relates to latent factors but that's about it. Perhaps the whole section 2.1 can be better motivated in the context of the paper.\n- The domain adaptation works generally work with a *covariate shift* assumption [2]. How does that fit into the proposed framework? Also, DA works like DANN and CORAL generally make use of additional unlabeled data. How are the unlabeled data chosen for the experiments? While it is inferred (sec 4.2) that domain adaptation methods lead to limited improvements, the DA methods considered are quite old and primitive. Perhaps more recent DA methods might help?\n- The authors could also contrast the in-distribution vs. out-of-distribution performance of the models. It seems that arch. like ViT might do very well on training domain but lack generalization on OOD test data. Why is this so? Does this have to do with the strong prior that CNNs induce that help generalization?\n- Is Imagenet pretraining performing better only because it is trained on larger scale data compared to other methods? In Fig 7, are all models trained/pretrained using same amount of data?\n\n\n1. Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" _(2019).\n2. Ben-David, Shai, et al. \"A theory of learning from different domains.\" _Machine learning_ 79.1 (2010): 151-175.", "summary_of_the_paper": "## Very useful study on calibrating OOD models and benchmarks\n\nThis paper provides a fine-grained analysis on nature of distribution shifts in image datasets through extensive experiments on various benchmark methods for robustness. They work with an assumption that the underlying factors of variation, including the label, are encoded through discrete attributes. The authors provide a concrete framework to put forth the latent factorization model, including appropriate grouping for various types of benchmark methods, distribution shifts as well as training conditions. Several experimental demonstrations are provided to compare and contrast robustness of existing methods leading to numerous useful observations.", "summary_of_the_review": "Overall, I believe that the experiments conducted as part of this paper are very well structured and provide multiple useful insights to the community. However, the analysis still falls short in several places (see above). The latent factorization models needs to be better motivated, and the generality of the framework needs to be clarified. Still, this is a very useful work to the community, and if the authors could clarify the questions raised, I would be happy to update the score. I haven't gone through the supplementary material in detail. If any of the questions above have a direct answer in the suppl. material, the authors can directly point to that and I would be happy to update my comments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OogjCtI0XVl", "reviewer": "Reviewer_zAfw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "===================\nStrengths:\nIn my opinion, the efforts to unify all the studies of distribution shifts into one general framework is the greatest contribution made by this paper. Recently, plenty of distribution shift settings become popular in various machine learning fields, such as long-tailed classification/detection, domain adaptation, out-of-distribution generalization, etc. However, they are all studied independently, despite the fact that a similar essence is shared by them under the surface. This framework puts them into the same testbed, which allows us to evaluate the scope and limitation of different methods in all kinds of settings and data types. Besides, the authors also provided a number of practical tips and useful conclusions based on massive experiments under diverse settings and datasets.\n\n===================\nWeaknesses:\n- However, some conclusions in this paper violate my own observations. For example, the heuristic data augmentation method, Rand Augmentation, is found to be very helpful in long-tailed classification (a specific type of low-data drift) and some label noise tasks. It can consistently improve 2-5 points of accuracy across different datasets and settings. Yet, Rand Augmentation looks to be the worst method in Figures 3 and 4. One possible explanation is that datasets I was using are all from real-world images, so the Rand Aug can reveal the underlying generative model p(x|y1:K), while the datasets used in this paper are mostly synthetic images or medical images, where Rand Aug didn't approximate the true underlying generative model p(x|y1:K). In order to prevent the conclusions in this paper from misleading future researchers, I suggest the authors summarize datasets into different types (like synthetic dataset, medical dataset, and normal dataset) and investigate them separately.\n- Another concern is that should the ImageNet pretraining be used as a valid method for the study of robustness to distribution shifts? It may violate the settings of SC, LDD, and UDS by introducing samples from unknown distributions. For example, unseen data distribution in the given task and dataset may be compensated by the ImageNet dataset. That's why pretraining is usually forbidden in the tasks like OOD image classification or long-tailed classification. As a result, the superior performance of pertaining is not just frustrating but also (could be) unfair.", "summary_of_the_paper": "===================\nSummary: \nThe robustness to distribution shifts is one of the biggest concerns in deploying machine learning systems. This paper summarizes all distribution shifts into three prototypes: spurious correlation, low-data drift, and unseen data shift, together with two additional conditions: label noise and dataset size. Other more complex distribution shifts can be regarded as compositions of these components. After that, they evaluated19 methods, spanning 5 categories: architecture choice, data augmentation, domain generalization, adaptive algorithms, and representation learning, in 6 datasets. In the end, they provided a number of practical tips, useful conclusions, and promising directions for future researchers in this field.", "summary_of_the_review": "===================\nJustification:\nI recognize the value of the proposed comprehensive framework and all systematic studies made by this paper. I would be more than willing to accept the paper once my concerns are properly addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HEAmu7OW1mZ", "reviewer": "Reviewer_U5qL", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "“find that a model trained on one set of hospitals may not generalise to the imaging conditions of another.” -> Training on “one set of hospitals” is a weird formulation and it’s not clear what is meant; set of MRIs from multiple hospitals or similar? Please rewrite this sentence.\n\n\n“These methods span the following 5 common approaches: architecture choice, data augmentation, domain generalization, adaptive algorithms, and representation learning.” The collection of approaches written in this way sounds a bit strange to me because the choices are neither orthogonal nor of the same kind. Architecture choice is a property of the model being trained (irrespective of the task), data augmentation heavily depends on the task, domain generalization IS a task, and representation learning is a whole area of machine learning. At this stage of the paper, I find this list very confusing and would suggest rewriting or clarifying this sentence.\n\n\n“We assume that distribution shifts arise…” Distribution (or covariate) shift is usually defined using the definition in [1], section 2.1.1. Can you please comment on whether there is difference to your definition and what it means for the approaches one would use to tackle it? From my understanding, it is actually the same because in [1], covariate shift is defined as (a) P(X) != P’(X) and (b) P (Y |X) = P’(Y |X). Here, the authors also have the condition (b) and write that P(y_1:k) != p_train(y_1:k) != p_test(y_1:k) which will result in the condition (a) P(x) != P’(X). In any case, distribution shift is a well-defined phenomenon; thus, it would be good if the authors could cite some standard definition (e.g., [1]) and comment why their definition is different (if it is different).\n\n\nPage 3: Data Augmentation. The authors write “alternatively” and I believe they refer to the previously proposed reweighting method. Writing “alternatively” means to me that they describe the methods separately first. Therefore, I am confused by the definition of p_aug since it still contains p_reweight. The authors should either (1) remove p_reweight from p_aug or (2) write “additionally” instead of “alternatively”. I would prefer (1).\n\n\nPage 3: Data Augmentation. I don’t quite agree with how data augmentation is defined here. The authors write that one can synthesize artificial data with a generative model that aims to approximate the true generative model and use this data as data augmentation. I don’t think this understanding of using data augmentation is how researchers generally think about data augmentation. I think the main driving factor for using data augmentation is to artificially increase the dataset size and to make the classifier more smooth around the data points, i.e. its decision should not change if the data distribution changes slightly. For this, researchers generally do not try to learn the true generative model, but rather use simple augmentations such as e.g., Gaussian noise, crops and horizontal flips. Later, the authors write that they use AugMix, RandAugment and AutoAugment as augmentations. All these augmentations are simple parametric functions that neither depend on the latent z nor on x since they can be used irrespective of the data and/or task. Learning the generative model is more of a GAN-like approach, so I would maybe split the paragraph into something like (1) Standard/Simple/Parametric/Heuristic data augmentation and (2) Learned data augmentation. The authors actually split the two notions of data augmentation in section 3. I would suggest just using the same split here. The authors actually claim in Takeaway 3 that “Heuristic augmentation improves generalization if the augmentation describes an attribute.” Nevertheless, I think defining data augmentation in this way here is a bit preemptive.\n\nPage 7: “We report the mean and standard deviation over the five seeds.” I absolutely appreciate the effort done here for reporting the error over 5 runs as this is unfortunately not standard over even common in most papers.\n\n\nPage 4: “Test distribution…” please define A_i\n\n\nPage 4: “Shift 1:” Should it be P_test on the right hand side of the definition?\n\n\nPage 4: “Shift 2: Low-data drift” -> Maybe mentioning that this issue is being studied by the fairness community would be good.\n\n\nPage 4: “Shift 3: Unseen data shift – Some attribute values are unseen under ptrain but are under ptest.” Suggest to add “present” as in “but are present under p_test”.\n\n\nShift 3: “which we make explicit due to its important real world applications” -> importance\n\n\nShift 3: I do not understand the right-most inequality in Eq.2. Please explain.\n\n\nI really like the structure and content of section 2.2. The authors put a lot of effort into properly and carefully defining and disentangling different distribution shifts. I also appreciate the provided examples from DSprites to illustrate all the different shifts.\n\n\nFigure 3: I find it weird that using more samples from the true distribution seems to hurt performance in many cases. Additionally, checking Figure 11 in the Appendix, it looks like higher N is correlated with higher accuracy though it is hard to judge only looking at the image. The authors should comment on this.\n\n\nFigure 4: Does this Figure show the accuracy on the biased or non-biased datasets? Similar to Figure 11, in Figure 12, it again looks like a higher N results in higher accuracy. In Figure 12, some bars are missing.\n\n\nFigures 10, 11 and 12 are generally hard to read because there are so many bars and the only information one can extract is that there is a monotonic increase over the different models which is pretty useless since the ordering has likely been chosen such that there is a monotonic increase. Judging how N changes the bar heights is not really possible. I would suggest to maybe plot these Figures as heat plots similar to Figures 3 and 4. With heat plots, the authors could also write the resulting accuracy numbers into the heat map squares. This will allow researchers to cite their numbers in the future and also make reproducibility easier since these numbers can be easily compared against.\n\n\nIt is annoying that the Figures 10-12 in the Appendix are not plotted together and also not close to the corresponding text. I would highly suggest restructuring the Appendix to make parsing it easier, especially, since it is very long. For example, the code for the framework can be put at the very end such that it does not break the flow when looking at the additional results. \n\n\nTakeaway 3. I do not see how the presented evidence leads to the drawn conclusion that “Heuristic augmentation improves generalization if the augmentation describes an attribute.” In B.2, the authors merely show that “No augmentation always leads to a strong boost in performance.” But it is not discussed in what way the successful augmentations approximate the true generative model. This type of analysis has done on ImageNet-C and CIFAR10-C in ref. [2] which the authors should cite here. In ref. [2], the authors define a minimal sample distance between the expected distribution shift and the added data augmentation and find strong correlation between the two. Here, a similar analysis would need to be performed in order to be able to do such a claim. In this light, tip 1 is also not grounded on evidence (although it is pretty obvious).\n\n\nI did not understand whether model selection is part of the framework code?\n\n\nThe tips in 4.2. are not really surprising (or novel). In my opinion, tip 1 is not grounded on the presented evidence since the authors did not show results that would support their takeaway 3. It is an obvious tip though and intuitively, it makes absolute sense, but the authors did not show results to support it. Considering tip 2, I think learning the perfect generative model is a hard task and I am not sure how feasible this task is for ImageNet scale datasets. Given how difficult and unstable GAN training can be, I am not sure how practical this tip is. The authors also did not show results on ImageNet, so it is hard to judge whether this tip would scale to ImageNet. Considering tip3, it is well known that pretraining on ImageNet is powerful as is has been shown in numerous previous works. As for tip4, I am not aware that DANN has been scaled to ImageNet. Additionally, DANN training (being a minimax optimization problem) can be unstable and depend on hyperparameters. Thus, I question the practicality of this tip.\n\tSome of these tips are not novel and the authors (merely) provide additional evidence for them. It would be nice if the authors could cite some papers where these findings have also been reported.\n\n\nReferences: \n[1] Bernhard Schölkopf et al. “On causal and anticausal learning”.\n[2] Eric Mintun et al. “On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness”\n\n\n", "summary_of_the_paper": "The authors perform an extensive study of different types of distribution shifts to judge which methods perform best on which dataset and/or method. For this, they carefully define which distribution shifts they are interested in and perform a large scale study for each of the shifts. They propose a standardized framework (which can easily be extended to new methods) to train and evaluate new models and methods on the distribution shifts. ", "summary_of_the_review": "I find the paper well written and easy to follow. The studied distribution shifts are thoughtfully and carefully defined. The authors perform a ton of experiments for the distribution shifts they want to study. I think the proposed benchmark / framework is a great addition to the research community as it will standardize model training and evaluation.\n\n\nPoints I would especially like to see addressed during the rebuttal (partially copied the most important points to me from the main review):\n\n1.\tFigure 3: I find it weird that using more samples from the true distribution seems to hurt performance in many cases. Additionally, checking Figure 11 in the Appendix, it looks like higher N is correlated with higher accuracy though it is hard to judge only looking at the image. The authors should comment on this.\n\n2.\tFigure 4: Does this Figure show the accuracy on the biased or non-biased datasets? Similar to Figure 11, in Figure 12, it again looks like a higher N results in higher accuracy. In Figure 12, some bars are missing.\n\n3.\tTakeaway 3. I do not see how the presented evidence leads to the drawn conclusion that “Heuristic augmentation improves generalization if the augmentation describes an attribute.” In B.2, the authors merely show that “No augmentation always leads to a strong boost in performance.” But it is not discussed in what way the successful augmentations approximate the true generative model. This type of analysis has been done on ImageNet-C and CIFAR10-C in ref. [2] which the authors should cite here. In ref. [2], the authors define a minimal sample distance between the expected distribution shift and the used data augmentation and find strong correlation between the two. Here, a similar analysis would need to be performed in order to be able to do such a claim. In this light, tip 1 is also not grounded on evidence (although it is pretty obvious).\n\n4. Contribution 1: \"We propose a framework to define when and why we expect methods to generalise.\" I think the authors addressed the \"when\" question with their benchmark, but not really the \"why\" question. Can the authors comment on which results lead them to conclude why a certain method should generalize?\n\n\n-> I am also happy to discuss any other points I mentioned but these would be the most important ones to me.\n\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a general framework to reason about fine-grained distribution shifts, evaluating a large set of different approaches in a variety of settings. All reviewers recommend acceptance. While concerns were raised, including questions about the generality of the framework, unsurprising “tips”, and unclear take-home messages, all reviewers find the work strong, with an elegant formulation, and useful insights. The AC agrees with the reviewers that this work addresses a very important problem, proposes an interesting unified framework and benchmark for domain shift analysis, and should be a valuable tool for the community to pursue further research in this area.
Decision: Accept (Oral)