Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ## Very useful study on calibrating OOD models and benchmarks

This paper provides a fine-grained analysis on nature of distribution shifts in image datasets through extensive experiments on various benchmark methods for robustness. They work with an assumption that the underlying factors of variation, including the label, are encoded through discrete attributes. The authors provide a concrete framework to put forth the latent factorization model, including appropriate grouping for various types of benchmark methods, distribution shifts as well as training conditions. Several experimental demonstrations are provided to compare and contrast robustness of existing methods leading to numerous useful observations.
main_review: ### Strengths
-----------------------------

- Firstly, this paper is a job well done! It was written very clearly and most parts were very easy to follow. The motivations of the paper are very clear. The major strength of the paper lies in the extensive experimental evaluation provided across the main paper and the supplementary, which may be used as a standardized reference for many future works. 

- The efforts made towards stringing various works aimed at generalization, as well as various natures of distribution shifts, on a common thread is a commendable effort. 

- The figures and the graphical illustrations are very well made, and effectively distill the major inferences of the experiments.

### Required Clarifications
-----------------------------

- My major question to the authors of the paper is how general the proposed framework? You seem to select different datasets, but take only two attributes on each datasets which raises questions if the findings of the study hold with larger scale real world datasets with multiple factors of variation, often heavily entangled. I understand that these settings are chosen to keep the experiments tractable, but it would be helpful to provide insight into whether the framework holds for Imagenet scale datasets with unknown (and possibly unobservable) factors of variation [1].
For example, the inferences and observations in sec 4.1 and sec 5 are conditioned on the assumption that the factors of variation are known, which gives us a cue to which model might work (CycleGAN for low data-drift, pretraining for unseen data shift etc). But can this knowledge be extrapolated to judge *in-the-wild* datasets? This brings me to the next point.

- I could not pinpoint a single take-home message from the paper. The empirical study results have high variance, and it can only be inferred that no single method works for all the cases. But can the authors, equipped with the knowledge of these experiments, draw any *meta recommendation* of mapping between (factors of variation, distribution shift, modeling) choices? Of course, I fully agree that the current findings are still incredibly useful, but I was expecting to see a more optimistic take home message for the readers : ) .
- I fail to see that significance of the latent factorization model proposed in sec 2.1. While the formulation of the data, discrete attributes and the labels seem clear, I do not see the significance of introducing the latent factor *z*. The explanation seems equally effective by using only notation of (x,y). Also looking at the models used to achieve robustness, none of weighted resampling or data augmentation require any latent factorization. Similarly, Imagenet pretraining also does not necessarily relate to latent factors in the attributes. Beta-VAE however relates to latent factors but that's about it. Perhaps the whole section 2.1 can be better motivated in the context of the paper.
- The domain adaptation works generally work with a *covariate shift* assumption [2]. How does that fit into the proposed framework? Also, DA works like DANN and CORAL generally make use of additional unlabeled data. How are the unlabeled data chosen for the experiments? While it is inferred (sec 4.2) that domain adaptation methods lead to limited improvements, the DA methods considered are quite old and primitive. Perhaps more recent DA methods might help?
- The authors could also contrast the in-distribution vs. out-of-distribution performance of the models. It seems that arch. like ViT might do very well on training domain but lack generalization on OOD test data. Why is this so? Does this have to do with the strong prior that CNNs induce that help generalization?
- Is Imagenet pretraining performing better only because it is trained on larger scale data compared to other methods? In Fig 7, are all models trained/pretrained using same amount of data?


1. Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." _(2019).
2. Ben-David, Shai, et al. "A theory of learning from different domains." _Machine learning_ 79.1 (2010): 151-175.
summary_of_the_review: Overall, I believe that the experiments conducted as part of this paper are very well structured and provide multiple useful insights to the community. However, the analysis still falls short in several places (see above). The latent factorization models needs to be better motivated, and the generality of the framework needs to be clarified. Still, this is a very useful work to the community, and if the authors could clarify the questions raised, I would be happy to update the score. I haven't gone through the supplementary material in detail. If any of the questions above have a direct answer in the suppl. material, the authors can directly point to that and I would be happy to update my comments.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors perform an extensive study of different types of distribution shifts to judge which methods perform best on which dataset and/or method. For this, they carefully define which distribution shifts they are interested in and perform a large scale study for each of the shifts. They propose a standardized framework (which can easily be extended to new methods) to train and evaluate new models and methods on the distribution shifts. 
main_review: “find that a model trained on one set of hospitals may not generalise to the imaging conditions of another.” -> Training on “one set of hospitals” is a weird formulation and it’s not clear what is meant; set of MRIs from multiple hospitals or similar? Please rewrite this sentence.


“These methods span the following 5 common approaches: architecture choice, data augmentation, domain generalization, adaptive algorithms, and representation learning.” The collection of approaches written in this way sounds a bit strange to me because the choices are neither orthogonal nor of the same kind. Architecture choice is a property of the model being trained (irrespective of the task), data augmentation heavily depends on the task, domain generalization IS a task, and representation learning is a whole area of machine learning. At this stage of the paper, I find this list very confusing and would suggest rewriting or clarifying this sentence.


“We assume that distribution shifts arise…” Distribution (or covariate) shift is usually defined using the definition in [1], section 2.1.1. Can you please comment on whether there is difference to your definition and what it means for the approaches one would use to tackle it? From my understanding, it is actually the same because in [1], covariate shift is defined as (a) P(X) != P’(X) and (b) P (Y |X) = P’(Y |X). Here, the authors also have the condition (b) and write that P(y_1:k) != p_train(y_1:k) != p_test(y_1:k) which will result in the condition (a) P(x) != P’(X). In any case, distribution shift is a well-defined phenomenon; thus, it would be good if the authors could cite some standard definition (e.g., [1]) and comment why their definition is different (if it is different).


Page 3: Data Augmentation. The authors write “alternatively” and I believe they refer to the previously proposed reweighting method. Writing “alternatively” means to me that they describe the methods separately first. Therefore, I am confused by the definition of p_aug since it still contains p_reweight. The authors should either (1) remove p_reweight from p_aug or (2) write “additionally” instead of “alternatively”. I would prefer (1).


Page 3: Data Augmentation. I don’t quite agree with how data augmentation is defined here. The authors write that one can synthesize artificial data with a generative model that aims to approximate the true generative model and use this data as data augmentation. I don’t think this understanding of using data augmentation is how researchers generally think about data augmentation. I think the main driving factor for using data augmentation is to artificially increase the dataset size and to make the classifier more smooth around the data points, i.e. its decision should not change if the data distribution changes slightly. For this, researchers generally do not try to learn the true generative model, but rather use simple augmentations such as e.g., Gaussian noise, crops and horizontal flips. Later, the authors write that they use AugMix, RandAugment and AutoAugment as augmentations. All these augmentations are simple parametric functions that neither depend on the latent z nor on x since they can be used irrespective of the data and/or task. Learning the generative model is more of a GAN-like approach, so I would maybe split the paragraph into something like (1) Standard/Simple/Parametric/Heuristic data augmentation and (2) Learned data augmentation. The authors actually split the two notions of data augmentation in section 3. I would suggest just using the same split here. The authors actually claim in Takeaway 3 that “Heuristic augmentation improves generalization if the augmentation describes an attribute.” Nevertheless, I think defining data augmentation in this way here is a bit preemptive.

Page 7: “We report the mean and standard deviation over the five seeds.” I absolutely appreciate the effort done here for reporting the error over 5 runs as this is unfortunately not standard over even common in most papers.


Page 4: “Test distribution…” please define A_i


Page 4: “Shift 1:” Should it be P_test on the right hand side of the definition?


Page 4: “Shift 2: Low-data drift” -> Maybe mentioning that this issue is being studied by the fairness community would be good.


Page 4: “Shift 3: Unseen data shift – Some attribute values are unseen under ptrain but are under ptest.” Suggest to add “present” as in “but are present under p_test”.


Shift 3: “which we make explicit due to its important real world applications” -> importance


Shift 3: I do not understand the right-most inequality in Eq.2. Please explain.


I really like the structure and content of section 2.2. The authors put a lot of effort into properly and carefully defining and disentangling different distribution shifts. I also appreciate the provided examples from DSprites to illustrate all the different shifts.


Figure 3: I find it weird that using more samples from the true distribution seems to hurt performance in many cases. Additionally, checking Figure 11 in the Appendix, it looks like higher N is correlated with higher accuracy though it is hard to judge only looking at the image. The authors should comment on this.


Figure 4: Does this Figure show the accuracy on the biased or non-biased datasets? Similar to Figure 11, in Figure 12, it again looks like a higher N results in higher accuracy. In Figure 12, some bars are missing.


Figures 10, 11 and 12 are generally hard to read because there are so many bars and the only information one can extract is that there is a monotonic increase over the different models which is pretty useless since the ordering has likely been chosen such that there is a monotonic increase. Judging how N changes the bar heights is not really possible. I would suggest to maybe plot these Figures as heat plots similar to Figures 3 and 4. With heat plots, the authors could also write the resulting accuracy numbers into the heat map squares. This will allow researchers to cite their numbers in the future and also make reproducibility easier since these numbers can be easily compared against.


It is annoying that the Figures 10-12 in the Appendix are not plotted together and also not close to the corresponding text. I would highly suggest restructuring the Appendix to make parsing it easier, especially, since it is very long. For example, the code for the framework can be put at the very end such that it does not break the flow when looking at the additional results. 


Takeaway 3. I do not see how the presented evidence leads to the drawn conclusion that “Heuristic augmentation improves generalization if the augmentation describes an attribute.” In B.2, the authors merely show that “No augmentation always leads to a strong boost in performance.” But it is not discussed in what way the successful augmentations approximate the true generative model. This type of analysis has done on ImageNet-C and CIFAR10-C in ref. [2] which the authors should cite here. In ref. [2], the authors define a minimal sample distance between the expected distribution shift and the added data augmentation and find strong correlation between the two. Here, a similar analysis would need to be performed in order to be able to do such a claim. In this light, tip 1 is also not grounded on evidence (although it is pretty obvious).


I did not understand whether model selection is part of the framework code?


The tips in 4.2. are not really surprising (or novel). In my opinion, tip 1 is not grounded on the presented evidence since the authors did not show results that would support their takeaway 3. It is an obvious tip though and intuitively, it makes absolute sense, but the authors did not show results to support it. Considering tip 2, I think learning the perfect generative model is a hard task and I am not sure how feasible this task is for ImageNet scale datasets. Given how difficult and unstable GAN training can be, I am not sure how practical this tip is. The authors also did not show results on ImageNet, so it is hard to judge whether this tip would scale to ImageNet. Considering tip3, it is well known that pretraining on ImageNet is powerful as is has been shown in numerous previous works. As for tip4, I am not aware that DANN has been scaled to ImageNet. Additionally, DANN training (being a minimax optimization problem) can be unstable and depend on hyperparameters. Thus, I question the practicality of this tip.
	Some of these tips are not novel and the authors (merely) provide additional evidence for them. It would be nice if the authors could cite some papers where these findings have also been reported.


References: 
[1] Bernhard Schölkopf et al. “On causal and anticausal learning”.
[2] Eric Mintun et al. “On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness”



summary_of_the_review: I find the paper well written and easy to follow. The studied distribution shifts are thoughtfully and carefully defined. The authors perform a ton of experiments for the distribution shifts they want to study. I think the proposed benchmark / framework is a great addition to the research community as it will standardize model training and evaluation.


Points I would especially like to see addressed during the rebuttal (partially copied the most important points to me from the main review):

1.	Figure 3: I find it weird that using more samples from the true distribution seems to hurt performance in many cases. Additionally, checking Figure 11 in the Appendix, it looks like higher N is correlated with higher accuracy though it is hard to judge only looking at the image. The authors should comment on this.

2.	Figure 4: Does this Figure show the accuracy on the biased or non-biased datasets? Similar to Figure 11, in Figure 12, it again looks like a higher N results in higher accuracy. In Figure 12, some bars are missing.

3.	Takeaway 3. I do not see how the presented evidence leads to the drawn conclusion that “Heuristic augmentation improves generalization if the augmentation describes an attribute.” In B.2, the authors merely show that “No augmentation always leads to a strong boost in performance.” But it is not discussed in what way the successful augmentations approximate the true generative model. This type of analysis has been done on ImageNet-C and CIFAR10-C in ref. [2] which the authors should cite here. In ref. [2], the authors define a minimal sample distance between the expected distribution shift and the used data augmentation and find strong correlation between the two. Here, a similar analysis would need to be performed in order to be able to do such a claim. In this light, tip 1 is also not grounded on evidence (although it is pretty obvious).

4. Contribution 1: "We propose a framework to define when and why we expect methods to generalise." I think the authors addressed the "when" question with their benchmark, but not really the "why" question. Can the authors comment on which results lead them to conclude why a certain method should generalize?


-> I am also happy to discuss any other points I mentioned but these would be the most important ones to me.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ===================
Summary: 
The robustness to distribution shifts is one of the biggest concerns in deploying machine learning systems. This paper summarizes all distribution shifts into three prototypes: spurious correlation, low-data drift, and unseen data shift, together with two additional conditions: label noise and dataset size. Other more complex distribution shifts can be regarded as compositions of these components. After that, they evaluated19 methods, spanning 5 categories: architecture choice, data augmentation, domain generalization, adaptive algorithms, and representation learning, in 6 datasets. In the end, they provided a number of practical tips, useful conclusions, and promising directions for future researchers in this field.
main_review: ===================
Strengths:
In my opinion, the efforts to unify all the studies of distribution shifts into one general framework is the greatest contribution made by this paper. Recently, plenty of distribution shift settings become popular in various machine learning fields, such as long-tailed classification/detection, domain adaptation, out-of-distribution generalization, etc. However, they are all studied independently, despite the fact that a similar essence is shared by them under the surface. This framework puts them into the same testbed, which allows us to evaluate the scope and limitation of different methods in all kinds of settings and data types. Besides, the authors also provided a number of practical tips and useful conclusions based on massive experiments under diverse settings and datasets.

===================
Weaknesses:
- However, some conclusions in this paper violate my own observations. For example, the heuristic data augmentation method, Rand Augmentation, is found to be very helpful in long-tailed classification (a specific type of low-data drift) and some label noise tasks. It can consistently improve 2-5 points of accuracy across different datasets and settings. Yet, Rand Augmentation looks to be the worst method in Figures 3 and 4. One possible explanation is that datasets I was using are all from real-world images, so the Rand Aug can reveal the underlying generative model p(x|y1:K), while the datasets used in this paper are mostly synthetic images or medical images, where Rand Aug didn't approximate the true underlying generative model p(x|y1:K). In order to prevent the conclusions in this paper from misleading future researchers, I suggest the authors summarize datasets into different types (like synthetic dataset, medical dataset, and normal dataset) and investigate them separately.
- Another concern is that should the ImageNet pretraining be used as a valid method for the study of robustness to distribution shifts? It may violate the settings of SC, LDD, and UDS by introducing samples from unknown distributions. For example, unseen data distribution in the given task and dataset may be compensated by the ImageNet dataset. That's why pretraining is usually forbidden in the tasks like OOD image classification or long-tailed classification. As a result, the superior performance of pertaining is not just frustrating but also (could be) unfair.
summary_of_the_review: ===================
Justification:
I recognize the value of the proposed comprehensive framework and all systematic studies made by this paper. I would be more than willing to accept the paper once my concerns are properly addressed.
review_title: null

Meta-review: The paper proposes a general framework to reason about fine-grained distribution shifts, evaluating a large set of different approaches in a variety of settings. All reviewers recommend acceptance. While concerns were raised, including questions about the generality of the framework, unsurprising “tips”, and unclear take-home messages, all reviewers find the work strong, with an elegant formulation, and useful insights. The AC agrees with the reviewers that this work addresses a very important problem, proposes an interesting unified framework and benchmark for domain shift analysis, and should be a valuable tool for the community to pursue further research in this area.
Decision: Accept (Oral)