Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a thorough and relatively in-depth analysis of how pre-trained models affect OOD generalization performance. Specifically, the authors dissect this issue and perform extensive ablations on learning rates, model architecture, dataset size, model size, etc. By studying large-scale pretraining on vision tasks, the paper claims that 1) pre-trained models typically have better OOD generalization performance (which prior works have somewhat discussed as well); 2) small learning rates during the fine-tuning phase tend to create more robust models; and 3) some of the previous understandings (e.g., of how ID and OOD accuracies are related; or of the effect of training data size) might need to be revisited.
main_review: Overall, I think this paper provides a good set of empirical studies on an important problem: what contributes to the better OOD generalization performance of pre-trained models (and relatedly, how are ID accuracies related to these OOD accuracies). 

Strengths:
1. Extensive experiments. I especially like that the authors tried to tackle at this problem from multiple angles, such as LR, model size, model type, augmentations, etc.
2. The paper is overall well-written and easy to understand.
3. The underlying problem studied here is very important and while some of the insights have been brought up by prior works, this paper provides an in-depth analysis of how pre-trained models behave (under the influence of all these different factors) when there's a distribution shift. I have no doubt that the conclusion of this paper will be very useful for future work.
4. Good results and on large scales. The authors are able to take advantage of all these findings to produce good OOD generalization outcomes.

Weaknesses/problems:
1. The usefulness of pre-training was already studied before (e.g., [1,2]), though in somewhat different directions. This paper puts more emphasis on training mechanisms, which provide useful, interesting, but still incremental insights.
2. The conclusions are still empirical, domain-specific, and have some important questions (in my mind) unanswered. For example, since learning rate was a major factor studied in the paper, it makes sense to consider the effect of the learning rate annealing schedule as well, which is a technique very commonly used in training image classification models. As another example, the paper mentions multiple times how a **"more diverse"** dataset for pretraining would help (e.g., IG-1B-Targeted vs. ImageNet-1k). However, I feel that this is a more relative and subjective measure--- e.g., how do we know a'priori whether a dataset is "diverse enough"? 
3. I like the discussion on ID and OOD accuracy, so a related question: let's say we train 1) a model A on dataset C to reach 70% accuracy on C's test set; and 2) a model B (with the same **architecture** as A) on a larger pretraining dataset D first, and then fine-tune a bit (just a bit, not thoroughly; for example, you can early stop) on C to make it also reach 70% accuracy on the downstream dataset C's test set. Does the conclusion of this paper imply model B will always have better OOD performance than model A? This is different from the results presented in Table 8 because the models there have substantially worse ID accuracy as well.



[1] http://proceedings.mlr.press/v139/yi21a/yi21a.pdf
[2] https://aclanthology.org/2020.acl-main.244.pdf
summary_of_the_review: I recommend (weak) acceptance of this paper because 1) the conclusions drawn in this paper, while somewhat incremental, are still very useful and would be a great addition to this line of work; and 2) the experiments are thorough and realistic. I don't identify any obvious flaw in the methodology or analysis.

---------------------

Post-rebuttal: See my comment below. I'm updating the score to 7 (which doesn't exist in the ICLR review form, but just to show that I'm advocating more for acceptance).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Within the now common pre-train/fine-tune training paradigm, this paper provide an empirical examination of the influence of pre-training on out-of-distribution accuracy *after* fine-tuning, specifically for computer vision. The authors showcase evidence suggesting that the choice of the pre-training dataset, the size of the model and the fine-tuning learning rate directly influence downstream OOD accuracy throughout a series of tasks. In addition, they provide some analysis of the effect that various training techniques (eg. label smoothing, augmentation etc..) have on OOD accuracy when they are used at fine-tuning time.
main_review: I will preface by saying that I am not intimately familiar with the computer-vision literature and since, despite its unnecessarily general title and abstract, this paper is very computer-vision specific, I may be overlooking some finer details.

That being said, I think that the paper places itself nicely within the existing literature by casting the focus on OOD generalization specifically. While the conclusions (that larger models and more diverse datasets are better) are not unexpected, such detailed empirical studies are worthwhile for practitioners and researchers alike. However I think the paper falls short by not going one step further and providing additional insight to try and explain some of its (many) experimental findings. This makes it harder to tell whether and how these results would transfer to other architectures, modalities or datasets considered here, which ultimately limits the impact of the paper. I think the paper is in a good shape, but it would need additional ablations and analysis to warrant acceptance at ICLR.

Strengths:
- While the overall conclusion that larger models pre-trained on more (and more diverse) are better is not surprising, I think it is useful to confirm this finding finding with respect to OOD generalization specifically.
- Detailed analysis and experimental results, which should be useful for practitioners in the field
- Experimental results are overall well presented

Weaknesses:
- Unnecessarily general title and abstract: the paper is very much vision specific and this should be made clear from the get go. As it stands I found the opening of the paper rather misleading. There is some work outside of vision which looked at this particular problem and bears mentioning, for instance in NLP Hendricks et al. "Pretrained Transformers Improve Out-of-Distribution Robustness" https://arxiv.org/pdf/2004.06100.pdf
- While there are a lot of experimental results, I think the paper suffers from not going one step further and investigating the underlying causes behind a lot of the results. As it stands, the reader is left with rather vague and somewhat expected conclusions: "larger and more diverse datasets are better" and "smaller learning rates are better". 
   * What specifically in the larger datasets helps? Is it really diversity or size? For example, if IG-1B-Targeted is downsized to the size of Imagenet, do the improvements observed persist?
   * What about the smaller learning rates make them more suitable? I have a suspicion that it has to do with the weights deviating less from their pre-trained starting point during fine-tuning. It would be interested to look at the interaction between "catastrophic forgetting" of the pre-training task and OOD generalization.
summary_of_the_review: I think that the paper places itself nicely within the existing literature by casting the focus on OOD generalization specifically. While the conclusions (that larger models and more diverse datasets are better) are not unexpected, such detailed empirical studies are worthwhile for practitioners and researchers alike. However I think the paper falls short by not going one step further and providing additional insight to try and explain some of its (many) experimental findings. This makes it harder to tell whether and how these results would transfer to other architectures, modalities or datasets considered here, which ultimately limits the impact of the paper. I think the paper is in a good shape, but it would need additional ablations and analysis to warrant acceptance at ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically studies simple techniques to improve out-of-domain (OOD) generalization. First, they show that fine-tuning a pretrained model significantly improves OOD accuracy over the non-fine-tuned baseline. Then, they show that fine-tuning with smaller learning rates outperforms larger ones on OOD data, even though both get approximately identical in-domain (ID) accuracy. Finally, the conclude that larger models, larger datasets, or (preferably) both improve OOD generalization.
main_review: On the methodology side, the paper is thorough, well-executed, with (mostly) clear results and well-designed experiments. Unfortunately, I think it lacks a strong selling point as most of the proposed insights are not novel enough.

For example, it is well-known that fine-tuning a pretrained model improves performance on downstream tasks, especially when those downstream tasks differ from upstream ones. See, for example, the work of Li et al. (2020) for an in-depth study on pretrained vision models, and Dodge et al. (2020) on language models. Similarly, the conclusion that larger models and datasets improve transfer is also well-studied — see the literature around "neural scaling laws", e.g., Kaplan et al. (2020) or Zhai et al. (2021) and references therein.

Maybe the most interesting contribution is the insight that smaller learning rates improve OOD accuracy even thought they perform on par with larger learning rates in ID data. But here the results are muddied, since this seems to be true for some settings but not all (p. 5, 1st paragraph). Could the authors expand on why ImageNet-pretrained models benefit from larger learning rates but not other models? What is their intuition for when practitioners should choose larger or smaller learning rates for OOD fine-tuning?

References:

- Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto, "Rethinking the Hyperparameters for Fine-tuning", 2020.
- Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith, "Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping", 2020.
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, "Scaling Laws for Neural Language Models", 2020.
- Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, "Scaling Vision Transformers", 2021.
summary_of_the_review: Although the experimental study of the paper is well executed, it provides few novel insights.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper empirically studies pre-trained model on OOD generalization. Specifically, the authors consider four datasets in this paper: PACS, Office-Home, VLCS and TerraIncognita. Based on the experimental results, the authors concludes three main observations . First,  larger models and larger datasets need to be simultaneously leveraged to improve OOD performance. Second, using smaller learning rates during fine-tuning is critical to achieving good results. Third, the strategies that improve in-distribution accuracy may lead to poor OOD performance. 
main_review: This paper provides extensive experimental results on four datasets to show how pre-trained data, pre-training model, learning rate and some ID tricks affect the OOD accuracy. Overall, the paper is easy to read and the organization is also good. The considered topic, OOD generalization, is interesting and important to the deep learning community. However, I have found some of the results unclear or incorrect. 

1. The claim "models trained with smaller learning rates achieve much better OOD generalization" may be misleading/incorrect. It seems to me that the authors want to claim that the smaller lr, the better performance. This result is not well-supported. For example, Figure 2(h) shows that the performance with lr$=10^{-4}$ is worse than that with lr$=10^{-3}$; Figure 2 (e) and (f) show that the result for lr$=10^{-3}$ and the result for lr$=10^{-4}$ are almost same. Although the authors provide experiments with a range of learning rates between 0.05 and 0.0001, more experiments with smaller learning rates such as $10^{-5}$, $10^{-6}$ and $10^{-7}$ are needed. 

2. The claim "methods listed in Table 2 does not significantly improve the OOD generalization across four datasets" is incorrect. We can see from Table 2 that both Label Smoothing (0.2) and SAM (0.02) are better than ERM (in Table 1) on 3 (of 4) datasets, while PatchGaussian (0.5) is better on 2 (of 4) datasets. 

Thus, if above two issues cannot be addressed, then the main claims about learning rate and ID techniques are not correct/well-supported, which makes the contribution of this paper limited. 

The novelty of using pre-training model to improve OOD generalization is not significant and this idea is not new. For example, see the ICML 2021 paper "Improved OOD Generalization via Adversarial Training and Pre-training". Although the authors show that the larger models and larger datasets need to be used simultaneously, but the contribution is incremental since the idea is too straightforward given the ICML paper. Furthermore, the intuition of using larger models and larger datasets simultaneously is unclear. It would be better if the authors could add some sentences to describe the intuition. 

The contribution about ID tricks in this paper is mazy. It seems the paper's topic is about pre-training and fine-tuning on OOD generalization, which is not directly related to ID tricks. I am confused about adding ID tricks in this paper. 

Two related papers are not discussed:

a. In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness, ICLR 2021

b. Improved OOD Generalization via Adversarial Training and Pre-training, ICML 2021
summary_of_the_review: 1. The topic is important and interesting.

2. Two main contributions about learning rate and ID tricks may be incorrect or not well-supported.

3. The idea of using pre-training model is not new and the novelty of this paper is limited.

4. Unnecessary ID tricks and missed related work.
review_title: null

Meta-review: The paper presented an empirical study of pre-trained models on the Out-of-distribution Generalization problem. 
Authors evaluated various factors (such as model sizes, datasets, learning rate, etc) and claim some major findings:  1) larger models have better OOD generalization, and combining both larger models and larger datasets is critical; 2) smaller learning rate during fine-tuning is critical; 3) strategies improving in-distribution accuracy may hurt OOD. Overall, this paper is a well-written empirical study with some useful insights, but the new findings from the empirical studies are generally not surprising and the overall contribution is not significant enough for acceptance.
Decision: Reject