Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper discovers an interesting behavior of model fine-tuning: the performance is worse compared to linear probing on OOD data (i.e., data from other domains), especially when the distribution shift between inner distribution and out of distribution are big. The explanation provided in the paper is that fine-tuning distorts the feature representations, overfits on inner distributions, and thus has a higher error on OOD data. The authors also provide a simple solution to this issue by fine-tuning with a classification head initialized from linear probing and had better results in all the benchmarks they have in the paper. 
main_review: The strength of this paper is the extensive and detailed toy and benchmark experiments. The reasoning and intuition of why fine-tuning underperforms on OOD data are well explained and discussed throughout the paper. The suggested solution by combining linear probing and fine-tuning also has good performance. 
summary_of_the_review: Although the solution proposed in this paper is not fancy, the reasoning, intuition, and experiments are well written.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the problem of how to fine-tune a pre-trained model and obtain better results for both ID and OOD. Two methods, fine-tuning and linear probing, are investigated and compared, then a new two-step variant called LP-FT is derived. Results further verify that LP-FT obtains the best performance for ID and OOD tests compared with FT and LP.
main_review: Pros:

1. the observation that fine-tuning performance worse on the OOD test is new and interesting 

2. the proposed method LP-FT is simple yet effective

3. theoretically analysis is further provided to show why LP-FT works

Cons:

1. some important baselines are missing like [a-c], which should be discussed and compared

[a]. Guo, Yunhui, et al. "Spottune: transfer learning through adaptive fine-tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[b]. Zhang, Jeffrey O., et al. "Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020.

[c]. Ge, Weifeng, and Yizhou Yu. "Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

2. why different pre-trained models are utilized for different ID and OOD pairs? In fact, the authors could use different models for each ID and OOD pair.
summary_of_the_review: This paper discovers that vanilla fine-tuning performs worse than linear probing for the OOD tests and then develops a new method combining these two techniques sequentially. Results on several datasets verify its effectiveness. Even there exists some minor problems, this paper is interesting and easy to read. Thus, I tend to give the "weak accept" score.

-----POST REBUTTAL-----

The authors have addressed my concerns. Thus, I increase my score from 6 to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores how different strategies for fine-tuning affect in- and out-of-distribution performance. The authors contrast linear probing (updating only the parameters of the final linear layer), end-to-end fine-tuning (updating all parameters of the model) and a two-stage approach, where linear probing is followed by end-to-end fine-tuning. While end-to-end fine-tuning typically improves in-distribution performance, the authors show that it can also underperform linear probing out-of-distribution. The paper theoretically analyzes the tradeoffs in a simplified scenario with two-layer networks, finding that end-to-end fine-tuning can "distort" pre-trained features. Their experiments on a number of datasets including CIFAR, WILDS-FMoW and others, confirm the intuitions from their theory. The proposed mitigation strategy, a two-stage fine-tuning approach where end-to-end fine-tuning follows linear probing is found to be beneficial, especially out-of-distribution.


**Update:** The authors addressed most of the concerns raised by this and other reviews, and I am raising my score accordingly.
main_review: **Strengths:**

1. This paper explores two increasingly impactful research directions, fine-tuning pre-trained models and generalization under distribution shifts. I believe their results, both theoretical and empirical, would be of interest to many in the community

2. The theoretical results, despite studying a very simple setting that is unlikely to be used in any real experiments, offer intuitions that transferred well to practical results.

3. The paper is clear and well written.

**Weaknesses:**

1. Experiments could be more comprehensive. For instance, while this paper analyses several distribution shifts (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR->STL, CIFAR 10.1 and FMoW), it was surprising that distribution shifts like ImageNetV2, ImageNet-R, ImageNet-A, ObjectNet and ImageNet Sketch were not considered by this work. Moreover, only a single architecture, ResNet-50, is used for the experiments. As they stand, it is unclear whether the results from this paper would hold at larger scales.

2. This work overlooks simple baselines. For instance, given the finding that end-to-end fine-tuning significantly changes the weights of the backbone (which arguably hurts OOD performance), it would be natural to consider having a smaller learning rate for the backbone, and a higher one for the untrained final linear layer. It is not uncommon to do so in practice. Moreover, authors could have explored regularizing the difference of the weights of the backbone to the original weights, encouraging them to not be changed too much. Finally, the authors do not mention the fact that models like CLIP allow the possibility of starting with a good set of initial weights for the final layer, as they can be used in a zero-shot setting. If the intuitions presented in this paper hold, this prevent the "distortion" that happens when the last layer is far from the optimum.

3. The theoretical results are disconnected from realistic settings. Some clear examples are the assumption of a two-layer network, a squared error loss (while the de facto standard for classification is cross-entropy), and considering the worst case loss over distributions of bounded norm.

4. Apart from the theoretical results, there is not a lot of novelty introduced by this work. The two-stage fine-tuning strategy where end-to-end fine-tuning follows linear probing is commonplace and thought in introductory courses in deep learning, e.g. in the first lesson of https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson1-pets.ipynb

5. Comparison to previous work is lacking. In particular, explicit comparisons of LP-FT with other recent robustness-oriented fine-tuning methods would greatly strengthen this work [1-4, among others]. 


**References:**

[1] Wortsman, Mitchell, et al. "Robust fine-tuning of zero-shot models." arXiv preprint arXiv:2109.01903 (2021).

[2] Aghajanyan, Armen, et al. "Better fine-tuning by reducing representational collapse." arXiv preprint arXiv:2008.03156 (2020).

[3] Jiang, Haoming, et al. "Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization." arXiv preprint arXiv:1911.03437 (2019).

[4] Zhu, Chen, et al. "Freelb: Enhanced adversarial training for natural language understanding." arXiv preprint arXiv:1909.11764 (2019).
summary_of_the_review: Overall, while there are several points of concern that could strengthen the paper, I believe the results and theory presented by this work would still be of interest to many in the community, so I recommend its acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper contrasts fine-tuning (i.e., modifying all network weights) and linear probing based on their relative ID/OOD performance. It is known that fine-tuning (FT) outperforms linear probing (LP) ID. This paper presents that the reverse is true OOD (FT outperforms LP). This paper suggests that this occurs because fine-tuning distorts features in conjunction with the final linear layer. Instead, if a final linear layer is trained first, the features do not have to move that much during full fine-tuning. The authors refer to this method as LP-FT and show it often outperforms LP and FT both OOD and ID.
main_review: This paper has many strengths:
- It is not well known that FT often outperforms LP OOD -- the empirical and theoretical study of this phenomenon may be very important to the community.
- LP-FT outperforms LP and FT on many distribution shifts.

In addition, the paper has some weaknesses with associated actionable items:

1) The main weakness of this paper was contextualizing LP-FT. In the abstract and introduction, it seems as though LP-FT is a method that is being introduced by this paper. For instance, the abstract states "our analysis suggests the simple two step-strategy of linear probing then full fine-tuning". It is not until the final page of the main paper that the related work section states "LP-FT has sometimes been used as a fine-tuning heuristic". Moreover, this is presented without any citations. I would very much appreciate clarity on this issue, is LP-FT something that has been used in the past? If so, by which papers and why?  Does this reference that in fine-tuning, sometimes the hyperparameters in the final layer are decoupled from those used for the encoder [1] and performance improves? As shown by [2, 3], performance improvement ID lead to performance improvement OOD, and so this could be an alternative explanation for  the OOD boost of LP-FT that is orthogonal to the authors theory. For instance, in the effective robustness framework of [2], do the solutions found by LP-FT exhibit more effective robustness than the FT solutions?

2) Section 4 is hard to follow in its current form as it is not clear exactly from where numbers are derived. For instance in the first line of the  "Results." paragraph of 4.2, where is the 76.5 number from?

3) One of the networks studied by this paper is CLIP, for which a zero-shot final layer can be constructed (e.g., the final linear layer does not need to be constructed from scratch before fine-tuning). One possible addition to the paper could be discussion of this setting, and in particular if LP-FT is still required.

4) There are a few claims in the paper which could benefit from additional support. In particular, in the conclusion it is stated that the "gap between FT and LP grows as the quality of pretrained features improve". Are the authors referencing the single MOCO-v1 vs. MOCO-v2 experiment or is there additional support for this claim? Are the authors referring to absolute or relative difference between FT and LP? I am wondering as this claim seems a bit counterintuitive, if the features are already good shouldn't LP be sufficient?

5) The empirical verification of the theory (Section 4.3) is very interesting and could be more thorough. In particular, are there associated error bars for the 0.019 and 0.017 numbers as this seems quite close to conclude that one is larger. This euclidean distance experiment is very interesting, why is it only conducted for one distribution shift? Moreover, what happens when analyzing cosine distance instead of cosine distance?

[1] https://arxiv.org/pdf/2106.04560.pdf
[2] https://arxiv.org/abs/2007.00644
[3] https://arxiv.org/abs/2107.04649
summary_of_the_review: This paper is very interesting and will be an important contribution if concerns are properly addressed. An essential concern is the contextualization of the method LP-FT --- is this method (or a modification) something that people have explored previously and if so in what context. In the current framing of the abstract and introduction, LP-FT seems to be introduced by this paper. In addition, this papers empirical verification of the theory (Sec 4.3) appears very promising but could benefit from additional detail and experiments (e.g., more than one distribution shift in the euclidean distance experiments).

Edit: authors have addressed many concerns and I have changed my score to 6.
review_title: null

Meta-review: The paper provides a solid and thorough analysis to the two basic methods of fine-tuning, linear probing (LP) and fine-tuning (FT). The authors provide an important and highly interesting observation about the performance of both in and out of domain (OOD) setting. They validate the known phenomena that FT outperforms LP in the in-domain (ID) setting, but demonstrate that when tested on OOD data, LP is in fact more performant and back this observation with a theoretical and empirical analysis.
The remedy provided is also a known, yet slightly less popular technique of setting the final layer (LP) first, then finte-tuning (FT-LP). The authors provide thorough experiments showing that this technique enjoys the best of both worlds, meaning ID and OOD. I found it worth noting that during the rebuttal period the authors provided experiments on additional larger scale datasets and models and the results of the paper carried over to these new setting.
The reviews agree that the analysis provided is both interesting and novel. Even though the paper does not provide a new technique, there is a consensus that the understanding it gives on known techniques is a welcome addition to ICLR.
Decision: Accept (Oral)