text: 362,158,1499097939435667457,1010536067886387200,Ananya Kumar,"How should you fine-tune a large pretrained model (CLIP, SimCLR) robustly? We find that standard fine-tuning can do poorly out-of-distribution (test data ≠ fine-tuning data). Our analysis leads to a simple fix, higher accuracy on 10 datasets. <LINK> (ICLR Oral) <LINK> (2/n) Joint work with Aditi Raghunathan, @rmjones96, and my advisors @tengyuma and @percyliang (3/n) We find that full fine-tuning (updating all model parameters) can be worse than linear probing (updating only the last layer) on out-of-distribution test examples, when the distribution shift is large and the pretrained features are good (4/n) We prove theoretically that this phenomenon arises even in simple and natural settings. One line explanation: while full fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features (5/n) This suggests the easy two-step strategy of linear probing then full fine-tuning (LP-FT). Intuition: head doesn't change as much, so features get distorted less (6/n) LP-FT gives large gains OOD: 10% better OOD, 1% better ID than full fine-tuning. Also outperforms linear probing both ID and OOD (7/n) Caption for Figure in Tweet 1/n: (a) full fine-tuning does better in-distribution (ID), (b) linear probing can do better out-of-distribution (OOD), (c) LP-FT does better on both, especially OOD (8/n) This work is part of a broader trend (e.g., prompt tuning, composed fine-tuning, prefix tuning), where tuning a small part of a pretrained model can be better than full fine-tuning, especially for robustness @CyrusMaher Yup! And to clarify we cited this and other papers, and mention in our abstract + intro that LP-FT is sometimes used as a fine-tuning heuristic (though not for robustness). Hopefully our analysis popularizes it, and explains when it can be particularly useful (OOD)",http://arxiv.org/abs/2202.10054,"When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer -- the ""head""). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head -- this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning). ","Fine-Tuning can Distort Pretrained Features and Underperform