text:   Out-of-Distribution",9,"['How should you fine-tune a large pretrained model (CLIP, SimCLR) robustly? We find that standard fine-tuning can do poorly out-of-distribution (test data ≠ fine-tuning data). Our analysis leads to a simple fix, higher accuracy on 10 datasets. <LINK> (ICLR Oral) <LINK>', '(2/n) Joint work with Aditi Raghunathan, @rmjones96, and my advisors @tengyuma and @percyliang', '(3/n) We find that full fine-tuning (updating all model parameters) can be worse than linear probing (updating only the last layer) on out-of-distribution test examples, when the distribution shift is large and the pretrained features are good', '(4/n) We prove theoretically that this phenomenon arises even in simple and natural settings. One line explanation: while full fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features', ""(5/n) This suggests the easy two-step strategy of linear probing then full fine-tuning (LP-FT). Intuition: head doesn't change as much, so features get distorted less"", '(6/n) LP-FT gives large gains OOD: 10% better OOD, 1% better ID than full fine-tuning. Also outperforms linear probing both ID and OOD', '(7/n) Caption for Figure in Tweet 1/n: (a) full fine-tuning does better in-distribution (ID), (b) linear probing can do better out-of-distribution (OOD), (c) LP-FT does better on both, especially OOD', '(8/n) This work is part of a broader trend (e.g., prompt tuning, composed fine-tuning, prefix tuning), where tuning a small part of a pretrained model can be better than full fine-tuning, especially for robustness', '@CyrusMaher Yup! And to clarify we cited this and other papers, and mention in our abstract + intro that LP-FT is sometimes used as a fine-tuning heuristic (though not for robustness). Hopefully our analysis popularizes it, and explains when it can be particularly useful (OOD)']",22,02,1854