Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Authors: Ananya Kumar
Aditi Raghunathan
Robbie Matthew Jones
Tengyu Ma
Percy Liang
Venue: ICLR 2022 Oral (2022)
Keywords: fine-tuning theory
transfer learning theory
fine-tuning
distribution shift
implicit regularization

Abstract: When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer---the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (BREEDS-Living17, BREEDS-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR-10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head---this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning).

Published as a conference paper at ICLR 2022
FININGUNING CAN DISTORT PRETRAINED FEATURES AND UNDERPERFORM OUT-OF-DISTRIBUTIO
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, Percy Liang Stanford University, Computer Science Department
ABSTRACT
When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updat- ing only the last linear taeer-the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, find that fine- tuning can achieve worse accuracy than linear probing aut-of-distriuution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (BREEDS--iving 17, BREEDS-Ennityyn DomainNet, CIFAR - STL,, CIFAR-10.1, FMoW, ImageNerV2, ImageNet-R, ImageNet-A, ImageNet-Sketch) fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between II) and OOD accuracy arises even in simple setting: fine-tuning overparameterizee two-layer linear networks. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine- finettuing and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning tuenttnnng).......
INTRODUCTION
Pretraining a model on large dataset before transferring to a downstream task's training data substantially improves accuracy over training from scratch-for example, pretraining a ResNet-50 on unlabeled ImageNet boosts accuracy on CIFAR-10 from 94% to 98% (Chen el all 2020at)).
High-stakes applications such as poverty mapping in inder-resourced countries (Jean et all 2016), self-driving cars (Yo all 202T), and medical diagnosis (AlBadawy al, 2018), require models that also generalize to circumstances not seen in the training distribution. addition to testing on data drawn from the downstream task's 's training distribution (in-distribution; ID), it is increasingly important to test on data distributions unseen during training (out-of distribbiit OOD).
After initializing with a pretrained model, two popular transfer methods are fine- tine-tuning (running gradient descent all the model parameters), and linear probing (tuning the head but freezing lower layers). In the ID setting it is well known that fine-tuning leads to better accuracy than linear probing (Kornbluth all MITS; Zhar et all 2020; He all 2020), and even when testing OOD, prior work usually fine-tune fine-tunes all parameters of their model (Hendrycks et alj, 2019;; Miller et 2021: Andreassen etall, (202I). Intuitively, fine-tuning all layers of network can improve pretrained features by adapting them to the specific task, while linear probing freezes these features.
In this work, we investigate the OOD accuracy of fine-tuning and linear probing and find that surprisingly, fine-tuning can do worse than linear probing the presence of a large distribution shift. We experiment on ten distribution shift benchmarks (BREEDS Living17, BREEDS Entity30, Do- mainNet, CIFAR -> STL, CIFAR1O.1 FMoW Geo-shift, ImageNetV2, ImageNet-R, ImageNet-A, Imageh)ttS initializing with good pretrained features from MoCo-v2 (Chen (T200) and CLIP (Radtord at 2021). While both methods offer gains over training from scratch, fine-tuning improves the average ID accuracy relative to linear probing from 83% to 85% but brings down the OOD accuracy from 66% to 59% (Figure M..
When and why does fine-tuning underperform linear probing? We theoretically consider fine-tuning two-layer linear network in overparameterized regression setting where the feature extractor layer has been pretrained to map high- high-dimensional inputs to useful, lower-dimensional, features. We prove that fine-tuning is worse than linear probing on directions outside the span of the training data when using "good" pretrained features. Even with an infinitesimally small learning rate, fine-tuning distorts pretrained features- the features of ID training data updated while those of OOD data

---

Published as a conference paper at ICLR 2022
Pretraining
(a) Fine-tuning
(b) Linear probing
Features
Ramdomly infiaitzed
Backorop
Randomly head
Backorop
Infiaize Iweed
Backprop
Piputt
ID test
85.1 1%
82.9%
85.7%
OOD test
59.3%
66.2%
68.9%
Average accuracies (10 distribution shilts)
Given a good feature extractor (top-left), a randomly initialized head is added to map features to outputs and we can (a) fine-tune the model parameters or (b) linear probe, which freezes the feature extractor and trains only the head. We run experiments on ten distribution shifts. Fine-tuning does well when the test example sampled from the fine-tuning distribution (ID), but underperform on test examples sampled from OOD distributions (when the distribution shift is large). (c) Our theory indicates that fine-tuning can distort the pretrained feature extractor and lead to poor OOD accuracy, but initializing with linear probed head can fix this empirically LP-FT gets better accuracies both ID and OOD.
Figure 1:
change less. Since the head and feature extractor are simultaneously optimized during fine-tuning to a configuration that works well on ID training data, the head only accomodates the distorted features of ID points and performs poorly (relative to linear probing) the less changed features of OOD points. Interestingly, we show that this feature distortion issue cannot be simply fixed by early stopping- througout the entire process of fine-tuning, do well OOD (relative to linear probing). On the other hand, given "good" features, linear probing extrapolates better OOD because it preserves pretrained features, but does worse than fine-tuning ID because linear probing cannot adapt the features to the downstream task.
never pass through parameters that
Technical challenges. Existing theoretical work on transfer learning focuses on linear probing (Wi cLal 2020: Iripuranenı et al) 2020; Ductaal 2020). In contrast, analyses of fine-tuning is scarce and challenging because it requires understanding the training dynamics, instead of only the loss function and its global minimizers. In fact, fine-tum tinettunng and training from scratch optimize the same training loss and only differ in their initializations (pretrained random). A mathematical analysis that distinguishes them needs to capture properties of the different minima that these algorithms converge the implicit regularization effect of initialization (Neyshabur et al, 2014). Accordingly, our analysis reasons about the parameters that gradient methods pass through starting from the pretrained initialization, which is challenging be- cause this is a non-convex optimization problem and there is no known closed form for this trajectory. Two-layer linear networks are widely studied in the literature on implicit regularization (Saxe all, 2014 Gunasekan kar et all 2017: Gidel et all 201% Arora et all, 2018). However, they analyze random and often small initializations, which don't capture pretraining.
to, phenomenon that is sometimes theoretically referred
Algorithmic implications. Our theory shows that fine-tuning underpeforms because when trying to fit ID training data with a randomly initialized head, the feature extractor changes significantly for ID examples, making features for ID and OOD examples largely inconsistent. This can be fixed by initializing with good head that does need to be updated much during fine-tuning, reducing how much the feature extractor changes. This suggests a simple two-step strategy of first linear probing to find a good head and then full fine-tuning (LP-FT). Empirically, LP-FT outperforms fine-tuning and linear probing, both ID and OOD. Even on CIFAR- 10.1 (small distribution shift), where fine-tuning better for both ID and OOD, we find LP-FT outperforms fine-tuning on both metrics. L.PFT and vanilla fine-tuning use similar amounts of compute because the first step of linear probing is relatively very cheap. Prior work has used LP-FT (Levine et 2016; Kanavali Tsuncki, 2021) (or variants such layerwise fine-tuning (Howard & Ruder, 2013 larger learning rates for the head layer (Prabhu et all 202T) however it has not been used for robustness OOD accuracy,
and we show that it addresses the ID-OOD tradeoff theoretically and empirically. Note that LP-FT not meant to be a SOTA method but rather a simple, principled way to get good ID and OOD accuracy- we hope analysis inspires even better methods for robust fine-tuning.
2

---

Published as a conference paper at ICLR 2022
Empirical validation. Finally, we find that fine-tuning fails and LP-FT works, for the reasons predicted by our feature distortion theory: (1) tine-tuning changes the features for ID examples more than for OOD examples, leading to distortions; (2) LP-FT indeed changes both ID and OOD features 10 - 100x less than fine-tuning does; (3) LP-FT gets the best of both worlds, achieving better accuracies than fine-tuning and linear probing both ID and OOD (Figure
2 SETUP
Task and evaluation. Given training examples sampled from some distribution Pia, our goal is to learn a predictor f R2 + 3/ to map inputs I E R to outputs E V. We evaluate predictors on their standard "in-distribution" (ID) performance Lid new test samples drawn from Pa that the training data is also sampled from. We also evaluate classifiers on their "out-of-distribtioon" (OOD) performance Loed test samples drawn from new distribution Pood that is different from Pia- Formally, for some loss function Ê, we evaluate classifiers on:
Lid(f)=
[(((()).)) and Lood(f)= (a.
E
E [(((())))) (a.
(2.1)
Models. In this work, we focus on predictors that leverage pretrained representations. We parameter- ize the final predictor as follows: given features gæ (x) & for some feature extractor parameters B E B, and linear "head" E V, we have fc. = 9B (1)) In our experiments (Section B), JB a deep network and in our theory (Section B). JB a linear projection.
We assume access to some initial pretrained feature extractor Bo that is obtained by training on potentially large amounts of data from a distribution that contains unlabeled or weakly supervised inputs from Pid and Pood- We focus two popular methods to learn a predictor fv, given training data from Pa: (i) linear probing where = Bo and the linear head is obtained by minimizing some loss (e.g.. logistic loss for classification, squared loss for regression) on the training data, and (ii) fine-tuning where both e and B are updated by performing gradient descent on some loss on the training data with B initialized Bo-
THEORY: TUNI-NUNING DISTORTS PRETRAINED FEATURES
Our goal is to understand under what conditions fine-tuning does worse than linear probing cut-of-distribution (OOD). We consider linear setting (feature extractor Jn is linear) where the pretrained features are "good" and the OOD shift large (Section B.J)). We prove our main result: that fine-tuning. in which all model parameters
updated, distorts features and gets suboptimal OOD error (Section 6 Theorem 3.2).. D. We use this result to show that linear probing gets better OOD error but worse ID error than fine-tuning (Section (.3) Finally, explain why linear probing then fine-tuning can mitigate this ID-OOD tradeoff (Section (.4))
Our analysis handles two key challenges which distinguishes it from prior work on transfer learning in linear models (Wi all 2020 Tripuraneni et all, 2020; Du el all, 2020; Xie el all, (02EF. Prior work focuses linear probing, while we study fine-tuning where the resulting optimization problem is non-convex. We also study overparameterized models where the training loss alone does not determine test performance- this captures the fact that both training neural networks from scratch and fine-tuning them have the same training loss but very different test performance. However, it also makes the analysis challenging because we need to reason about the trajectory of gradient methods starting from pretrained initialization, which has no known closed form.
3.1 LINEAR
SETTING
For our analysis, we focus on regression, where y R and ( is the squared loss.
Models. Recall from Section that we parameterize predictors terms of the feature extractor and head parameters. In this section, study models where the feature extractor is linear, i.c. fe,B(x)=o Ba where B B and RR.
Good pretrained features. For simplicity, we assume the models are well-specified i... y=v_ B. where E = and B. E Rkxd Note that B, and v, only unique up rotations, i.c., for any rotation matrix U, (Uv,)" (U B B,x. x. As in prior work (Inpuranen cf all, 2020) suppose B,
!Our main contribution, analysis of fine-tuning (Theorem (2)). does not require well-specification. We compare FT with I.P by adapting carlier work lincar probing which requires well-speciffcatioo.
3

---

Published as a conference paper at ICLR 2022
and By have been orthogonalized to have orthonormal rows. Suppose we have a pretrained feature where the distance d is defined (where the min is over
extractor By close to B., so d( B.) I
rotation matrices U E R
d (B, "in"")=inn|BBUII)
(3.1)
Training data. Let X XERxxd, + be matrix encoding n training examples from Pia where each of the n rows training input. Let Y E R be the corresponding outputs. Let S = rowspace( (X)) be the m-dimensional subspace spanning the training examples. We consider an overparameterizze setting where 1 S m < dd-k. Intuitively, the input dimension is high (e.g,, 10K), feature dimension k is lower (e.g., 100) and m is in the middle (e.g.. 5K).
Large OOD shift. We assume that the OOD data contains examples outside the span of the training data. Formally, let Pood have second moment Σ B[xx" | where z~~ for invertible Σ.
Training methods. Given training data and a pretrained feature extractor Bo. we study the two popular methods of linear probing (LP) and fine-tuning (FT) to learn the final predictor. Both methods involve optimizing the training loss via gradient descent (or variants). In order to effectively analyze these gradient based algorithms, we study vanishing step sizes leading to gradient flows. Gradient flows can be thought of as a continuous time analogue of gradient based methods and have been extensively studied in recent years way to understand gradient based methods (Cunasekar et all, 2017: Arora all 2018; ell 20T8). Formally, for training loss L(v, B) = |X T YI the gradient flow differential equations for LP and FT are as follows:
0,un (t) - V (vx (t)),, (t) VV
(3.2)
0/ = - v -.. (t), Bo). (p)
(3.3)
initialized with (+) = (0) = Bn and in (0) = (0) = tg- In practice, the head parameter Up initialized randomly-out results hold for any standard random initialization (Glorot & Bengio, 2010, for example tip ~ (0, for any o², or zero initialization where up = 0. Recall that the initial value of the feature extractor Bo is obtained via pretraining.
The final LP and FT solutions are the limit points of the corresponding gradient flows:
lim Ust (t) and B t+050
lim (t),
(3.4)
t-0n0
Up
lim Upp(1) and B lim (1)= By-
(3.5)
10000
t
3.2 FINE-TUNING DISTORTS PRETRAINED FEATURES
The common method of using pretrained feature extractor is fine-tuning (FT) which typically improves ID) performance relative to linear probing (L.P). In this section, we show that FT can distort features leading to poor OOD performance. We first explain the key intuitions and then present formal theorem lower bounding the OOD error of FT (Section (222))
3.2. KEY INTUITIONS
We use two main observations to characterize when and why FT has higher OOD error than LP.
1. Features get distorted: representations change only in the ID subspace (i.e., subspace spanned by the training data) and are unchanged in the orthogonal subspace. To see this, we take the derivative of the training loss [vv, B) ||X TT,
2 with respect to the feature extractor parameter B:
V
(3.6)
By definition, if is a direction orthogonal the training subspace S rowspace(X). then VBL(v, B)u = 0, that is the gradient updates to do not modify Bu for a E S+. However, the gradient is non-zero for directions in the ID subspace and the corresponding features Bu change across the fine-tuning process. We call this feature distortion: the features in some directions are changed but not others. Next, explain why this can lead to high OOD error.
Distorted features can lead to higher OOD error. Consider a toy example (Figure 2 where d 2 and the dimensionality of the representations k = 1. The linear head e is scalar quantity that denotes how much the feature extractor B has to be scaled by. Suppose the ID-susspace the r-axis. There are different ways of fitting the IDD subspace depending on the feature extractors B as
4

---

Published as a conference paper at ICLR 2022
00D
W.
OCD
Wip
Inner probing
ODD Error of fine-luning
Pip
Wip = B% Uhp
distortion
Win
Bg
Ryc
ID
correct ID
(a) example (Linear probing)
(b) example (fine-tuning)
Figure 2: A toy version of our theory illustrating why fine-tuning distorts features, with inputs in 2D. Given input æ, the ground truth output y- w. x. The ID data along the æ-axis and the pretrained feature extractor is Bp. (a) Linear probing learns Wip, a scaling of the pretrained feature extractor that gets the ID data correct (Wip and w, have the same z coordinate as indicated by the vertical dotted line). (b) Fine-tuming updates the pretrained feature extractor along the ID data (so horizontally) to get BR, and then learns a scaling of these features that gets the ID data correct. While both methods get II) data correct, fine-tuning makes large errors perpendicular to the IDD data, because fine-tuning updates Bo along the ID direction but not the perpendicular direction.
shown in the foture-both fine-tuned and linear probed estimators match the true parameter in the ID subspace (since Wh, w. have the same projection on the z-axiss. If the feature extractor were optimal scaled versions of the optimal, good performance on the ID subspace would translate to good performance everywhere, even in directions orthogonal to the ID subspace. However, in FT, the features change only for inputs in the II) subspace (see (1)) and thus the updated features not simply scaled but distorted. In Figure A this corresponds to the feature extractor Bo changing along the ær-axis. In this case even if the ID error is low, error in directions orthogonal to the IID subspace can be high, leading to high OOD error.
The only way the pretrained features are not distorted and only scaled during FT if the initial feature extractor Bo is exactly aligned with the ID subspace. In Figure D1 if Bp is along the æ-axis (the ID subspace), then updating the features exclusively along the æ-axis would simply scale the initial features. In this case linear probing and fine-tuning will have identical behavior. However, if the angle between Bo and the r-axis is non zeroo the updates would lead to distortions. In high dimensions, we measure the alignment between Bo and the ID subspace with the largest principal angle:
Definition 3.1 (largest principal angle). Let A and B be arbitrary subspaces, and E and F be matrices with orthonormal columns that span A and B respectively, with r min(dim(()) di))))))))B)) Then coso (A,B) = (E F), which is the r-th largest singular value of E'
3.2.2 GENERAL RESULT ON THE OOD ERROR OF FINE-TUNING
Our main theorem lower bounds the OOD emror of fine-tuning outside the span of the training data.
Theorem 3.2. In the overparameterized linear setting, let S = rowspace )))
Ro rowspace Bo), and B. be the optimal parameters with w. B. max (Ro,S s > 0, then for all time stepst, t, the OOD error of the fine-tunine iterates (Br,(t), Un. (t)) is lower bounded:
cosO max R
min( /11..112) -e).
Lood (v=((()) (()) Vamin(D (E)
(3.7)
Vi
(1 l|
where =2==(10 v.) =2 - v.) is defined to be inital head alignment error and e 2 d( Bo, B.) is the error in the pretrained feature extractor.
Proof sketch. Since the features do not change for examples in S (perpendicular to the training data), we show that in order to achieve low error on 5 the linear head (+(() would have to become very similar to the optimal v. some time f. The head initialization uo is random (or zero) and likely to be far from v. (measured by the alignment error (P), so the head would have to change lot to get close to v. As see from the fine-tuning gradient flow (GM2), (i (t) and (t) change in a "coupled" manner, and a "balancedness" invariant in Du el all (2018) holds across the fine-tuning trajectory. Correspondingly, if v4 (t) changes lot and gets close u4, the features (t) also change a lot for
5

---

Published as a conference paper at ICLR 2022
examples in S-wee show that this would lead to high error on examples in S. Either way, fine-tuning would get some subspace (S S !) of examples wrong, leading to high OOD error. The full proof appears in Appendix
Interpretations of various quantities. Quality of pretrained features (e). To unpack the bound
consider a special case where the pretrained features are perfect (c = 0). With perfect features, Proposition 2 shows that linear probing gets zero OOD error. Theorem ? shows that Lood Un (f) (t)) > at all times toso fine-tuning underperforms when the features are perfect.
Alignment ervor of random head imitialization (4²). The lower bound (Equation A. increases as 4² increases, because the gradient updates to the head and feature extractor are coupled. If the head were somehow initialized perfectly at v., fine-tuning updates may not increase the OOD error. However, when the head is randomly initialized as is standard in fine-tuning, the alignment error is high, leading to high OOD error. We use this insight in Section 6.4 to show that better head initialization (via linear probing) improves OOD performance of fine-tuning.
3.3 LINEAR PROBING VS. FINE-TUNING
In this section, we use our main theorem on fine-tuning (Theorem (22) and adapt prior work on linear probing to show that linear probing is better than fine-tuning OOD, but worse ID), when the II) distribution has density on lower m < d dimensional subspace S, and Bo is close to B..
Assumption 3.3 (ID subspace assumption). We assume that the ID data lies on an m-dimensional subspace S where k < < d - ki and we have 2 training examples. Formally, let Pa be distribution on R which has density, and let the columns of F E Rdxm form an orthonormal basis for S. Then Pi has the distribution of Fz where 2 2~
Recall that the ID error is the expected mean- mean-squared error over the ID distribution Pa::
L)
E am Pdd
[(v, B.r-v' (xx)²
(3.8)
OOD comparison: Under mild non-degeneracy conditions, we show that as the feature extractor error e goes to 0, linear probing does much better than fine-tuning OOD: the ratio of the losses goes to 0. The non-degeneracy conditions are similar to Section B.2--we require that the training data cannot be exactly in the same direction or orthogonal to the pretrained features, formally that cas (R.,,) and cos (R,,S* are not 0 where R, = rowspace(B..) (B.).
Theorem 3.4 (Informal version of Theorem A. In the linear overparameterizeed setting under the where
ID subspace assumption (Assumption B.3). if cos c S) o 0 and cos (R., S o R. = rowspacel B,), then,
Toood ( (B)) 0,ss B..
(3.9)
Lood( un (n())))))
This holds for all times t for FT (and therefore also for the limit of BB to upp ,Bo as a result of the gradient flow on a convex problem.
) and the LP iterates converge
Intuitively, if the pretrained features are good, LP learns a near optimal linear head which has small OOD error (Lemma A. but fine-tuning has high OOD error (Theorem (.2). We give more formal version of Theorem 4 and a proof in Appendix A
ID comparison: When the pretrained features have some error, we show that fine-tuning does better than linear probing ID because fine-tuning can update the features to fit the ID data. The non-degenyracy condition on Raug below is similar to our previous results, and holds with probability I if the ID subspace is chosen randomly, from Lemma
Proposition 3.5. In the linear overparameterized setting, under the ID subspace assumption (Assumption B.3), let Ro = rowspace( Bo), and Raug Span( (w.} U Ro). Suppose w, E Ro. cos0max(S,RR) Raug) # 0, and that fine-tuning converges to a local minimum of its loss, then fine-tuning does better ID almost surely: Lia Li ,BRe) < Lia(Upp .B)) with probability I (over the randomness of the training examples).
To summarize, we proved that there are tradeoffs between ID and OOD error: FT has lower ID error but higher OOD error than LP. In the next section, we extend our theoretical insights to illustrate why simple variant of FT may mitigate such tradeoffs.

---

Published as a conference paper at ICLR 2022
3.4 LINEAR PROBING THEN FINE-TUNING: A SIMPLE VARIANT TO MITIGATE TRADEOFFS
The advantage of fine-tuning it can adapt the feature extractor to fit the downstream task. Can we keep this benefit while ensuring that our OOD error is low when we have good pretrained features?
see that the alignment error in the head initialization "0² = 1%% v, - (v_ 2 plays an important role. The issue with FT was that under random zr initialization, 6² is usually large and since the gradient updates to the feature extractor param- eter are coupled with that of the head parameter, the features get distorted in a manner that increases the OOD error. This suggests that we should use a better head initialization -nee obtained from linear probing. If the pretrained features are decent, a linear probed head would be much better aligned with v, allowing the features to be updated in a manner that does not increase the OOD error much.
Going back to Theorem
We formally prove this intuition in simple setting where we have perfect pretrained features. Note that in this case, linear probing alone gets zero OOD error- so Proposition BTG is just a first cut result to illustrate that if initialized well, full fine-tuning does not distort features.
Proposition 3.6. Given perfect pretrained features Bo = UB. for some rotation U. Let Ro = rowspace Bo). Under the non-degeneracy conditions cosê S) + 00,c0sm ( *0:
( (t.LccaBBreC (t)) 0, vo 10 is randomly initialized (FT),
(3.10)
Vt, ( BA (t) T vn((t) (t)(tt))-))) =0, vn is initialized to oo (LP-FT).
(3.11)
4 EXPERIMENTS
We run experiments on ten benchmark datasets with deep neural networks and see that given good pretrained features, fine-tuning (FT) does better ID but worse OOD than linear probing (LP). As predicted by the theory, we find that LP-FT does better than both methods. Finally, we see that number of predictions from the feature distortion theory hold up in practice. For more details on datasets, pretraining models, and experiment protocols, see Appendix B.
We use standard distribution shift datasets: DomainNet (Peng et al, 2019 Tan et all 2020),
BREEDS-Living-17 (Santurkar et al, 2020). BREEDS-Entity-33
(Santurkar et all 402())
CIFAR-10 + STL (Krizhessky) 2009 Coates ef all 2011: French et all 2018), CIFAR-10 CIFAR-10.1 (Recht et all 2018), ImageNet-1K (Russakovsky et all 2015)- where the OOD test sets are ImageNet (Recht et all 2019). ImageNet-R (Hendrycks et all 2020). ImageNet- A (Hendrycks et al,, 2019)), and ImageNet-Sketch (Wang et all, 2019) and FMoW Geo-shift which is adapted from the satellite remote sensing dataset Functional Map of the World (Christie atall 2018 etal, (02TI). See Appendix B for more details the datasets.
Pretraining and models. We use a CLIP pretrained ViT-B/16 for ImageNet. For the other datasets we use a ResNet-50 architecture and consider diverse range of pretraining methods and datasets: MoCo-v2 (Chen all (020h), CLIP (Radlord c M02T), and MoCo-TP (Ayush el al,, (02(). In Appendix El we also show results for a CLLI--VI-B/LL and more fine-tuning baselines on Living-17.
4.1 LINEAR PROBING VS FINE-TUNING
Experiment protocols. We initialize with the pretrained model, and fine-tune linear probe on ID training examples. For fine-tuning on each dataset we swept over 6 learning rates, using cosine learning rate schedule and batch size of 64. We early stop and choose the best learning rate using ID validation accuracy. For linear probing we train an fz-regularized logistic regression classifier on frozen features from the penultimate layer of the pretrained model, selecting the best fz-regularization hyperparameter based on ID validation accuracy. For all methods, we run each hyperparameter configuration 3 times (with different random seeds), and take the average accuracy. We used a slightly different protocol for ImageNet because the dataset is much larger and running these experiments involves more computational resources: we used a batch size of 128, swept over learning rates for both fine-tuning and linear probing (we did not sweep over (g-reulaarization) and ran each hyperparameter configuration once. In all cases, OOD data was only used for evaluation.
Results. Fine-tuning (FT) does better than linear probing (LP) 5 out of 6 ID datasets (average accuracy of 85.1% for FT vs. 82.9% for LP, see Table M). This is consistent with prior work and intuitions. However, linear probing does better 8 out of 10 OOD datasets (average accuracy of 66.2% for LP vs. 59.3% for FT, see Table D -P does better on all datasets except CIFAR- 1 and I2V, where the OOD is designed to closely replicate the ID dataset. This matches
7

---

Published as a conference paper at ICLR 2022
ID accuracies with 90% confidence intervals over 3 runs fine-tuning does better than linear probing on all datasets except DomainNet (which could be because the version of the DomainNet training dataset from Tan all (2020) is fairly small, with around 20K examples). LP-FT does the best on all except FMoW where is in between linear probing and fine-tuning.
Table 1:
CIFAR-10
Ent-30
Liv-17
DomainNet
FMow
ImageNet
Average
FT LP
97.3 (0.2)
93.6 (0.2)
97.1 (0.2)
84.5 (0.6)
56.5 (0.3)
81.7 (-)
85.1
(0.0)
90.6 (0.2)
96.5 (0.2)
89.4 (0.1)
49.1 (0.0)
79.7 (-)
82.9
LP-FT
(0.1)
93.7 (0.1)
97.8 (0.2)
91.6 (0.6)
51.8 (0.2)
81.7 (-)
85.7
Table 2:
OOD accuracies with 90% confidence intervals over runs. Linear probing does better than fine-tuning on all datasets except CIFAR-10.1 and ImageNetV2, where the ID and OOD) are similar (consistent with our theory). L.-FT does the best on all datasets.
STL
CIFAR-10 10.1
Ent-30
Liv- 17
DomainNet
FMoW
FT
82.4 (0.4)
92.3 (0.4)
60.7 (0.2)
77.8 (0.7)
55.5 (2.2)
32.0 (3.5)
LP
85.1 (0.2)
82.7 (0.2)
63.2 (1.3)
82.2 (0.2)
79.7 (0.6)
36.6 (0.0)
LP-FT
90.7 (0.3)
93.5 (0.1)
62.3 (0.9)
82.6 (0.3)
80.7 (0.9)
36.8 (1.3)
ImNet-R ImNetV2
ImNet-Sk
ImNet-A
Average
FT
71.5 (-)
(-)4
40.5 (-)
27.8 (-)
59.3
LP LP-FT
69.7 (-)
7-)6
46.4 (-)
45.7 (-)
66.2
71.6 (-)
7-)9
(-)4
49.1
68.9
our theoretical predictions. Our training datasets vary in size from 20K examples to over million examples, so LP does not appear to perform better than FT simply because of small training set.
4.2 LINEAR PROBING THEN FINE-TUNING (LP-FT)
Experiment protocols. For LP-FT, we initialize the neural network head using the linear probed solution, and then fine-tune the model. LP-FT and fine-tuiing use similar compute because the linear probing step is much faster than fine-tuning. As with fine-tuning, we swept over 6 learning rates, early stopping using ID validation accuracy. For the ImageNet experiments we swept over 3 learning rates, and explicitly ensured that LP-FT and fine-tuning use exactly the same compute (we ran each stage of LP-FT for half as many epochs as we ran vanilla fine-tuning).
We find that LP-FT gets the best accuracy ID (average: 85.7%) and OOD (average: 68.9%). This is true for 5/6 ID and 10/10 OOD datasets-evert dataset except FMoW ID, where LP-FT better than linear probing but worse than fine-tuning. Since the ID accuracy on FMoW low (56.5%) this could be because the pretrained features are not good.
Results.
4.3 EXAMINING THE FEATURE DISTORTION THRORY
Early stopping does not mitigate feature distortion. Our theory predicts that fine-tuning can do worse OOD (than linear probing) throughout the process of fine-tuning, and not just at the end. To test this, we early stop each fine-tuning method and choose the best learning rate based on OOD test accuracy. As expected, fine-tuning does improve a little, but linear probing (average accuracy: 67.1%) 1%) still better than fine-tuning (average accuracy: 61.3%). See Appendix B for per-dataset results.
ID-OOD features get distorted from fine-tuning. The feature distortion theory predicts that fine-tuning changes features for ID examples more than for OOD examples, which is why fitting a head on ID examples performs poorly OOD. To test this, for each example x in Living-17 17 (results for other datasets are in Appendix B), we took the Fuclidean distance of the ResNet-50 features before and after fine-tuning: [| (x) IB, (z)l12. As expected, the average distance for II) examples (0.0188 + 0.0001) more than for OOD examples (0.0167 + 0.0001). The theory also predicts that LP-FT changes features less than fine-tuning does. As expected, the average distance changed by LP-FT both ID (0.00114 1..001110.000 0..0001) and OOD (0.0009+ is 20 smaller than for fine-tuning
Pretrained features must be good, ID-OOD far apart. Our theory says that linear probing does better than fine-tuning OOD, but only if the OOD and ID data are quite different, and the pretrained features are otherwise fine-tuning can do better OOD by adjusting the feature extractor ID.
8

---

Published as a conference paper at ICLR 2022
Feature quality: We use a checkpoint of MoCo-vl that got 10% worse accuracy (on ImageNet) and compare linear probing and fine-tuning on Living- 17. With worse features, both methods do worse, but fine-tuning (96% ID, 71% OOD) does better than linear probing (92% ID, 66% OOD).
ID x OOD: We fine-tune/ / linear probe on CIFAR-10, and test CIFAR-10.1, dataset collected using a similar protocol to CIFAR-IO. As expected, fine-tuning (92.3%) outperforms linear probing OOD (82.7%). Even in this case, where we have no tradeoffss, LP-FT does the best (93.5%).
RELATED WORK AND DISCUSSION
Fine-tuning vs. linear probing. Fine-tuning (FT) and linear probing (LP) are popular transfer learning algorithms. There substantial evidence of FT outperforming LP in-distribution (ID) including recent large-scale investigations (Kornblith et all 2019; C ctal, 20241 Zhai etal, 2020; Chen 20206) (the only notable exception is in Peters ct (2019) where LP performs better than FT when using ELMo representations, but worse using BERT). FT is therefore the method of choice for improving accuracy, while LP is used analyze properties of representations (Peters et all, 2018: Belinkov et all 2017: Hewitt & Manning, 2019). In our work, we find that FT can underperform LP especially when using high quality pretrained features in the presence of a large distribution shift. There are variety of other fine-tuning heuristics (Ge & Yu, 2017: Guo et all, 2019; Zhang et al}. 2021: Zhu et all M02[; Jiang et al., 2021: Aghajanyan et al. 2021) combining our insights with these ideas might lead to better methods.
The benefit of preserving pretrained features. Our work adds to growing evidence that lightweight fine-tuning, where only a small part of pretrained model are updated, can perform better under distribution shifts-and we give a theoretical grounding to why this might be the case. Zero-shot language prompting in vision (Radtord ct all 2021) and other lightweight fine-tuning approaches in NLP (Houlsby ct all 201% & Liang, 2021: Xie et 20216 et 2021; Utama et all, 202R Zhouetall 202I) have been shown to improve OOD performance. Anndreassen all (202T) observe that through the course of fine-tuning, ID accuracy increases but OOD accuracy plateaus.
Mitigating ID-OOD tradeoffs. While LP-FT has sometimes been used as fine-tuning heuris- tic (Levine al al. 2016: Kanaval Sunckt MIPAN ASER)), it has not been used for robustness/ OOD accuracy, and we show that it addresses the ID-OOD tradcoff' theoretically and empirically. Tradeoff's between ID and OOD accuracy are widely studied and prior work self-trains on large amounts of unlabeled data to mitigate such tradeoffs (Raghunathan et all, 2020: Xie all 2021a; Khani & Liang, 2021). In contrast, LP-FT uses no extra unlabeled data and is a simple variant of fine-tuning. In concurrent and independent work, Wortsman et all (2022T show that ensembling the weights of zero-shot and mee-tuned model mitigates the ID-OOD tradeoff between these approaches, and this method could be promising for our datasets as well.
Theoretical analysis of transfer learning, Prior works on transfer learning mainly analyze linear probing (Wilet all, 2020; (Tripuraneni all, 2020: all 2020). Recent works (Chua etall 2021; Shachaf et all 1)21) study fine-tuning, but in the underparameterized regime (where there is a unique global optimum) or assuming a balanced initialization. Prior works also focus on ID error, while we analyze OOD error. See Section for additional related work on theory of overparameterized models.
6 CONCLUSION.
There is a strong trend towards leveraging pretrained models to improve downstream performance, and whenever feasible, it is common to fine-tune all model parameters. In this work, we show theo- retically and empirically that preserving features might be important for robustness, and simpler ap- proaches like linear probing can improve -ittr--distribution (OOD) performance. This OOD gap be- tween fine-fine-iee fine-tuiing and linear probing grows as the quality of pretrained feerainndffatuurr features improve, so we believe results are likely to gain significance over time with growing innovations and scale of pretraining.
Finally, we showed LP-FT can mitigate tradeoff's between ID and OOD accuracy in our context. LP- FT could be useful in other situations, for example in CLIP we could initialize the final layer with the zero-shot classifier and then fine-tune the entire model, as done in concurrent work (Wortsman et all could first prompt-ture
2021). In NLP, linear probing is not as good good-here then fine-tune the entire model. LP-FT is just a first step in leveraging the intuition from our theoretical analysis and we hope that this work inspires new methods of leveraging powerful pretrained models.
Lesteretal. 2021) and

---

Published as a conference paper at ICLR 2022
Proofs and Reproducibility: We include proofs for our theoretical results in Appendix A and additional experiment details in Appendix E Updated code is available netbs:
learning and this CodaLab worksheet.
Acknowledgements: We would like to thank Kumar Ayush and Burak Uzkent for MoCo checkpoints pretrained on unlabeled FMoW images, Nilesh Tripuraneni for clarifications on his work and refer- ences principal angles, Daniel Levy for useful suggestions experiments run, Niladri Chatterji, Jeff' Z. HaoChen, and Colin Wei for useful papers and comments on figures, Niladri Chatterji and Kaidi Cao for reviewing the paper at ML paper swap, Kevin Yang for his help with analyzing differ- ential equations, Tri Dao and Pang Wei Koh for help with writing. Suriya Gunasekar, Adam Kalai, Simon Kornblith, Ting Chen, Sang Michael Xie, Albert Gu, and Kendrick Shen for useful discussions, and Pang Wei Koh, Niladri Chatterji, and Tri Dao for suggestions on framing our results better.
Ananya Kumar was supported by the Rambus Corporation Stanford Graduate Fellowship. Percy Liang was supported by the Open Philanthropy Project and NSF Award Grant No. 1805310. Aditi Raghunathan was supported by a Google PhD Fellowship and Open Philanthropy Project AI Fellowship. Tengyu Ma acknowledges support of Google Faculty Award, NSF IIS 2045685, the Sloan Fellowship, JD.com, SAIL, and SDSI.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations (ICLR), 2021.
EA AlBadawy, Saha, and MA Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med Phys., 45, 2018.
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distriuution robustness throughout fine-tuning. arXiv, 2021.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning (ICML), pp. 253, 2018.
Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, M. Burke, D. Lobell, and Stefano Ermon. Geography-aver self-supervised learning. arXiv, 2020.
Peter L. Bartlett, Philip M. Long, Gt'abor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. arXiv, 2019.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural ma- chine translation models learn about morphology? In Association for Computational Linguistics (ACL), pp. 861-8722 2017.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. arXiv, 2019.
Koby Bibas, Yaniv Fogel, and Meir Feder. A new look at old problem: A universal learning 2019 IEEE International Symposium on Information Theory
approach to linear regression. (ISIT), pp. 2304-2308, 2019.
Tianle Cai, Ruiqi Gao, J. Lee, and Qi Lei. theory of label propagation for subpopulation shift. In International Conference on Machine Learning (ICMI),, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning 2020a.
(ICML.), pp. 1597-1607, 2020a
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self supervieee vision transformers. arXiv preprint trXiv:2104.02057 2021.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Computer Vision and Pattern Recognition (CriiionnnCC (CVPR),
Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning. arXiv preprint arXiv:2105. 02220, 2021.
Adam Coates, Andrew Ng, and Honlak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the Fourleenth International Conference Artificial Intelligence and Statistics, volume 15, pp. 215-223,2222 2011.
Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv, 2020.
Simon Shaolei Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced. In Advances in Neural Information Processing Systems (NeurIPS), 2018
fastai. fastai tutorial on transfer httpss////tttbbco//aaaaaaaaaaoaa b /llbbmmsteer/bs//dl1a.
www.pppp...... ipynb.

---

Published as a conference paper at ICLR 2022
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference Learning Representations, 2018.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In Computer Vision and Putlern Recognition (CVPR), 2017.
Gauthier Gidel, Francis R. Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in deep linear neural networks. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
Gene H. Golub and Charles F. Van Loan. Matrix Computations. The Johns Hopkins University Press, 2013.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6151-6159, 2017.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.
Spottune: Transfer learning through adaptive fine-tuning. In Computer Vision and Pattern Recognition (CVPR),2019
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv: 1903. (90t::050000000 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In Computer Vision and Pattern Recognition (CVPR), 2020.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML), 2019a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv: /907 1774, 2019b.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006. 2020.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Association for Computational Linguistics (ACL), 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. arXiv, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Association for Computational Linguistics (ACL), 2018
Neal Jean, Marshall Burke, Michael Xie, W. Matthew Davis, David B. Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353, 2016.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In International Conference Learning Representations (ICLR), 2021.
Fahdi Kanavati and Masayuki Tsuneki. Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning. In Medical Imaging with Deep Learning, 2021.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups dis- proportionately. ACM Conference on Fairness, Accountability, and Transparency (FFeeeccCCFFFF...... (FAccTT) 2021.
12

---

Published as a conference paper at ICLR 2022
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In Computer Vision and Pattern Recognition (CVPR), 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Thomas Laurent and James H. von Brecht. Deep linear neural networks with arbitrary loss: All local minima are global. In International Conference Machine Learning (ICML), 2018.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint wrXiv:214.00691, 2021.
S. Levine, Chelsea Finn, Trevor Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies, Journal of Machine Learning Research (JMLR), 2016.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with
convolutional networks. In International Conference on Machine Learning (ICML), 2018.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv: 05:008 2019.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning (ICML), 2021.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83. 2020.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. search of the real inductive bias: On the role of implicit regularizationnn in deep learning, arXiv, 2014.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In International Conference on Computer Vision (ICCV), 2019.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. North American Association for Computational Linguistics (NAACL), 2018
Matthew Peters, Sebastian Ruder, and Noah Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP 2019), pp. 7-14, 2019.
Viraj Prabhu, Shivam Khare, Deeksha Karthik, and Judy Hoffman. Selective entropy optimization via committee consistency for unsupervised domain adaptation. In International Conference on Computer Vision (ICCV),
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICMI), volume 139, pp. 8748-8763. 8763, 2021.
13

---

Published as a conference paper at ICLR 2022
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In International Conference Machine Learning (ICML), 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 generalize to CIFAR-10? arXiv, 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML). 2019.
Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics, 02:1707-1739, 2009.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 1 115(3) 1-2222 2015.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv, 2020.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamies of learning deep linear neural networks. arXiv, 2014.
Gal Shachaf, Alon Brutzkus, and Amir Globerson. A theoretical analysis of fine-tuning with linear teachers. In Advances Neural Information Processing Systems (NeurIPS), 2021.
Shuhan Tan, Xingchao Peng. and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey. arXiv preprint arXiv: 10320, 2020.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig
Schmidt. Mcasuring robustness to natural distribution shifts in image classification. arXiv preprint arXiv:2007.00644 2020.
Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. arXiv, 2020.
Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning, 2015.
Prasetya Ajie Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. Avoiding inference
heuristics in few-shot prompt-based finetuning. arXiv preprint arXiv:2109.04414: 2021.
Haohan Wang. Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.001903 2021.
Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference on Learning Reprexentations (ICIR), 2020.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma. and Percy Liang. In-N-out: Pre-training and self-training using auxiliary information for out-of of-oiiiriibuttinn robustness. In International Conference on Learning Representations (ICLR). 2021a.
Sang Michael Xie, Tengyu Ma, and Percy Liang. Composed fine-tuning: Freezing pre-trained denoising autoencoders for improved generalization. In International Conference Machine Learning (ICML), 2021b.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. 1d0k00k: A diverse driving dataset for heterogencous multitask learning. In Computer Vision and Pattern Recognition (CVPR), 2020.
14

---

Published as a conference paper at ICLR 2022
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlshy. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv, 2020.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning; A baseline for network adaptation via additive side networks. In European Conference on Computer Vision (ECCV), 2020.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01 2021.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. FreeLB: Enhanced adversarial training for natural language understanding. In International Conference Learning Representations (ICLR), 2020.
15

---

Published as a conference paper at ICLR 2022
A PROOFS FOR SECTION B
A.1 PRELIMINARIES IMPORTANT NOTATIONS AND PRINCIPAL ANGLES
Big-Oh Notation: For convenience, we use big-oh notation in way that differs from standard theoretical computer science texts. When we say O(<exprl>) we mean that this can be replaced by <<exprl> for some universal constant such that the statement holds. As example, we can say 5x² & ((x²) because there exists some universal constant (c (c= 5) such that 5x I 5x More examples: can also say 5x 2 O(²²) or if 2 then 7 I 0(xr)) and 0.1x² 2 (z)).
Singular Values: Given a rectangular matrix A E RRX"" let r= min(m.n). The minimum singular value is defined as the r-th largest singular value of A, so o min (A) = (A).
Working with minimum singular values requires more care than maximum singular vectors. In particular, when have rectangular matrices some bounds depend whether the matrix is "fat' (has more columns than rows) or 'tall' (has more rows than columns).
Given matrix A, the operator norm ||A||2 is the maximum singular value: ||A||2=0mmx<4)
Projectors: Given a subspace of R4, let I denote the orthogonal projection onto R, satisfying that for all e Rd:
(x) E R and Vr E R, [|>-In[(>)]25 (-----2--
(A.1)
If F E (dxdim(RR) has orthonormal columns that form basis for R, then we have:
R = EE
(A.2)
From this we can easily check that 113; = II and I1 IIR. See e.g., Chapter 2.5.1 Goluh Loan (2013) for more information.
Principal Angles: Given two non-zero vectors and y. the cosine of the angle between them, cosê, is:
27 y
cos
(A.3)
|
If we consider the I-dimensional subspaces (so basically lines) S. and Sy spanned by x and y respec- tively, then the angle between them, cost' is given by the absolute value (since lines are undirected):
1x
coso'
(A.4)
Principal angles generalize this notion to higher dimensions. Seeegg, Chapter 6.4.3 in Golub Loan (2013) for more information on principal angles.
Definition A.1. Given two non-empty subspaces and S of Rd, where r min(dim(R)) Rim (S)) we have r principal angles:
0501 S... 50,
(A.5)
The directions of the inequalities swap when we lake the cosine of the principal angles:
1 12 cos0, 2 ... 2 coso,, 20.
(A.6)
The cosines of the principal angles are given by the SVD-let E Rdxdim(RR and E Rdxdim(SS) have orthonormal colunns which span R and S respectively. Then have:
coso, [F F).
(A.7)
where o; denotes the i-th largest singular value. In this paper, we are interested in the cosine of the largest angle between them, given by:
cost cos5max(RRS) =coS)=
(A.8)
We can massage this into a variational characterization of the maximum principal angle, which is important for lower bounding the error of fine-tuning outside the span of the training data.
Lemma A.2. Suppose dim(R) R) < dim(S), and let F E (Sximi(S) have orthonormal columns that form a basis for S. We have:
cos0max -
min [FT(r)]l
(A.9)
16

---

Published as a conference paper at ICLR 2022
Proof. Let E E Rd and F E Rdxdim(S) have orthonormal columns that span R and S respectively. Since dim RR) di)( (a crucial condition!), F E is 'tall' matrix (it has more rows than columms) so we have:
mmin (F
E)=
min || |F Eell2. ||
(A.10)
The result now follows from some algebra:
costmax
(R,S) = min [F' E)
(A.11)
min | Eull2
(A.12)
|| ==
min
||FT (r)]l2-
(A.13)
rEnt rll.
A.2 FEATURE DISTORTION THEOREM
We first prove our core theorem, that fine-tuning distorts pretrained features.
Restatement of Theorem 3.2. In the overparameterized linear setting, let S = rowspace x)+ Rg rowspace Bu), and v., B. be the optimal parameters with w. B.V.. If Ifcoso max! ( S- ) > 0. then for all time steps t, the OOD error of the fine-tuning iterates (B) (t)) is lower bounded:
c0000 max ( min ( [12.112) (
-c).
v /Tood Utee (4),
Omin (EE)
(A.14)
Vk
(1+ (1+1]==112)
where x²² = [0g v.)2 - 2 is defined to be inital head alignment error and e B.) is the error in the pretrained feature extractor.
We follow the sketch in the main paper. begin with a few lemmas, showing that certain quantities are preserved throughout the fine-tuning process.
Our first lemma says that the representations 2 do not change for examples perpendicular the span of the training examples. Note that the final output u'z BByaa still changes, because uiz changes.
Lemma A.3, For all times t and all x St, we have:
Box= BB,
(A.15)
Proof. We initialized fine-tuning with the feature extractor (0) = By- It suffices to show that 0, 0, forall xeS" Recall that B), given by the gradient flow update equation:
- - 0 wー 31 u-1
(A.16)
Computing the RHS explicitly using multivariable chain rule, we get:
a,B,, - - v
(A.17)
Since x is a constant, we get:
-D0,,=-20(XX0 v-Y) o-Y) Xx
(A.18)
But Xx: = 0 for x E S since E S is defined as is perpendicular to the rowspace of X (i.e., perpendicular to the rows of X). So the RHS is 0 that is, ,), =0 as desired.
Next, we show that the change in the head and feature extractor 'coupled'. So if the head changes in a certain way, then the feature extractor cannot just stay the same. In the literature, this is sometimes called the "balancedness" lemma, and has been proved in prior work on two layer linear networks.
Lemma A.4. For all we have:
1000 - -B, B. =1/00/1
B¹, B
(A.19)
17

---

Published as a conference paper at ICLR 2022
Proof. This follows by showing that the derivative is 0:
o T BBj, T 1=0
(A.20)
Which can be verified by direct calculation. See Theorem 2.2 in Du (2018) and the proof of Theorem in Arora all (2018).
For our proof we will require that every feature r E R can be generated from some OOD direction, that is Bou for some E S We will show that this is implied by the condition the principal angle: cos max R, S-) > where = rowspace( Bo), which we assumed in Theorem 3.21 The following lemma shows this (and also quantifies that the norm of u does not shrink too much when projected onto Rt).
Lemma A.S. Let R, S be subspaces of Rd with dimn(R) < dim(S). For all T E R with ||rl|== cost (R,,)) there exists s E S with Iı{(s) = and || 1. Here II E Rdxd projects vector onto R.
Proof. Let cos A max S). Firt, get rid of an easy case if O, then need to show the claim for all r E R with ||r||== c = O, which means r = 0. Then we can just pick s = 0, and 1==))=0=>> and |||| = 0 1. So for the rest of the proof we assume e>> 0.
Consider arbitrary vector R with |||||== Let E E Rexdim(S)) ,F RRddii((R) have orthonormal columns, which form a basis for R and S respectively.
Step 1: Finding s: Since the columns of E span R, r= Ez for some : Rddim()) c=0 (E' SAFFF > 0, which means that E F E (ddi(()))diull has rank dim [t)) since dim Rt) S dim( other words, E F has full column rank since the column dimension is smaller than the row dimension. So E Fw for some w rowspace(E" E F). Then we set == Fw-thittt this means S S because the columns of F form basis for S. In addition, following the steps above have Ez EE Fw EE We note that ; EET is the projection onto R (see e.g. Chapter 2.5.1 of Golub Loan (2(1)))
Step Bounding norm of
suffices show that ||s||2 1. Since F has orthonormal columns, Is | ||l]|2 so it suffices to show that || < 1. Since E' has orthonormal columns, 21|2- Recall that 2 E Fw since w crowspace (E" F), from Lemma A.G we have:
|| min ( (E
(A.21)
Rearranging, we get
|w || /////////// 1, as desired.
In the lemma above, we used standard linear algebraic result that we include for completeness. This says that cannot shrink vectors in rowspace too much, where the shrinkage factor is given by the minimum singular value of A.
Lemma A.6. Let A E g=x* Let r = min(m, n). Then if x E rowspace(A)) we have (A)]
Proof. We bound the norm of æ using the SVD. Consider the singular value decomposition (SVD) of A:
A-UDV
(A.22)
Where U E RRxrr D D E Rerr, VT E Rexn, where U and V have orthonormal columns, and D ===iag(o(,....pp is diagonal matrix with 01 2 .... 20,
| JUDVVarl) arl2
[Definition o]
(A.23)
= -DV æll2
[U'R" has orthonormal columns]
(A.24)
2 20,1l T>]]2
[D is diagonal]
(A.25)
=,,11=||2
[rows of T are orthonormal, x is in rowspace]
(A.26)
(A)(aa)
(A.27)
18

---

Published as a conference paper at ICLR 2022
Where for the fourth step, we used the fact that if x E rowspace VT) and the rows of VT are orthonormal, then || >=|==2 ||2||2- One way to see this is by writing -, (ay where rows of VT, and then noting that V x (a....,0 and so x and V have the same norm.
We recall that Food has second moment Σ: B|rr [] ]= E when aod. where E. is invertible. So with some simple algebra we can write the OOD error Lood in terms of Σ (the proof is standard and basic, but we include just for completeness):
Lemma A.7.
Toood (1. (v. () (B, u. v.- BTd) E(B, 14 B S Smiin [0min(=)]] v.-B... T
(A.28)
Proof. Let Pood. We have,
Lood =B]()]]((v] B_ B Bx)
(A.29)
=B[(B, v. -B-BB. -BTu) xx (B, v. - v)]
(A.30)
= =BBTu.. u.B.B.BB.. (3v)) Efær [B. v. B v)
(A.31)
= -B.V.. L.B BB v).
(A.32)
The inequality follows immediately because Jmin (A) (for square matrix A) is simply the min over I with unit lg norm of x Az.
We now prove Theorem 12 following the steps outlined in the main text.
Proof of Thhorem 3.2 Let c cos max (R, S*). From Lemma A.7. we have Lood (v' BBp,) S
min (E)IB] v, Bj, u/, 12 so it suffices to bound |BB v, BB, eflll..
Because it makes the proof much easier, we will prove the contrapositive, and then convert back to the original theorem statement. We assume [B_ -B,, o. A, and will show that:
((v 2- ) A +e (1
(A+c)²
92 (2((2x]2))
(A.33)
c²
Where 91 and g2 are non-negative polynomials we will bound in the proof.
We gave a basic outline the proof in the main paper, and here we are just trying to be careful about capturing all the dependencies. We also give intuition for each step before diving into algebra (which we include for completeness).
Recall that in the overparameterized linear setting assumed have orthonormal Bo with BB - U B. I for some U. We note that the setup is rotationally symmetric so without loss of generality we can suppose B BBo-B.ll25- e. This is because can let B3 = UU and = Uv. and we have entire proof in terms of B", and v'
B e, (U (UVv). where w, is the optimal classifier so we can now write the
I| A/c: We first give intuition and then dive into the math. The key insight is to use the fact that in 'many' directions B and Bg are the same (formally, for all x E S!, By, BD.).. But Bg and B, are close by assumption, which means that By, and B, are close in 'many' directions. Then since we assumed in the contrapositive that uf, By, and v._ B. are close, we get that v'y, and v. close in 'many' directions. Because S+ covers the rowspace of Bo, get that 'many' is k. which is precisely the dimensionality of v., the vectors ut, and v, must be close.
Step 1: Show that ||0/
We now dive into the math. Since Bo has orthogonal rows, Bo has full column rank.
Let z be given by:
(1/ -v.)
(A.34)
10y ft V.Il
We note that || Then, we can find y E R rowspace( Bb) such that Boy = (since B, has full column-rank) and then 2= ||=||== (since Bg has orthonormal rows).
19

---

Published as a conference paper at ICLR 2022
(R, 1 ) 0, and E with l from Lemma A. we can choose x E E with Since c= cosê cas&max RR
||>| and [2) = Then, we have Byz=z.
From Proposition AWE since I E S Bo does not change in directions when fine-tuning so we have: =nz= J
The claim now follows from simple algebraic manipulation, following the intuition we described. The algebra just captures what "close" means and adds up the error terms.
det V.)
1l0/ - 1+112 = (u/ - vx)T(
[Algebra]
(A.35)
110 12112
- (wa)-
[Definition x]
(A.36)
(1'/ -v,) Box
[Since =b 2]
(A.37)
1 (1/) Box Box)
[Algebra]
(A.38)
c
1 (1/) BBy u. Box)
1B B6x since x S
(A.39)
(U/, B¹,-- - Bo)
[Algebra]
(A.40)
$ 1o'e T By,- u1 Bol12|10112
[Cauchy-Schwarz]
(A.41)
1 e
1ob By - Boll2
[since ||
(A.42)
106 B/o- (i.-uBBB.l B+ll2+
||u B, -v. Bolla
(Triangle inequality]
(A.43)
< -IB o/y - B.Tu+l2++= |1__ B. _ Boll2
[Taking transpose]
(A.44)
T u//1 B v. vvl112 mmax (B - B. 11,)11111 |By,
[definition of max singular value]
(A.45)
1 1
T o B u. -elllll
[since T (Bo-B.)5e)
(A.46)
A +e|lu.l12
[since |B_ v. -B/, T
(A.47)
(A.48)
Which shows that - (A
Step 2A: Show that |1 is small: The key insight is to take the trace on both sides of Proposition A.4, which bounds the Frobenius norm of By, and therefore the operator norm.
Rearranging Proposition A. we have:
By BBB +u - -000
(A.49)
Taking the trace everywhere, we get:
BY, 7)
Tr - ) -
(A.50)
For any matrix A, Tr(A = || and for a vector the Frobenius norm is just the fo-norm, so Tr(voT)= |
So we have:
1 -1
(A.51)
Squares non-negative, get the inequality:
|| |1 Bn +|
(A.52)
20

---

Published as a conference paper at ICLR 2022
Step 2B: Show that |B 1, |B), T 12112 is small: This step doesn't 't involve much insight, and is standard peturbation analysis- we simply factor the difference of squares and bound each term.
First, we bound |B
T u'y B' T v.l12:
BB T 6//1 BB/ uxl12 S max m (B))))l
(A.53)
|BBullellu
(A.54)
- v |B +l 11:ll 12112
(A.55)
A A+ellual1a
- / 111111+++
(A.56)
Next, we bound
[| v. BByB BB/, U. 11a
BB B/y 4 1| -
BBj v+ B. 1x 1= B3 1/ U- Bjz+ 21 ll2
(A.57)
S max Bo - 1--B+)l 1 1121 | |1, Bly 1212
(A.58)
S <d|u,, 2 [B_ v. -B u=l12
(A.59)
< sellu, 22+ [B_ 14 BB 1 +1 BBy T 1/y=- -Bje T 1112
(A.60)
A A (
< <e|lu, 1 + / Bo /0|| +1
(A.61)
A2
(A.62)
Finally, we bound [| e. - B, T v. 12). using the identity:
|| - ( v) (u++)]
(A.63)
$ </1u /||u-u/aalu
(A.64)
(A.65)
Applying this:
v. IR.TT..11
11.1-1111 T
[ v, B¹, T
v. 12 v. 1.l12+1143 v.- -B,, v.ll2)
(A.66)
$ A B/J v. + A2)
(A.67)
A2 B v. 11 + 21 0 B +2))2
(A.68)
< A2 w. 12 + 1c
(A.69)
A:
(A.70)
Step 3: Use Proposition A.4 to show un and v, must be close: The key insight is that we start from Proposition A.4, and left and right multiply by v., after that we use the previous steps and do some some standard perturbation analysis.
We start from Proposition 4
10%0 - Bo =0/0001 BB T
(A.71)
The key step is to left multiply both sides by v.I and right multiply both sides by v, to get:
(ng e. )2
B e. 113 (0/, T '2,²= BB, e.
(A.72)
Rearranging, and then using Equation A.66 we get:
Kuje v.)² -20 v.) = v. [BJ v. 11315
(A.73)
This is close to what want, except we have (v/2 0, on the LHS instead of (v. v.)². We previously showed that u and v, close, in Step 1, with some algebra bound the
21

---

Published as a conference paper at ICLR 2022
difference between (0/ T v.)2 ) and (w. v.)
1(1/2 T ".,²-
(2_ v. 2. | [10 v, -.. v.) v.) (0/ v. tutu.uu v.]l
(A.74)
= 1(68/ v, -..-,,1 v.) [20, u. (u/e v--u.uu v.]]
(A.75)
[vv, (1/c- - v) [2 vx (v] (xf:
(A.76)
11// -0 12.11 111
(A.77)
(A/c))l (2lt/e))))))
(A.78)
Above, from the third line to the fourth line, we used triangle inequality and Gauchy-Schwarz
So finally, by -iingll-neequaltty we can now bound [(0_] v.)2 (..)2-((n) (20 v. ).
1 v.)² (wo 2 11 (uyy T xx) ++ T v2)² - v.) vvgvv))
(A.79)
A3++A3
(A.80)
Wrap up i.e,, writing out A4 Aa explicitly: This is basically the bound we want, but would like to express A,, A4 terms of A and e. Note that this step has no insight, and is just algebra-we include the details for reference and verifiability. We recall:
A = (A/ va (la A3 = 22 w.
(A.81)
222222/ll 1e. 11++
(A.82)
BBll +1u. A+elleala) 11ee/l..12+44
(A.83)
c
Since Bo has orthogonal rows (by assumption), Bu has orthogonal columns, so w|,, 1|2= [| v. v- |2+12- In addition, since Bo has k orthogonal rows, || |BBll-== vh. We also note that B b Boll lI + -1 v +//+|//.ll |/2. Since cs 1, we have:
(
A A
1|,+112+A5
(A.84)
So for Ag, up constant factors we can ignore the el|e, 112 term this means we get:
A ( A A+elluulla
(A.85)
Using the fact that Vk+ ||,, 2 /k[1+/|ww we get:
A2 50 (Vk(((++11u (Atelluall))
(A.86)
since A A+|||w=+1225 (1 w+ ((A+)). we get:
A2 < 0 [1( 11) A+e (
(A.87)
Now for Ag, first note that 5 2, since B, and Bn have orthogonormal rows so |B, - 1B.-Bol1-- 2.. This || ww, ||2. so A3 simplifies to:
means that e/] |
Ag
(A.88)
Substituting the bound for A2 into Ag. we get:
A Atclluulla) | + +*(1+11lu, 11) (ttululll))))))
As A ( 2
(A.89)
For As, we get:
A. s 0 || 1
A
|| 112( (
(A.90)
Since A/c5(A-e) |1
1|2) we have for the final error Ag + Aa:
A+c + 1|,++111110 113)
A A+02
As+ A=+AivVeuLL+|euulee
(A.91)
22

---

Published as a conference paper at ICLR 2022
Wrap up ie., taking the contrapositive: So we've ve shown that if |B_ 1, B-BBie T
U// 1/1133 A, then:
[(, v,)² -wo v,) 3 A+E w11+|l -1 a
c² (1+ [l,.11))
(A.92)
We'd like to flip this around: suppose [(v] - v.) 2 8² for some s. To lower bound [| v. Bi e/ell:. we simply take the contrapositive of what we have proved. Let A be given by:
A A-min w(1+ w., 12) V [1+ 13)4
(A.93)
In this case with some algebra, we can show that:
A+E w (1+ 1|,, 11, V
(A 2 c² (1 1|,,, 12))
[0__ u. 2 (vo v.)
(A.94)
To see this, we bound each of the terms in the RHS separately using our definition of A. Then, from the contrapositive of what we proved (compare with Equation 92, we get:
| v. By, T u A
(A.95)
Finally, we can massage A to combine terms and make it look slightly nicer:
min((p;p2/111) /1/p::///111/11222 /l/ Vi (1+1|w-|l))
(A.96)
Then applying Lemma A we get the desired result. For even more interpretability.. if || 12= and
o is bounded above by some constant, then you can think of A approximately vk x² 6. This completes the proof.
A.3 Vs FT (OOD)
We now prove Theorem SEL which compares linear probing and fine-tuning in the linear overparam- eterized setting, when the ID data lies in a lower dimensional subspace. We first define distance (formally, a pscudometric) between pretrained feature exactors, which measures the operator norm difference between them up to rotations (c.g., if B U' for rotation matrix U then B and B' are equivalent since this means we can obtain one feature extractor's representations from another just by rotating):
Definition A.8 (Feature Extractor Distance). The distance between feature feature extractors B,B' E Rixd (with orthonormal rows) is given by (where the min is over rotation matrices U E <Rt*k):
d d(B.B' = = B B-U B'
(A.97)
We state a more precise version of Theorem 3.4 basically we fix all problem parameters except Bo (which limits to B.). To define the limit, consider a sequence of pretrained feature extractors: [ We define the corresponding limit points of fine-tuning and linear probing when we start from the i-th pretrained feature extractor. That is, let en." (t), Br (t) denote the parameters at time t of fine-tuning if we initialize with Bi (see Equation 3 for the fine-tuning updates). Let Ulp , be the linear probing solution when initialized with un Bj (see Equation 3.3 for the linear probing updates). We note that the LP iterates converge to up BB as result of gradient flow convex problem.
Finally, Theorem 34 says that the pretrained representations get better, linear probing does much better than fin-ttnning OOD:
Theorem A.9 (Formal statement of Theorem 5.4). In the linear overparameterized setting, under the ID subspace assumption, fix the dimensions of the setting d, d,m,m, m, number of examples n, the ID subspace S. ID distribution Pia, the distribution over the head vn and the ground truth parameters v. B,, Assume the non-degeneracy conditions cosOmax (o.0max.RRR,S) > 0 and cosO (R, > 0 where R. rowspace(B.). Given sequence of pretrained feature extractors {Ba) with Bi + B., where the limit is in the pseudometric given by Definition 8 the ratio of O0D errors of linear probing and fine-tuning converges in probability to 0:
Lood(1fe ,B²)
0,as 00000.
(A.98)
inf InfeooLood UV (t), (t))
23

---

Published as a conference paper at ICLR 2022
The purpose of the infimum is to capture the fact that the bound holds for all times for fine-tuning (and therefore also for the limit oi Bh when it exists). Note that the ratio is a random variable because the training data sampled from Pia and the head is sampled (Vo (0,00 for some σ²).
Proof. Recall that we say sequence of real- vellev random variables converges in probability (written as X; 3 0) if for every d', O, for all large enough i (that is, for all 12 N₁ for some N2), we have:
P([X) > 56
(A.99)
Accordingly, fix arbitrary '',0>> 0, and we will show that the ratio of errors is eventually smaller than with probability least 1 8-8.
Lower bounding fine-tuning error: Since Bi + B., from Lemma A. we have that
cos (R',S - mastmax where R = rowspace(Bij Since cosêmax ( 0, this means that for all large enough we have:
cos0 (R' ( > max (R., 1/2
(A.100)
Next, from Lemma (6y - (v_ v,)² 2 og for some cs > O. Plugging this into the fine-tuning bound in Theorem 322 this means that for all large enough i with probability least 1 56/2:
have that with probability at least 8/2, Head-Errorts v.)
inf V (on (t) (0) 220
(A.101)
for some <3>00 O. But since Bjj - B, we have dl B.) as i oo. So this means that for all large enough with probability at least 1 - 5/2:
inf Lood (th Bt) (1)) 20 220
(A.
for some of > O.
Upper bounding the linear probing error: Since Bj B., from Lemma A.T have that
cosb R SS) cosbmax S)) and so since coso R. S) > 0, for all large enough i we have:
cost RR' (,)) > cas S)/2.
(A.103)
Plugging this into the RHS of Lemma 1. Equation A. which upper bounds the OOD error of linear probing, we get that for all large enough with probability least 1 8
( BB) S ug (g(d(Bi,B.)) B.))²
(A.104)
for some ug > 0. Again since d( BB B.) 0 as i - xc, this means for all large enough with probability at least 1 1-2 B.) will be small enough that:
(1 ,B,) < cse.
(A.105)
Taking the ratio: So taking the ratio of the lower bound for fine-tuning, and upper bound for linear probing, we get with with probability least 1-8:
Lood (1 ,Bb) SC,
(A.106)
int I Lood (1) ((1)
as desired.
We now prove the Lemmas that we used in the above proof.
24

---

Published as a conference paper at ICLR 2022
A.3.1 CONVERGENCE OF PRINCIPAL ANGLE
Theorem BE assumee conditions on the angle between the perfect feature extractor B, and the ID subspace S. However, fine-tuning and linear probing start from features Bp with some error, and do not get access to B.. We show that if By and B, are close, then the angles between their rowspaces to a third subspace T' (which could the the ID subspace 5) is similar.
Lemma A.10. Given two feature extractors Bo, B, E Rkxd with orthonormal rows, where Ro rowspacec R,,, = rowspacel BB),, and subspace with dimension at least we have:
|cos@ (Ro. -cos: sd
(A.107)
Proof. Recall that dim Ro) = dim dim(R.). Letr= min(k: di)m())) and be a d-by-dim( (T) matrix with orthonormal columns that form basis for T. We have, for arbitrary rotation matrix bxk
cosê max (Ro (B:oF))))
(A.108)
=o,(UB(F)
(A.109)
(B
(A.110)
(A.111)
(A.112)
-o,(B.F)- BBBBll
(A.113)
-ossmax IUBo B-ll2
(A.114)
Here in the first step used the definition of cos max (Definition (.I)). and the fact that Bg has orthonormal columns which form a basis for Ro (the rowspace of Bo), so in Definition 3 we can subtitute E' Bi To get Equation AIH used Weyl's theorem, which bounds the singular value under perturbations: (A + B) 2 (A) o1 (B). To get Equation A.2 we used the fact that [|Fu||2||| |||||| since F has orthonormal columns.
Since this holds for all rotation matrices U, we can take the minimum over U to get:
cos60 Ro, 7) 2 max RR,TT) T) minn|[U |U Bo B2 cosfllmmm (R.,T) dl Bg B.)
(A.115)
Since the relationship between Bg and B, are symmetric (and the distance d symmetric), this gives us the desired result:
(A.116)
Lemma A.11. Given a sequence of pretrained extractors { with Bi - B., where Bi B. E lxd have orthonormal rows, let R rowspuce(BB). R, = rowspace B.). Then for any subspace T. we have:
coso (T) + cas0max Ta,,aa aas 00.
(A.117)
Proof. This follows directly from Lemma A. II
B + B, means d(B), B.) 0. Then from
|costmax (R' I) coso (R,T)) + 0,as i0
(A.118)
This means cos0 (R', ,T) cos00 Ro T)) as +
A.3.2 BOUNDING THE HEAD ERROR
(u. v.)21. We prove a lower bound on Head-Erron Error v.) ((n]] - in the fine-tuning lower bound (Theorem (.2)). Note that if the head is initialized as wp = 0. then Head-Error(vo, r.,) = ||v..12 1|3= |/0.+112- In practice, the head usually initialized randomly, for example normally distributed. Intuitively, the head error is still high because we do not know which direction the head is pointing in, so most of the time the initial (randomly sampled) head will be pointing in the wrong direction. If vo (0, can show that for any o² the head error will still typically be at least [2[[[v.]la This an illustrative result, one can show similar results for other random initializations well.
which was a key term
We first prove an anti-concentration lemma, which says that if is univariate Gaussian, then it cannot be too close any particular constant a, matter how the variance of the Gaussian is chosen.
25

---

Published as a conference paper at ICLR 2022
Lemma A.12. For some universal constant c, given a > 0, for all v² if ~ N(0, then for all 05651:
P(Iu-alScdon <
(A.119)
Proof. Consider such that å S 1/10. Then for all u with / u-al al S ôa, we have u 2 9a/10. For all 2 1//9//1000 the density [u) is upper bounded (from the formula for the density of a Gaussian random variable) by:
f(u) <0( V
2-102
(A.120)
We can maximize this explicitly (c.g., use Mathematica or by taking the logarithm and then setting the derivative to 0) and we get for some universal constant é 10 (it is OK choose larger universal constant than needed):
f(u)
(A.121)
a
Since the density is less than c/a and if Iu al S da the size the interval is 25a, we get for all 5 1/10:
P(lu-al al Sōa)= v 2d'da 2
(A.122)
Now, we substitute =2'2.. We get for all 8' 3 100
P(uu-al - 2c
8'a 58 58
(A.123)
Since e' 210 210022/1102 so the statement true for all 0 0 $1
We now bound the error in the head if the initialization is Gaussian. This bound holds for all initialization variances o². Similar bounds can be shown for other (an-Gaussian) head initializations using similar anti concentnnentoiionn arguments.
Lemma A.13. For some universal constant c, for all v, E R2 with e # O. O & [0,1), if
to N (0,o Tx), we have with probability at least 1 5:
(Head-Error(Ce v1 ))² := 1(ug v.) (v. v.) c6 v.)²
(A.124)
Proof. First note that Head-Error( Fo, v.) Head-Error( Up, v.) and up symmetric around (vo and -uy have the same probability), and is almost surely not exactly O. So without loss of generality, we can suppose that oo v. 20.
Suffices to bound [o]] JI rz/: We decompose the error:
1(vg v.) ) (v_ v. v. = 1o0 v+ v._ u. (lug Ux uu vxl)
(A.125)
2 Iul v4 - u. Un (v. v+))
(A.126)
So we bound |o] v. - v. vel.
No u, is normally distributed: We note that vp n, is distributed as:
2 v. ~ NN,,200,22 v.) In other words, a normal with mean 0, and variance of = o²vu v., and therefore standard deviation
(A.127)
σ1 ===0v12 v..
Apply Gaussian anti-concentration lemma: Then, from Lemma 12 we have for some universal constant that with probability at least 1 5:
I00 04 v. v.l 2 cōu.
(A.128)
So substituting this back into Equation 1.5 get the desired result:
[ - (u_Tue.))12000 v.)²
(A.129)
26

---

Published as a conference paper at ICLR 2022
A.3.3 UPPER BOUNDING LINEAR PROBING ERROR
We showed a lower bound for the OOD error of fine-tuning in Theorem 32 To compare this with linear probing, we prove an upper bound on the OOD error of linear probing.
For completeness we include an elementary lemma (note that the condition that the matrices are tall important for composing min. unlike for maxx and we included this lemma be careful about these conditions):
Lemma A.14. Suppose we have two matrices A, B of shape (r,s) and (8,6) respectively, and they are tall matrices so r 22t.. Then we have:
mmiu (AB) 2 minn (A)o
(A.130)
Proof. For a tall matrix A, we have:
min (A) min
|
(A.131)
| 1=
So have:
Cmin (AB)= min
|| || min (A) (i)))inBB))))) min |0]|2
(A.132)
|allaS]
And minl || which completes the proof.
Lemma A.15. In the linear overparameterized setting, under the ID subspace assumption, fix arbitrary P.. Then there exists such that with probability at least 1 1- all d.n.m.k,.. feature extractors B., Bg, and ID subspaces with corresponding F (whose columns are orthonormal and form a basis for S), if f'cosmmax (S,R) O, we have:
Lood (1p _Bo) max (S,R)
C8
d( d(Bo,
[w|||2
(A.133)
If P= is isotropic Gaussian so Im),, then we derive bound for cs analytically: if 2 5m and Z 10log & then with probability at least 1-5, the linear probing OOD error is upper bounded by:
V Food (up Bb) so( log(n/5) dl B.)llw.)l..lll
(A.134)
(cos0max (R,5))²
Proof. From the ID subspace assumption, the data matrix X of shape (n, d) can be written as X ZF where Z be a matrix of shape (n.m) with each row sampled iid from Pz, and F is matrix of shape (d.m) whose columns are orthonormal and form a basis for the ID subspace S.
Let IB Bo 125. We first prove the bounds for e, in terms of d(Bo, B.) and later handle the fact that the feature extractor distance involves the min over rotation matrices U: ((),,B))=min ||U B.
Bounding key singular values: Before proceeding with the proof, we examine a key quantity XB, ZF By which comes up in the Hessian of the loss function. We will show that this is invertible almost surely, and get lower bound on its min singular value.
First, we examine the shapes of the matrices. ZFT Bi has shape (n.d) where has shape (n,m) and FT B has shape (m,k). Since 2 m >> k we have that Z and F Bo are tall matrices, and so from Lemma A. we can write the min singular value of 2 FT BB as:
muin (ZFT BJ 2 min (Z) min (FT BJ)
(A.135)
Now from the definion of the principal angle (Definition B. have:
G (F) BJ coso >R
(A.136)
Since we assumed P. has density in the ID subspace assumption, from Lemma 3 in Xie et (2021a) we get that for some dg > that depends on and Pz, with probability at least 8-8:
min 2Z)
(A.137)
Note that this also means that miin (ZFT)) B > 0 and so XB, ZFT BJ has full rank & almost surely. This also implies that X XBJ is matrix of shape (k,k) that is invertible almost surely.
27

---

Published as a conference paper at ICLR 2022
Main proof Since B, XB B is invertible almost surely, there is a unique global minimum (minimizing over u) to the loss optimized by linear probing:
argmin||X
XB_ v. 11 BB
(A.138)
We can see this by noting that the loss function on the LHS strongly convex in since the Hessian Bo X X Bg is invertible. Then, gradient flow converges to the unique minimizer on the RHS, so:
U (B, (B, B XB.
(A.139)
We now bound the square-root OOD error (taking the square root makes it easier apply triangle inequalities), starting with the definition:
v Lood p ,Bo) = =IBB -Bo Uip l2
(A.140)
< (B, 14 -Bo v+) (Bg U. Bgy u l|2
(A.141)
1 B. v, - By v+ 1+1l2++112 U, By upp )|
(A.142)
(1)
We bound each term on the RHS of the last line. For term (1):
I V. Bg 0 5 (B,
Bo)llu-11z
(A.143)
Sc|luz112
(A,144)
=c||w, wcllw+l2.
(A.145)
Where we note that ||=-||== |w-ll2 because w, = B. v, where the rows of B, (columns of B.)) are orthonormal.
Let Σ = XTX. For term (2), we first subtitute dip and do some algebra (again noting that
||u, 2| 2l,.|22) get:
B u+ By up 11 (B,E8))) By of ll2
(A.146)
=114, - ).) uall2
(A.147)
3 5mmax (B (BoEB - (Bo - B. B.)..lus 1120,11
(A.148)
maxx (Bj (BoEBT Bo [|
(A.149)
5 (Bo)² Fmax
M BBEBI e|lwal12
(A.150)
max (Bo)² T (X)²
<||w=|12
(A.151)
mmin (X Bu )2
O mmax 20 max (2
-||ww1112
(A.152)
T muin ( (ZFT)) Bg )2
O mmxx (Bo)² o Tmaxx (2)
Mmin (c)s max (2 E lll.ullb
(A.153)
(A.154)
Where in the first line we subtituted in the closed form for dip from Equation A.13% and in the last line we used the fact that Tmax (2nxx(22') (T < (2) since FT has orthonormal rows, and Ouia (ZFT B )= 7 ZZ costm (RS) (R.,S) as explained in Equation 3.5 and Equation A.13
So it suffices to bound the quantities in the RHS. Since Bp has orthonormal rows, m (Bo)=1
No Gaussian assumption: For the first part of the Theorem (Equation A where we make no Gaussian assumptions, but give a less quantitative bound), we just use the fact that o max (Z) is upper bounded almost surely, and mmin (Z) 2 d' with probability least 1 1-8. This implies that for some C5 > 0 with probability at least 1 --: 8:
C5
2
V (1.0 (,)) cos0 (S
ell e||w=112,
(A.155)
28

---

Published as a conference paper at ICLR 2022
where c- B
Gaussian assumption: For the second part of the Theorem (Equation A 34 where assume P. is Gaussian), we use results in random matrix theory to lower bound and upper bound min (Z). For the lower bound
result from Rudelson & Vershynın (2009) (see page 4, in the equation below Equation 1.1 I), since Z E I n is a matrix with each entry sampled from / (0,1), we get all 0:
Pi (Z) S /n- -mn-vmmm) 12/
(A.156)
With bit of algebra, this gives us that with probability at least 1 - 5:
1
min (2) vn vmm V 2log
(A.157)
8
We assumed n 2 5m and n> 10 so we get:
min (2) (v)
(A.158)
The upper bound a standard matrix concentration bound-wc use the high probability bound in Theorem 4.1.1 from Tropp (2015) (see Section 4.2.2 which calculates the variance statistic for rectangular Gaussian matrices, also notice the square on the LHS below):
O Omax (Z)² S OOnlog 8
(A.159)
Substituting the lower and upper bounds on Gmin (Z) into Equation A. 44 we get:
log(n/b /5)
B v. - BJJ Up 11 (cost (R,S))² < [.. (wxl12)
(A.160)
Substituting into equation A 140 we have:
log(n/6)
Lood ,B)) 2o
(cos@mac (R,S))² (llww.ll))
(A.161)
where c= |Bo B2 Which completes the proof of the second part (Equation A.
Handling the rotation matrix U: We now handle the fact that the feature extractor distance involves the min over rotation matrices U: d(Bo, B.) = ming [ B, 1|2- Let of (Bp) denote the linear probing head solution if we use pretrained feature extractor Bo. We first note that for any k-by-k rotation matrix U, we have:
(oo
(Bo), (Bod) (UB)).UBo). UUBB))
Lood
(A.162)
This follows from using the closed form we derived above for Upp (Bo) and some simple algebraic manipulation (e.g., recall that U-1 U-!=-" TY
(UB) upp (U =U (UBB))((UB) (U XBJJUT)-UBOX C
(A.163)
=BJJUU XB.))UUTUBA XBJ
(A.164)
=B
U) XBB. u.
(A.165)
=B (BoXX X ) - "
(A.166)
BJ dip (Bo)
(A.167)
So the final predictors in both cases, (U Bo) Upp (B and BJ (Bo) are identical. This means that the OOD error Lood
Lood(., (v, |B B. v. ||a is the same in both cases.
This means that we can just take the min over all rotation matrices U (where the first step follows since the identity matrix is rotation matrix, and the second step is from Equation A.155):
(uo (Bo), S min (2 (U) Bo)
(A.168)
c(O)
<min
IIU Bo-Bo-B - B, Ba
(A.169)
cos0 max (S,R) c(O)
d( Bo B.)
(A.170)
coso (S,R)
which is as desired. Theorem statement.
We repeat the same thing for Equation A.16 to get Equation A.
34 in the
29

---

Published as a conference paper at ICLR 2022
A.4 vP FT (OOD). NON-ASYMPTOTEE RESULT FOR GAUSSIAN COVARIATES
Theorem [.4 showed an asymptotic result: if the error d(B,, B.) + O, then linear probing (LP) achieves better out-of out-of-distribution (OOD) error than fine-tuning (FT). Here we give more quanti- tative version of Theorem 33.44 for Gaussian covariates. The result can be extended the case there each entry of P2 is independent and identically distributed, mean-zero, constant non-zero variance, but instead Gaussian is sub- aub-Gaussian with constant sub-Gaussian variance moment-thhs can be shown using Theorem 1.1 in Rudelson & Vershynm (2009), which is different matrix concentration inequality.
We show that LP does better than FT cittof-ddisrrbbttion if the enror is less than specific quantity (in terms of the representation dimension k and the angles between the ID subspace and the important pretrained directions R. = rowspace(B.))
Theorem A.16. In the linear overparameterized setting, under the ID subspace assumption, as- sume the non-degenyyyyy conditions cos8max (,,,
and max S-) > where R.
rowspace B.). Suppose the covariates are generated from Gaussian distribution on the ID sub- space S, so P: =0.Tm.. Let ||w_]|2 a fixed constant. Given failure probability 1 1582 0. for all d. ifn 5m, and 10logg. the error the pretrained representation is not too high:
(.B, < <0 cosOmax ChanxxIi R.,
(R,,St , (c0s00 (B., 5)))382
(A.171)
Vklog(n/d)
then with probability at least 1 - 8, the OOD error of linear probing is lower (better) than for fine-tuning at all time steps t 2 0 the fine-tuning trajectory:
(1) ,B)) < inf Lood (M ,B)).
(A.172)
120
Proof. Let = d(Bo, B.). We first note that the condition in Equation A. implies that d(Bo,B.) O(cos0max (R,,s+) and d(Bo, B.) < O(cos0 (R,,,, This because the cosine angles are between O and 1, 5 is between 0 and 1, and & and n are at least 1. We now simplify and combine the linear probing and fine-tuning bounds.
Let Ro = rowspace Bb). Warning: note that the Equation A. in the Theorem statement assumes conditions on the angles between R. (corresponding to the optimal representation) and the ID subspace S. However, our results that bounded the fine-tuning (Theorem (22) and linear probing (Lemma A 344 errors require conditions on the angles between Ro (corresponding to the representation that linear probing and fine-tuning use) and S. So have be careful about this distinction, and use Lemma A. to relate the two, which we do below.
Fine-tuning: From Theorem
get:
castmax (ossmmaxRRo,SS (R (annpppp//llae (1+ |.., 112)
Lood Looa((n(()),Bn(t)
(A.173)
Vk
Where op is the head-error, which we lower bounded Lemma A.E subtituting this bound and noting that (nnee)9?)==(()))))) 1|=+||== ||w (which we assumed is constant), this gives us:
coso max (Ro.S') (2) be)-c
(nt)
(A.174)
Vi.
Now, since d(Bo. Ba)== we use Lemma A. to get that:
cosO S coso max (R, (1,
(A.175)
Subtituting this into Equation 1.4 we get (notice the R. instead of Ro below):
costmax (R..'' ) - 82)
(tt)
(A.176)
Vk
Since e {((oss0max S+)),, this can be simplified to:
coso (R..') S...... 82) -E
Lood Bn)) 20
(A.177)
VT.
30

---

Published as a conference paper at ICLR 2022
Linear probing: From Lemma
we get:
log(n/6)
Lood Upp ('ip, BB) BB))=00 50(
(cossl Ro S)) ////////lll
(A.178)
Again, we use Lemma to get:
cos (Ro costmax (Ry
(A.179)
Substituting into Equation A.178 and using the fact that S O fmax (R.; S)), and since we assumed ||w=|] 2l2 a constant, we get:
log(n/6)
(1 .B)) BB)5s00
00 (cos0max S))
(A.180)
Combining the two: We want to show that the OOD error of I.P less than for fine-tuning:
log(n/6) ( (cos8max (R. ())
costmax( ( 62 63)
(A.181)
Vk
We bring the the LHS, so this equivalent showing:
o log(n/5) (cos@ (R. R))5))
+cs0 cos0 Ru, St)
5²)
(A.182)
Since log(n/0)2 > and coso S)) is between 0 and 1, this is equivalent folding the inside the big-oh on the LHS:
log(n/o)
cosO max SR, 0²)
( |llw=lla) so
(A.183)
(cos@mas max R. S))2
Vk
But assuming the condition Equation 171 of the Theorem statement, this easy show with bit of algebra.
A.5
PRINCIPAL ANGLES ARE LIKELY NON-ZERO
we assumed the cosine of the largest principal angle between the representations and ID subspace (or complement of the ID subspace) was non-7ero. For example, Theorem 3.4 assumed the largest principal angle between R. = rowspace B.) and the II) subspace is non-zero, and similarly for the angle between R. and S-. Having an angle of a degenerate condition. As example, look at Figure X here the input dimension d 2, the representation dimension ==1, and the ID subspace has dimension The only way these angles can be 0 is if B. exactly in the same direction as S or S which seems like too much of a coincidence. intuitively, nature introduces even a small amount of randomness in either the optimal representation or ID subspace, the angle will be non-zero.
In Theorems G 64 and P
This example was in two dimensions to make this intuition a bit more formal in higher dimensions, prove simple claim. Lemma A.17 shows that if the S is randomly selected dimensional subspace, then the angles cos0max (R.,,, and coso R2 S are non-zero (and we get quantitative lower bounds on them).
Lemma A.17. Let R be a fixed k dimensional subspace, and let S be uniformly random m dimensional subspace (formally, a uniform measure on the Grassmannian manifold) in Ed with m > k. Then with probability at least 1 ò,
Vm-
2log
cos0 (R,S)
(A.184)
dllg 2d
(es0max(RRSS) (R,S) almost surely (with probability In addition, we get that cosê
If m 2 5k and m 2 1 then we get with probability at least 1-8:
cos max (R,S) (2,5)20(1
m
(A.185)
dlog
2d 8
Recall that big-oh oh notation here means that the RHS is true for some universal constant (independent of any other problem parameters).
31

---

Published as a conference paper at ICLR 2022
Proof. Note that Princisal angles are invariant if we rotate R and S by the same rotation matrix U. That if we let U Ra be rotation matrix, and E E Rd F Raxm have orthonormal columns which form a basis for R and S respectively, then we have:
cos0 (R,
(A.186)
This symmetry means that we can fix S and instead consider to be uniform random & dimensional subspace on the Grassmannian manifold. Without loss of generality, we can also fix S to be the span of the first m standard basis vectors: (m), where e; E Rd has I in the i-th entry and a 0 in every other entry.
Equivalently, let Mn be d-by-k matrix, where each column is sampled independently from ((o,Ta)-since the columns MR span uniformly random k-dimensional subspace, we can let R be range of Mg. This is equivalent sampling each entry of MR from (0,1)).
Let c= cosf cos R.S). From Lemma can be written as:
min r r
r1l2=
min rER, |r|a21
| |FTr1
r112
(A.187)
Since is the range of MR. any e R can be written as -arr-Mp MRÀ for some AERK We first show that ||llll cannot be much smaller than ||rl-- This is because:
||||-= || Mx o m (mn))) (Mn)l
(A.188)
So this gives us:
||1/2
|
(A.189)
o max (MR)
So every E R. can be written MRd where ||Al[2 is lower bounded as above.
We now simplify the definition of starting from Equation A.187
min 22
(A.190)
min
F MR
(A.191)
|X] 21/0mae (Mr))
min
Omin (F MR)1
(A.192)
|X 2 (Ma)
Umin (FT MR)
(A.193)
maxx (MR)
So now we want to lower bound the ratio of two random matrices. We note that F n is matrix of size km, with each entry sampled independently from N (001)) (this is because F simple selects the first m rows of Mn). nt a matrix of size (d.k) with each entry sampled independently from (0.1).
Now, as in the Gaussian assumption step of the proof of Lemma A. we apply standard matrix concentration bounds (page 4, below Equation 1.11, in Rudelson & Vershymin (2(C1) for the bound on mmin. and Theorem 4. 4.1.1 in Tropp (2015) for the bound maxx). We get that with probability at
least 1 5:
mmin (F MR) m Vi:
2log g
(A.194)
2d 0 maxx (Mn) S dlog 8
(A.195)
Note that we can use alternate bounds for Fmin tighter.
Rudelson & Vershynin (2009) that are sometimes
For the ratio of the two, we get that with probability at least 1 - 6, we have:
Jmin (FT MR)
Vm Vi= v 2log
(A.196)
T Tmax (Mr)
dlog 2d
32

---

Published as a conference paper at ICLR 2022
For interpretability, ignoring log factors this is approximately:
m Vk
(A.197)
Vd
The result when m 2 5k and n 2 1 follows with simple algebra.
For the result where we show costmaa (R,S) > almost surely, we recall that F MR a matrix of size (m, A)) with each entry sampled independently from (0,1)). Then applying Lemma 3 in Xie atal E)) we get that miin (FT MR) > almost surely. Since maax (Mn) finite, this gives us cos0 max R,S) >> 0 almost surely.
In our case, the dimension of the ID subspace S is m. and the dimension of R. = rowspace( (B)) is k, with l < m and k <dd-r m. If a uniformly random m-diimensional subspace, then S a uni- formly random d m dimensional subspace. In this case, Le us that cosf0 RR,S) > 0 and cosol R +-)
> >>>>>>>>>)
almost surely, and gives us quantitative lower bounds for these angles.
A.6 VS FT (ID)
We prove Proposition 3.5 where we show that if the representation is imperfect, then fine-tuning does better than linear probing, in-distribution.
Restatement of Proposition B.S. In the linear overparameterized setting, under the ID subspace assumption (Assumption 3. let Ro = rowspace( Bo). and Raug Span( U Ro)- Suppose w. & Ro, casemax (S, + O, and that fine-tuning converges to a local minimum of its loss, then fine-tuning does better ID almost surely: Tia( BBx) ) < Lid (n BB)) with probability I (over the randomness of the training examples).
Proof. Fine-tuning gets 0 ID loss: It is well known from prior work (I Laurent & von Brechi, 2018) that all local minima are global for optimizing two layer linear networks under convex losses (which is our setting), so if fine-tuning converges to a local minimum, it actually converges to global minimum of the train loss. Since there exists parameters that achieve loss on the training data
(namely, B,),,) this means fine-tuning gets 0 loss on the training data as well. So for all training examples x (that is, rows X):
(A.198) Since the models are linear, this implies that fine-tuning gets all examples in the span of the training examples correct as well. Since Pg has density, and the number of training examples n is at least as large as the ID subspace dimension m, the training examples span the ID subspace almost surely, so fine-tuning gets every example in & & correct almost surely, giving us:
B x=
(t B =0
(A.199)
Linear probing gets positive ID loss: Lemma A shows that the ID error of linear probing is greater than zero under the same assumptions as this Proposition, so
Lid (1 BB)) >o)>0.
(A.200)
which finishes the proof.
We now state and prove the Lemmas that we used to lower bound the ID error linear probing.
Lemma A gives conditions for when the projection F w of a vector w is not contained in the projection Range Eg) of the column space a matrix Eg.
Lemma A.18. Let w E Rdd be a vector and F E TRdx Egg ERix FFaug E Rd+x(++1) have orthonormal
columns, with Range( (aagg) = (pan([w]] Range Eo)). If > k, we have:
F Eaug is full rank
(A.201)
(0) F. Eaug has higher rank than F Eo
(A.202)
10 FT w of Rangel FT Eo)
(A.203)
33

---

Published as a conference paper at ICLR 2022
Proof. The proof of (a) is clear- FF Eaug E Ixx((++)) has rank + (since it is full rank and m 2 +++1,, but F Eaug E Ru*k has rank at most k and is therefore lower rank. The assumption that m> k is crucial here.
For (b), let an, - OK be the columns of En, which form a basis for Range Eo). Then F a1, ... FFaa, ag. FTw spans Range(F" Eaug), while F a1; ... F as spans Range(F1 (FT Eo). So (notice the first list of vectors has an additional F (')) this means that dim(Re [Range ( (FT Faug)) #dim IF Fa)) iff FT w is linearly independent from the rest, that is, F w & Range FT Eg). Note that the rank of a matrix is the dimension of its range (column space), that is, dim (Range A)) rA) this is what wanted to show.
The next Lemma says that if the projection FT w. of the optimal linear model w, onto the ID subspace S, is not contained in the projection Range(F" Eo) of the features, then linear probing incurs non-zero ID error.
Lemma A.19. In the linear overparaaetericedsettiin under the ID subspace assumption, Ff w. & Range(F" F Eo), then Lia Lia(vip Bo) > O, where Eo E R and F E RRd have orthonormal columns that form basis for the feature rowspace Ro rowspace (B)) and ID subspace S respectively.
Proof. We prove the contrapositive. Suppose (1p BBo)=O. =0 This means that:
Tia (ip 00 Bo) = E P-Pa
[(v_ - 0o Up BB 11=
(A.204)
Since the squared error is always non-negative, this means that uip Bor w.. x: almost surely when I ~ Pia (recall that we defined w, = B. v.). Recall Pa is defined as: first pick E Pz (which has density) and then output x Fz. Since P. has density, this implies that get all examples in the ID subspace S correct:
Up
00 B w. r for all x S.
(A.205)
Since the columns of F' form an orthonormal basis for S, this gives us (since each column of F in S);
1p B
(A.206)
Note that the rows of Bo also form orthonormal basis for Ro just like the columns of Eo. So we can choose v with v Egj
up Bg- Then we have:
v T JT F= F#F Eou=FTu
(A.207)
F E RRagee(F" Eo),
(A.208)
where we took the transpose of both sides in the first step. This finishes the proof of the contraposi- tive.
Finally, Lemma A. combines Lemma A 1 and Lemma A. to give more interpretable condition for the ID error of linear probing: when the ID subspace S has some components along the optimal linear model w, and the feature rowspace Ru, then linear probing has non-zero error. This is measured in terms of the principal angle cos max (Rag. S) between the ID subspace S and Raug which the illustrative example,
span of Ro combined with w.. This angle will typically be non-zero- from Lemma A. we have that this angle will be non-zero almost surely the ID subspace S is a uniformly random subspace.
Lemma A.20. In the linear overparameterized setting, under the ID subspace assumption, let R. rowspace (Bo). and Raug Span([ (pan(([.,} Ro). If w. & Ry and cos max (Rau.. S) > then Lid ( (B.) 0.
Proof. After bit of setup, the proof simply combines Lemma A.I and Lemma A. 19 If W, & Ro- then Raug has dimension + 1. Let Eaug E (Rdx(++1) F E Rdxm have orthonormal columns which form a basis for Raug and S respectively. We assumed cos 0 mmaxmmaagg S) = amin ( [FT Faug) > which means that FT Eaug is full rank. The ID subspace assumption assumes that m > k. So from Lemma 18, F "w & Range(FF Eg) where Eg e dR has orthonormal columns that form basis for Rg. Then from Lemma A.19, [na Bo) >>>00000000
34

---

Published as a conference paper at ICLR 2022
A.7 LP-FT
We start by showing a simple proposition, that if the initial feature extractor is perfect, then linear probing recovers the optimal weights.
Proposition A.21. In the overparameterized linear setting, let R rowspace(Bo) If Bo B,, and cos () > 0, then (%p ,Bo) = for all t.
Proof. We first show that because cosomax R2,)) > O, the training loss for linear probing is strongly convex. Recall that the training loss is:
[ V-Y
(A.209)
Linear probing keeps B fixed as Bo= B, and only tunes so we are interested in the Hessian of the loss with respect to a evaluated at v.B.:
(A.210)
For strong convexity, it suffices to show that the min singular value of the Hessian is bounded away from 0 by a constant. Recall the definition of cos R). For some F whose columns form an orthonormal basis for S, we have (since the rows of B. form an orthonormal basis for R):
0h = = cos@ R,S) >0
(A.211)
Note that F. F k-by-n matrix, so if the k-th singular value positive must be full rank. Since the columns of X span F (since we defined F to be such that the columns of F are an orthonormal basis for S, i.e. the rows of X), this means B, X rank But that means the Hessian (B, ) is rank & well. So the linear probing loss is strongly convex.
Since the loss is strongly convex, there is a unique minimizer, and gradient flow converges to that, However, since are in the well-specified setting, we know the training loss is:
[v. IX 8- XB, 1.
(A.212)
So c = v. achieves 0 loss and must be the (unique) minimizer. Therefore we have shown that linear probing converges to the unique minimizer dip U.. which attains O loss, as desired.
Note that the entire proof works if if Bo= = B, for some rotation matrix U. In that case, the Hessian becomes 2U (B,X (B.X TUT which is still rank k, since multiplying by square rotation matri- does not change the rank. In this case, the minimizer of the loss is v= Uv,, since (UB,) (Uv,) B_ U. So linear probing converges to ufp Uv, which achieves 0 loss, as desired.
Restatement Proposition some rotation U. Let Ro
3.6L Given perfect pretrained features Bo UB, for rowspace(Bpp Bo). Under the non-degeneracy conditions
costmax (Ro,S) 3) + 0,c0s0 maxoss
Lool Bn (1) Utt (1)) if ~N(0077 randomly initialized (FT);
(A.213)
Vt,Lood (t) on (t)) 00 vo is initialized to dip (LP-FT).
(A.214)
Proof. We first use Proposition A.21, which in the proof showed still works if Bg UB, for some rotation matrix U (which doesn't 't have to be identity). We get that di = -v., Then we have BJ op = v.
We now just show that the gradients with respect to the training loss L at (ug BB) is O, so gradient flow does not update the parameters at all.
The training loss is:
[ v-XB, v1,
(A.215)
The derivative with respect to v is:
0,L(v.B)-2BX' (XB
(XX -ー v- UU,
(A.216)
Then since BJJ up =B, v., have:
0, 1. Bo) =0
(A.217)
35

---

Published as a conference paper at ICLR 2022
Next, the derivative with respect to B is:
0g 2 (-B(XXBBBBBBBB o-x,,,,) v,)'x
(A.218)
Then since Bi
Bij upp - =B, Uxs we have:
On T BBo= =0
(A.219)
So since both the derivatives are we have (+ 0 and 0g Btt 0. which means the parameters don't change all-at all times & we have Ust (t) = Uv. and (+( U B, which gives us zero OOD loss: Lood (t) (+(()) = as desired.
B MORE INFORMATION ON EXPERIMENTS
In this Appendix, we include more details on the datasets, pretraining methods, and adaptation methods. We also include the OOD accuracies for fine-tuning and linear probing if we early stop and choose the learning rate based on OOD data, where we see that linear probing still typically better than fine-tuning OOD. Finally, we include results for additional basclines, pretraining models, and conclude with a discussion about the effective robustness of LP-FT.
B. OVERVIEW OF DATASETS
We first give overview of the datasets used in our paper, before diving into more details of the exact training procedures (e.g., number of epochs, pretraining method, etc). The datasets we use are:
Domain (Peng et all 2019) is a standard domain adaptation dataset. Here, our ID dataset contains "sketch" images (e.g., drawings of apples, elephants, etc). and the OOD dataset contains "real", "clipart", and "painting" images of the same categories. We use the version of the dataset from Tan el all (2020).
Living-17 and Entity-30 are sub-population shift datasets from the BREEDS bench- mark (Santurkar all (020). In Living-17 the goal is to classify an image one of 17 animal categories such as "bear" example, the ID dataset contains images of black bears and sloth bears and the OOD dataset has images of brown bears and polar bears. In Entity-30 the goal to classify an image as one of 30 entities such as "fruit" or "insect".
FMoW Geo-shift is adapted from the satellite remote sensing dataset Functional Map the World CChrstteretall 20TS; ethll (02I). The goal is to classify a satellite image into one of 62 categories such "impoverished settlement" or "hospital", Our ID dataset contains images from North America, and the OOD dataset contains images from Africa and Europe.
CIFAR-10 + STL is standard domain adaptation dataset Frencheraa 2018), where the ID is CIFAR- 10 (Krizhessky, 2009). and the OOD STL (Coates all 201 The task is to classify image into of 10 categories such as "dog", "cat", or "airplane"- usual,
we remove the "monkey" class in STL since CIFAR- 10 has no "monkey" images.
(Recht all 2018) is dataset collected using very similar protocol to CIFAR-10, and the authors describe it as "a minute distributional shift". The hope is that a classifier trained CIFAR-10 gets high accuracy on CIFAR-10.1.
CIFAR-10 CIFAR-10.1
ImageNet-IK (Russakovsky et all 2015) is a large scale dataset containing over a million images, where the goal is to classify image into of 1000 categories such as "Yorkshire terrier", "Labrador retriever", "acoustic guitar", "library", "school bus". etc. We fine-tune on ImageNet as the ID dataset, and evaluate on four standard OOD datasets: ImageNetV2 (Recht et all 2019), ImageNet-R (Hendryeks et al, 2020), IntgeNettA (Hendrycks etal] 20195), and ImageNet-SSktch (Wang al] 2019).
B.2 DATASET AND METHOD DETAILS
We use a diverse range of datasets and pretraining strategies.
CIFAR-10 > STL: We fine-tune or linear probe on CIFAR- (Kizhevsky, 2019) and test on STL (Coates eall I) This is benchmark used in domain adaptation papers (Crench aLal 2018). CIFAR- 10 and STL share classes, so we follow the common practice of omitting the unshared class in STL (which is the 'monkey' class) when reporting accuracies. We use a publicly available MoCo-v2 ResNet-50 checkpoint pretrained on unlabeled examples from ImageNet-1k (Russakovsky et all 2015), and fine-tune for 20 epochs.
36

---

Published as a conference paper at ICLR 2022
Table 3:
OOD accuracies with 90% confidence intervals over 3 runs, for each of the three OOD domains in the split of DomainNet used by Dan (2020); Prab (202TT. LP does better than FT across the board, and LP-FT does the best.
Real
Painting
Clipart
Fine-tuning
55.29 (0.52)
50.26 (0.98)
60.93 (2.15)
Linear probing
87.16 (0.18)
74.50 (0.58)
77.29 (0.12)
LP-FT
86.82 (0.51)
75.91 (0.73)
79.48 (0.90)
DomainNet: We use the dataset splits in Tan et all (2020) which is also used by follow-up work, c.g., in Prahhu
(202T). This is different from the original version of the DomainNet dataset (eeng ct al, 2019), specifically Tan et all (2020) note that some domains and classes contain many mislabeled outliers, so they select the 40 most common classes from the 'sketch', 'real", 'clipart' and 'painting' domains. We use the 'sketch' domain as ID, and all other domains (" ('real', 'clipart', 'painting') ) as OOD, and the main paper we report the average accuracies across the OOD domains. In Table E we see that the same trends hold for euch of the three OOD domains. We use a CLIP (Radlord ef all 202T) pretrained ResNet-50 model, and fine-tune for 50 epochs (since this is smaller dataset).
Living-17 17 and Entity-30: We use publicly available MoCo-v2 ResNet-50 checkpoint pretrained on unlabeled examples from ImageNet- (Russakovsky et all 2015), and fine-tune for 20 epochs. Note that Living- 17 and Entity-30 are subpopulation shifts derived from ImageNet, but the pretraining is done on unlabeled data and does not see any OOD labels, following the pretraining and fine-tuning strategy in Cat (2021). Entity-30 is relatively large dataset that contains around 140K training examples.
FMoW Geo-shift: We adapt the version of the dataset from (Koh all (02II). We use training data from 'North America' fine-tune or linear probe, and then evaluate on validation data from Africa and Europe. We use MoCo-Tr TPPCoTTPon (Ayush et all, e020) checkpoint, pretrained on unlabeled FMoW satellite images. We fine-tune for 50 epochs here since the ID) training dataset is smaller (around 20K examples).
CIFAR-10 + CIFAR-10.1 (Recht 2018): We follow the same protocols as CIFAR-10 + STTL, except we test on 10.1
ImageNet: we linear probe or fine-tune on ImageNet (Russakovsky et al, 2015), and evaluate on ImageNetV2 (Recht et all 2019), ImageNet-R (Hendrycks et all 202(). ImageNet-A (Hendrycks et all 20195), and ImageNet-Sketch (Wang et all 2019.. We use a CLIP pretrained ViT-B/16 (vision transformer), the largest publicly available CLIP model (Radtord 2)2I). We ran fine-tuning for 10 epochs, linear probing for 10 epochs. To equalize the runtime for LP-FT, ran the linear probing stage for 5 epochs, and then the fine-tuning stage for 5 epochs. We used a batch size of 128 for all methods.
Tuning for ImageNet experiments. We swept over three learning rates for fine-tuning (0.0001, 0.0003, 0.001) and linear probing (0.01, 0.03, 0.1)-as as is standard we use larger learning rates for linear probing. For LP-FT, we swept over learning rates (0.01, 0.03, 0.1) for the 5-epoch linear probing step. We took the run that had the best ImageNet (ID) validation accuracy, and then swept over 3 learning rates (0.00001, 0,00003, 0,0001) for the 5-epoch fine-tuning step-we use lower learning rate for LP-FT since the experiments on the other datasets suggested that the optimal learning rate that maximizes ID validation accuracy for LP-FT is smaller. We did not find the comparisons to be particularly sensitive to learning rate choice.
Augmentations for ImageNet experiments. We used augmentations for fine-tuning, and no augmentations for linear probing, following Kornblith et all (2019). This might raise question of whether linear probing and LP-FT do better OOD because of the lack of augmentations. So as an ablation we also tried fine-tuning without augmentations, however that led to worse accuracy (than -uninu with augmentations) both ID and OOD. We now give details on the preprocess- ing and augmentations that used. On ImageNet, for linear probing and LP-FT, we used no augmentations- we just resized each image so that the smaller side has size 224 with bicubic interpolation, and then center-crop to 224-by-224 image. For fine-tuning, used augmentations:
37

---

Published as a conference paper at ICLR 2022
4able4: OOD accuracies with 90% confidence intervals over runs, when fine- tine-tuning gets to choose learning rate and early stop, and linear probing gets choose Eg regularization weights, on OOD data. We see that linear probing still typically does better OOD (the only flip from before is on FMo
CIFAR-10.1
STL
Ent-30
Liv-17
DomNet
FMoW
FT
92.27 (0.36)
85.97 (0.38)
64.09 (0.19)
78.63 (0.53)
59.43 (2.49)
40.23 (3.12)
LP
82.67 (0.22)
86.53 (0.01)
69.15 (0.13)
82.39 (0.14)
79.91 (0.24)
37.12 (0.01)
ImNetV2
ImNet-R
ImNet-Sk
ImNet-A
Average
FT
7-)5
(-)
(-)
27.8
61.3
LP
69.7
70.9
(-)
46.1 (-)
67.1
Table 5: In-distribution (ID): Average distance that features move before and after fine-tuning or LP-FT, multiplied by 100 make things easier to read. For linear probing the numbers all O, since the features are not tuned. As predicted by our theory, we see that features for ID examples (this table) move more than features for OOD examples (Table (.. Both sets of features change substantially less for LP-FT. As usual we show 90% confidence intervals over three runs.
CIFAR-10
Entity-30
Living- 17
DomainNet
FMoW
FT
2.23 (0.03)
3.05 (0.02)
1.88 (0.01)
207.6 (12.31)
4.87 (0.15)
LP-FT
0.07 (0.00)
0.03 (0.01)
0.11 (0.01)
0.19 (0.03)
0.57 (0.19)
specifically we use RandomResizedCrop in Torch Vision, with the default arguments and setting the size the crop to 224, and then apply a random horizontal flip.
Notes on pretrained model choice. We note that our results say that the pretraining has to be good (e.g., at least get reasonable accuracy ID) for linear probing to outperform fine-tuning OOD. So, for example, we use model pretrained on unlabeled satellite images for the satellite image dataset- if pretrain the model on ImageNet, DomainNet we use a CLIP pretrained model, which is pretrained on the very large WebImageTText dataset, and sees variety of photo and sketch like images. Pretraining on ImageNet alone does not lead to high accuracies on DomainNet (features are not very good), so we do not necessarily expect linear probing to outperform fine-tuning with these lower quality features (for example, see the MoCo ablation in our main paper where we used a worse pretrained model, and fine-tuning did better OOD).
expect that fine-tuning might do better. Similarly, for
Sanity check of fine-tuning implementation. As a sanity check of our implementation, fine-tuning did substantially better than training from scratch all datasets (both ID and OOD) and matched existing fine-tuning numbers where available (e.g. ResNet50 on CIFAR-10 (Chen el all 2020F) and Entity-30 (Cai all ()2I)). Fine-tuning and linear probing also both do substantially better than training from scratch, ID and OOD, across the datasets. For example, on Living-17, 17, training from scratch gets 89.3% ID and 58.2% OOD (Santurkar at 2020) which is over 5% worse ID and nearly 20% worse OOD, than all the adaptation methods. For reference linear probing gets 96.5% ID and 82.2% OOD, and fine-tuning gets 97.1% 77.1% ID and 77.8% OOD. This is even though training from scratch was run for 300 epochs, which is 15 times longer than fine-tuning and LP-FT.
B.3 TARGET EARLY STOPPING
In the main paper, one ablation we mention early stopping each fine-tuning method and choose the best learning rate based on target validation accuracy. As expected, fine-tuning does improve little, but linear probing (average accuracy: 67.1%) is still better than fine-tuning (average accuracy: 61.3%). Table H shows the full results for all datasets.
B.4 FEATURE CHANGE
We examine how much the features changed for ID and OOD examples in each dataset. Specifically, for each dataset, for each input example in the held out validation set, computed the Euclidean distance of the ResNet-50 features before and after fine-tuning. We averaged these numbers across the dataset, showing the results for IDD validation examples in Table El and for OOD examples in Table R
38

---

Published as a conference paper at ICLR 2022
Out-of-distribution (OOD): Average distance that features move before and after fine-tuning or LP-FT, multiplied by 100 to make things easier to read. For linear probing the numbers are all O, since the features are not tuned. As predicted by our theory, we see that features for ID examples (Table ( move more than features for OOD examples (this table). Both sets of features change substantially less for LP-FT. As usual we show 90% confidence intervals over three runs.
Table 6:
STL
Entity-30
Living- 17
DomainNet
FMoW
FT
1.70 (0.04)
2.60 (0.02)
1.67 (0.01)
159.97 (16.23)
5.62 (0.30)
LP-FT
0.04 (0.00)
0.02 (0.00)
0.09 (0.01)
0.18 (0.02)
0.54 (0.17)
Table 7: ID and OOD accuracies on Living-17 using a CLIP ResNet-50 model pretrained on the WeblmageTexx dataset, instead of unlabeled ImageNet examples. Similar findings hold-here fine-tuning does similarly to linear probing ID, but does worse than linear probing OOD. LP-FT does better than both ID, and closes 86% of the gap OOD. As usual we show 90% confidence intervals over three runs.
ID
OOD
LP
(0.2(
78.6 (0.5)
FT
(0.1)
67.3 (0.8)
LP-FT
95.6 (0.6)
77.0 (0.6)
The feature distortion theory predicts that the features for ID examples change more than for OOD examples. This bears out in 9 out of 10 cases, that is all cases except for FT on FMoW. To see this, compare each cell Table B with the corresponding cell in Table 0 the former higher in 9 out of cases.
The feature distortion theory says that this large feature change is caused because the head is randomly initialized- the head needs to be updated by a large amount, the feature extractor is also updated a lot because the updates are coupled. Our theory predicts that if the head is initialized via linear probing then the feature extractor should change a lot less for both ID and OOD examples. As predicted by the theory, across all the datasets in Table 5 and Table & the features change lot less for LP-FT than for FT. For example, on CIFAR-10, 10, the features change 30x less for LP-FT than for FT.
These results suggest that fine-tuning underperforms OOD, and LP-FT does well ID and OOD, for the reasons predicted by the feature distortion theory.
B.5 ADDITIONAL. ARCHITECTURES, FINE- TUNING METHODS
The main contributions of our paper are conceptual understanding and theory. However, to strengthen the empirical investigation we ran two additional models (a CLIP vision transformer and CLIP ResNet-50), as well as three additional fine-tuning heuristics. We focus on the Living- 17 dataset be- cause some of these ablations require lots of compute and can take a long time to run on all the datasets.
Architectures and pretraining source: In the main paper, we showed results when initializing with MoCo-v2 ResNet-50 model pretrained unlabeled ImageNet examples. Here examine how the results change when we 1. Use ResNet-50 model pretrained on CLIP's WeblmageTex dataset (Table W. and, 2. Use much larger vision transformer model (ViT-B/16) pretrained on CLIP's WeblmageTText dataset (Table ttis is the largest publicly available CLIP model at the time of writing. We see that similar findings to our main paper probing ID, but does worse than linear probing ('underperforms') OOD, Finally, LP-FT does better than both methods ID, and closes most (75%-90%) of the gap OOD.
fine-tuning does better than linear
These results are from early stopping on ID validation data. If we early stop on OOD validation data, LP-FT achieves 87.9 +..4% OOD accuracy, and LP gets 88.3 68.3+0.224 OOD accuracy and here there no statistically significant difference between the two. On the other hand, even if we early stop on OOD validation data, fine-tuning gets 84.4 OOD accuracy which is lower.
Fine-tuning heuristics: Transfer learning (initializing with a pretrained model, and then adapting to downstream task) is the standard way to build modern MI. models, because it improves accuracy
39

---

Published as a conference paper at ICLR 2022
Table 8: ID and OOD accuracies on Living-17 17 using CLIP ViT-B/16 (Vision Transformer) model pretrained on the WebImageText dataset, instead of unlabeled ImageNet examples. This the largest publicly available CLIP model that we could find. The same findings fin-ttnn-uuning does better than linear probing ID, but does worse than linear probing OOD. L.-FT does better than both ID, and closes 75% of the gap OOD. As usual we show 90% confidence intervals over three runs.
ID
OOD
LP
97.5
87.6 (0.5)
FT
97.8
81.5 (2.1)
LP-FT
98.0 (0.0)
86.1 (0.1)
and speeds up training. Since this paradigm is so widely used, there are many heuristics people use when training their models (as mentioned in the main paper, LP-FT has sometimes been used as a heuristic well, although not in the context of OOD). We showed that LP-FT is one way to do well ID and OOD, but we hope that our theory leads to even better fine-tuning algorithms.
In this section, we compare LP-FT with additional fine-tuning heuristics: using a larger learning rate for the head layer, regularizing the features towards their original values, and side-tuning (Zhang etall 202() where we freeze the features but add a side-network.
The intuitions from our theory suggest two other potential ways to improve OOD accuracy: 1. We could use higher learning rate on the linear layer, that the linear layer learns quicker and the features do not get as distorted, and 2. We could regularize the weights of the feature extractor towards the pretrained initialization, prevent feature distortion. These heuristics have been used in prior work on fine-tuning as well, for example method 2 corresponds to L2-SP in (Lictall 2018).
We run these two approaches on Living-17. 17. For approach (1), we use a 10x higher learning rate for the linear layer, and for approach (2) we regularize the Euclidean distance between the current feature extractor weights (so ignoring the linear head) from the pretrained weights, multiplying by a hyperparameter A. We grid search over the same learning rates as fine-tuning for both methods, and in addition for (2) we grid search over À E /1.0,0.1,0.01. 0,0.0.00000000000 J. so this amounts to sweeping over 30 hyperparameters as opposed to just 6 for fine-tuning and I.P-FT. For each hyperparameter configuration we run 3 replication runs with different seeds to reduce the estimation variance, and early stop and model select using ID data just like for fine-tuning and LP-FT. Just like for fine-tuning and LP-FT, we use a cosine learning rate decay and train for the same number of epochs. Indeed, we find that both (1) and (2) are able to close part of the OOD gap between fine-tuning and linear probing. However, LP-FT does better than both methods ID and OOD. The full results in Table
We also compare with another method, (3) side-tuning (Zhang al, 2020). Side-tuning freezes the pretrained features g but trains another 'side' model s(z). and then outputs v (g(x) + h(x)), where the head u and the parameters of the side model are tuned. The intuition for trying this is that side-tuning also preserves the pretrained features which likely reduces feature distortion. In the supplementary of Zhang et al; (202() they use ResNet-50 for both the original model and the side model in their vision experiments, so we do the same. We sweep over twelve learning rates (3-10-5,11 10- 4,3- 10-4 ,.... 1.0,3.0, 0.0, 10.0), with three replication runs with different seeds for each learning rate. Just like for fine-tune fine-tuning and LP-FT, we use a cosine learning rate decay and train for the same number of epochs, and we early stop and model select using ID validation data. We checked that the best learning rate was not at the boundary of the grid search. On OOD, side-tuning (81.00 (8100%) improves over fine-tuning (77.7%). However, side-tuning doesn't 't do as well IDD L.--T did better ID and OOD. This could be because side-tuning does not get refine the pretrained features for the ID task-while the side-network is powerful enough to learn good features, it is initialized randomly and effectively trained from scratch, so it might not be able to learn these good features the limited sized training dataset (around 40K examples). The results are also in Ta
We also include results for training from scratch in Table 5 these results are from Santurkar et all (2020) Note that training from scratch was done for 450 epochs, whereas fine-tuning was done for 20 epochs. As a sanity check, all the fine-tuning methods and linear probing do substantially better than training from scratch, both ID and OOD.
40

---

Published as a conference paper at ICLR 2022
Table 9: ID and OOD accuracies on Living-17 including three additional fine-tuning heuristics, where we (1) Use 10x larger learning rate for the head, or (2) Regularize the Euclidean distance of the feature extractor weights to the pretrained initialization, and (3) side-tuning where we freeze the pretrained model but add side network that is fine-tuned. As a sanity check, all methods do better than training from scratch ID and OOD, and we show 90% confidence intervals over three runs. As per the intuitions from the feature distortion theory, these methods do mitigate feature distortion to some extent and improve OOD accuracy over fine-tuning. LP-FT does better than all methods ID and nonehoons,,,sss, we believe that LP-FT is just the first step and hope that our theory can be used to inspire derive better algorithms.
ID
OOD
Scratch LP FT FT (10x Linear)
(1.3)
(2.4)
(0.1(
82.2 (0.2)
(0.1)
77.7 (0.7)
(0.2
80.4)
FT (regularized)
97.1
80.0 (0.4)
Side-tuning
95.5
81.0 (0.7)
LP-FT
97.8
82.6 (0.3)
B.6 DISCUSSION EFFECTIVE ROBUSTNESS
LP-FT gets higher OOD accuracy than fine-tuning, but it sometimes gets higher ID accuracy as well. laor (2020) and Miller et all (2021) show that OOD accuracy can often be correlated with ID accuracy, and suggest examining the effective robustness: intuitively the extra gain in OOD accuracy than can be predicted from improved ID accuracy alone. Is LP-FT simply better in-distribution, does it have higher effective robustness robustree as well?
We start out by noting that linear probing clearly has higher effective robustness in most of our datasets. Linear probing does worse than fine-tuning ID so based on the effective robustness framework we would expect it to worse than fine-tuning OOD as well. However, linear probing does better than fine-tuning OOD and therefore has higher effective robustness.
87.5
85.0
82.5
000 Accuracy
80.0
77.5
75.0
72.5
70.0
67.5
95.0
95.5
96.0
96.5
97.0
97.5
98.0
ID Accuracy
Figure 3:
We plot the OOD accuracy against ID accuracy on Living- 17 for the three methods we con- sider, when start from three different pretrained models (CLIP ResNet-50, CLIP VIT-B/16, MoCo- V2 ResNet-50). The line for linear probing and LP-FT lie above fine-tuning which suggests that they have higher effective robustness. Each point is produced by averaging over three random seeds.
The solutions found by LP-FT also appear have higher effective robustness than fine-tuning, be- cause when they have similar ID accuracy, LP-FT does much better OOD. For a few pieces of evidence:
1. On CIFAR-1O 10 + STL, there is no statistically significant difference between FT and LP-FT ID, but LP-FT gets 8% higher accuracy OOD in Table Ø
2. If we look at checkpoints earlier in training for CIFAR-10 -+ STL exactly equalize ID accuracy and compare OOD accuracies. In-distribution, LP-FT and FT both get 97.2% accuracy, but OOD, L.--T (90.2%) is much better than FT (81.8%).
41

---

Published as a conference paper at ICLR 2022
3. Finally, in Figure Bl we plot the OOD accuracy against the ID accuracy for fine-tuning and LP-FT on Living-17. 17. We plot these for three different pretrained models (CLIP ResNet-50, CLIP ViT-B/16, MoCo-V V2 ResNet-50). We see that the ID-OOD line for LP-FT is above the line for FT indicating effective robustness.
Note that higher effective robustness does not mean method is better. For example, a method A can have higher effective robustness B by doing lot worse in-distribution even when they have the same OOD accuracy. In this case, A is clearly inferior since it does worse ID and same OOD, but has higher effective robustness because of its worse ID accuracy.
We believe the finding that linear probing and LP-FT has higher effective robustness than fine-tuning when the distributon shift is large is particularly interesting because Daon (2020) and Miller at (2022I show that it is uncommon for methods to have higher effective robustness. In our case linear probing and LP-FT appear consistently have higher effective robustness which suggests that with good transfer learning methods we can get both high in-distribution accuracy and higher effective robustness.
C ADDITIONAL RELATED WORK
Theoretical analysis of overparameterized models. Modern deep learning presents an interesting paradigm for theoretical analysis where the number of parameters is much larger than the number of training points. The model class is highly expressive and several solutions obtain zero training loss even in the presence of noise. Such overparameterized models have received a lot of interest recently especially with a focus on understanding "benign overfitting" or the phenomenon where fitting noisy training data
loss leads classifiers that generalize well. By analyzing different
linear overparameterized settings Belkin et all (2019); Haste et all (2011)) Barrlett of all (2019);
Muthukumar
(2020): Mc1 Montanari (2019); Bibas all (201)) study various statistical
properties such as the "double descent curve" in addition to benign overfitting. One important aspect of overparameterized models that there is no unique minimizer of the training loss. We need some inductive bias which is typically implicit via the optimization procedure. Prior works study the statistical properties of the explicit inductive bias of minimum norm interpolation. In contrast, we study the effect of gradient based optimization from particular pretrained initialization where effectively capture the exact implicit inductive bias of gradient based fine tuning.
42


Reviews:
[{"review_id": "YX8sLU74N8b", "reviewer": "Reviewer_VT2b", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. the observation that fine-tuning performance worse on the OOD test is new and interesting \n\n2. the proposed method LP-FT is simple yet effective\n\n3. theoretically analysis is further provided to show why LP-FT works\n\nCons:\n\n1. some important baselines are missing like [a-c], which should be discussed and compared\n\n[a]. Guo, Yunhui, et al. \"Spottune: transfer learning through adaptive fine-tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[b]. Zhang, Jeffrey O., et al. \"Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020.\n\n[c]. Ge, Weifeng, and Yizhou Yu. \"Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n2. why different pre-trained models are utilized for different ID and OOD pairs? In fact, the authors could use different models for each ID and OOD pair.", "summary_of_the_paper": "This paper studies the problem of how to fine-tune a pre-trained model and obtain better results for both ID and OOD. Two methods, fine-tuning and linear probing, are investigated and compared, then a new two-step variant called LP-FT is derived. Results further verify that LP-FT obtains the best performance for ID and OOD tests compared with FT and LP.", "summary_of_the_review": "This paper discovers that vanilla fine-tuning performs worse than linear probing for the OOD tests and then develops a new method combining these two techniques sequentially. Results on several datasets verify its effectiveness. Even there exists some minor problems, this paper is interesting and easy to read. Thus, I tend to give the \"weak accept\" score.\n\n-----POST REBUTTAL-----\n\nThe authors have addressed my concerns. Thus, I increase my score from 6 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "To9Q8g8Nhrc", "reviewer": "Reviewer_5UET", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n1. This paper explores two increasingly impactful research directions, fine-tuning pre-trained models and generalization under distribution shifts. I believe their results, both theoretical and empirical, would be of interest to many in the community\n\n2. The theoretical results, despite studying a very simple setting that is unlikely to be used in any real experiments, offer intuitions that transferred well to practical results.\n\n3. The paper is clear and well written.\n\n**Weaknesses:**\n\n1. Experiments could be more comprehensive. For instance, while this paper analyses several distribution shifts (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR->STL, CIFAR 10.1 and FMoW), it was surprising that distribution shifts like ImageNetV2, ImageNet-R, ImageNet-A, ObjectNet and ImageNet Sketch were not considered by this work. Moreover, only a single architecture, ResNet-50, is used for the experiments. As they stand, it is unclear whether the results from this paper would hold at larger scales.\n\n2. This work overlooks simple baselines. For instance, given the finding that end-to-end fine-tuning significantly changes the weights of the backbone (which arguably hurts OOD performance), it would be natural to consider having a smaller learning rate for the backbone, and a higher one for the untrained final linear layer. It is not uncommon to do so in practice. Moreover, authors could have explored regularizing the difference of the weights of the backbone to the original weights, encouraging them to not be changed too much. Finally, the authors do not mention the fact that models like CLIP allow the possibility of starting with a good set of initial weights for the final layer, as they can be used in a zero-shot setting. If the intuitions presented in this paper hold, this prevent the \"distortion\" that happens when the last layer is far from the optimum.\n\n3. The theoretical results are disconnected from realistic settings. Some clear examples are the assumption of a two-layer network, a squared error loss (while the de facto standard for classification is cross-entropy), and considering the worst case loss over distributions of bounded norm.\n\n4. Apart from the theoretical results, there is not a lot of novelty introduced by this work. The two-stage fine-tuning strategy where end-to-end fine-tuning follows linear probing is commonplace and thought in introductory courses in deep learning, e.g. in the first lesson of https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson1-pets.ipynb\n\n5. Comparison to previous work is lacking. In particular, explicit comparisons of LP-FT with other recent robustness-oriented fine-tuning methods would greatly strengthen this work [1-4, among others]. \n\n\n**References:**\n\n[1] Wortsman, Mitchell, et al. \"Robust fine-tuning of zero-shot models.\" arXiv preprint arXiv:2109.01903 (2021).\n\n[2] Aghajanyan, Armen, et al. \"Better fine-tuning by reducing representational collapse.\" arXiv preprint arXiv:2008.03156 (2020).\n\n[3] Jiang, Haoming, et al. \"Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.\" arXiv preprint arXiv:1911.03437 (2019).\n\n[4] Zhu, Chen, et al. \"Freelb: Enhanced adversarial training for natural language understanding.\" arXiv preprint arXiv:1909.11764 (2019).", "summary_of_the_paper": "This paper explores how different strategies for fine-tuning affect in- and out-of-distribution performance. The authors contrast linear probing (updating only the parameters of the final linear layer), end-to-end fine-tuning (updating all parameters of the model) and a two-stage approach, where linear probing is followed by end-to-end fine-tuning. While end-to-end fine-tuning typically improves in-distribution performance, the authors show that it can also underperform linear probing out-of-distribution. The paper theoretically analyzes the tradeoffs in a simplified scenario with two-layer networks, finding that end-to-end fine-tuning can \"distort\" pre-trained features. Their experiments on a number of datasets including CIFAR, WILDS-FMoW and others, confirm the intuitions from their theory. The proposed mitigation strategy, a two-stage fine-tuning approach where end-to-end fine-tuning follows linear probing is found to be beneficial, especially out-of-distribution.\n\n\n**Update:** The authors addressed most of the concerns raised by this and other reviews, and I am raising my score accordingly.", "summary_of_the_review": "Overall, while there are several points of concern that could strengthen the paper, I believe the results and theory presented by this work would still be of interest to many in the community, so I recommend its acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IgPepTnRgp", "reviewer": "Reviewer_Gpdo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of this paper is the extensive and detailed toy and benchmark experiments. The reasoning and intuition of why fine-tuning underperforms on OOD data are well explained and discussed throughout the paper. The suggested solution by combining linear probing and fine-tuning also has good performance. ", "summary_of_the_paper": "This paper discovers an interesting behavior of model fine-tuning: the performance is worse compared to linear probing on OOD data (i.e., data from other domains), especially when the distribution shift between inner distribution and out of distribution are big. The explanation provided in the paper is that fine-tuning distorts the feature representations, overfits on inner distributions, and thus has a higher error on OOD data. The authors also provide a simple solution to this issue by fine-tuning with a classification head initialized from linear probing and had better results in all the benchmarks they have in the paper. ", "summary_of_the_review": "Although the solution proposed in this paper is not fancy, the reasoning, intuition, and experiments are well written. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H4ZGiZVZtvs", "reviewer": "Reviewer_VgHa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has many strengths:\n- It is not well known that FT often outperforms LP OOD -- the empirical and theoretical study of this phenomenon may be very important to the community.\n- LP-FT outperforms LP and FT on many distribution shifts.\n\nIn addition, the paper has some weaknesses with associated actionable items:\n\n1) The main weakness of this paper was contextualizing LP-FT. In the abstract and introduction, it seems as though LP-FT is a method that is being introduced by this paper. For instance, the abstract states \"our analysis suggests the simple two step-strategy of linear probing then full fine-tuning\". It is not until the final page of the main paper that the related work section states \"LP-FT has sometimes been used as a fine-tuning heuristic\". Moreover, this is presented without any citations. I would very much appreciate clarity on this issue, is LP-FT something that has been used in the past? If so, by which papers and why?  Does this reference that in fine-tuning, sometimes the hyperparameters in the final layer are decoupled from those used for the encoder [1] and performance improves? As shown by [2, 3], performance improvement ID lead to performance improvement OOD, and so this could be an alternative explanation for  the OOD boost of LP-FT that is orthogonal to the authors theory. For instance, in the effective robustness framework of [2], do the solutions found by LP-FT exhibit more effective robustness than the FT solutions?\n\n2) Section 4 is hard to follow in its current form as it is not clear exactly from where numbers are derived. For instance in the first line of the  \"Results.\" paragraph of 4.2, where is the 76.5 number from?\n\n3) One of the networks studied by this paper is CLIP, for which a zero-shot final layer can be constructed (e.g., the final linear layer does not need to be constructed from scratch before fine-tuning). One possible addition to the paper could be discussion of this setting, and in particular if LP-FT is still required.\n\n4) There are a few claims in the paper which could benefit from additional support. In particular, in the conclusion it is stated that the \"gap between FT and LP grows as the quality of pretrained features improve\". Are the authors referencing the single MOCO-v1 vs. MOCO-v2 experiment or is there additional support for this claim? Are the authors referring to absolute or relative difference between FT and LP? I am wondering as this claim seems a bit counterintuitive, if the features are already good shouldn't LP be sufficient?\n\n5) The empirical verification of the theory (Section 4.3) is very interesting and could be more thorough. In particular, are there associated error bars for the 0.019 and 0.017 numbers as this seems quite close to conclude that one is larger. This euclidean distance experiment is very interesting, why is it only conducted for one distribution shift? Moreover, what happens when analyzing cosine distance instead of cosine distance?\n\n[1] https://arxiv.org/pdf/2106.04560.pdf\n[2] https://arxiv.org/abs/2007.00644\n[3] https://arxiv.org/abs/2107.04649", "summary_of_the_paper": "This paper contrasts fine-tuning (i.e., modifying all network weights) and linear probing based on their relative ID/OOD performance. It is known that fine-tuning (FT) outperforms linear probing (LP) ID. This paper presents that the reverse is true OOD (FT outperforms LP). This paper suggests that this occurs because fine-tuning distorts features in conjunction with the final linear layer. Instead, if a final linear layer is trained first, the features do not have to move that much during full fine-tuning. The authors refer to this method as LP-FT and show it often outperforms LP and FT both OOD and ID.", "summary_of_the_review": "This paper is very interesting and will be an important contribution if concerns are properly addressed. An essential concern is the contextualization of the method LP-FT --- is this method (or a modification) something that people have explored previously and if so in what context. In the current framing of the abstract and introduction, LP-FT seems to be introduced by this paper. In addition, this papers empirical verification of the theory (Sec 4.3) appears very promising but could benefit from additional detail and experiments (e.g., more than one distribution shift in the euclidean distance experiments).\n\nEdit: authors have addressed many concerns and I have changed my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a solid and thorough analysis to the two basic methods of fine-tuning, linear probing (LP) and fine-tuning (FT). The authors provide an important and highly interesting observation about the performance of both in and out of domain (OOD) setting. They validate the known phenomena that FT outperforms LP in the in-domain (ID) setting, but demonstrate that when tested on OOD data, LP is in fact more performant and back this observation with a theoretical and empirical analysis.
The remedy provided is also a known, yet slightly less popular technique of setting the final layer (LP) first, then finte-tuning (FT-LP). The authors provide thorough experiments showing that this technique enjoys the best of both worlds, meaning ID and OOD. I found it worth noting that during the rebuttal period the authors provided experiments on additional larger scale datasets and models and the results of the paper carried over to these new setting.
The reviews agree that the analysis provided is both interesting and novel. Even though the paper does not provide a new technique, there is a consensus that the understanding it gives on known techniques is a welcome addition to ICLR.
Decision: Accept (Oral)