Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Authors: Yoonho Lee
Annie S Chen
Fahim Tajwar
Ananya Kumar
Huaxiu Yao
Percy Liang
Chelsea Finn
Venue: ICLR 2023 poster (2023)
Keywords: Transfer learning
fine-tuning
parameter freezing
distortion of pre-trained models

Abstract: A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task. This paper shows that in such settings, selectively fine-tuning a subset of layers (which we term surgical fine-tuning) matches or outperforms commonly used fine-tuning approaches. Moreover, the type of distribution shift influences which subset is more effective to tune: for example, for image corruptions, fine-tuning only the first few layers works best. We validate our findings systematically across seven real-world data tasks spanning three types of distribution shifts. Theoretically, we prove that for two-layer neural networks in an idealized setting, first-layer tuning can outperform fine-tuning all layers. Intuitively, fine-tuning more parameters on a small target dataset can cause information learned during pre-training to be forgotten, and the relevant information depends on the type of shift.

Published as a conference paper at ICLR 2023
SURGICAL FINI-TUNING IMPROVES ADAPTATION TO DISTRIBUTION SHIFTS
Yoonho Lee" Stanford University
Annie S. Chen Stanford University
Fahim Tajwar Stanford University
Ananya Kumar Stanford University
Huaxiu Yao Stanford University
Percy Liang
Chelsea Finn Stanford University
Stanford University
ABSTRACT
A common approach to transfer learning under distribution shift to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapt- ing to the new task. This paper shows that in such settings, selectively fine-tuning a subset of layers (which we term surgical fine-tuning) matches or outperforms commonly used fine-tuning approaches. Moreover, the type of distribution shift in- fluences which subset is more effective to tune: for example, for image corruptions, fine-tuning only the first few layers works best. We validate our findings systemati- cally across seven real-world data tasks spanning three types of distribution shifts. Theoretically. we prove that for two-layer neural networks in an idcalized setting, first-layer tuning can outperform fine-tuning all layers. Intuitively, fine-tuning more parameters on a small target dataset can cause information learned during pre-training to be forgotten, and the relevant information depends on the type of shift.
INTRODUCTION
While deep neural networks have achieved impressive results in many domains, they are often brittle to even small distribution shifts between the source and target domains (Recht al., 2019; Hendrycks & Dietterich, 2019; Koh et al., 2021). While many approaches to robustness attempt to directly generalize to the target distribution after training on source data (Peters et al., 2016; Arjovsky et al., 2019). an alternative approach is to fine-tune on a small amount of labeled target datapoints. Collecting such small labeled datasets can improve downstream performance in a cost-effective manner while substantially outperforming domain generalization and unsupervised adaptation methods (Rosenfeld et al., 2022; Kirichenko et al., 2022). We therefore focus on settings where we first train a model on a relatively large source dataset and then fine-tune the pre-trained model on small target dataset, as means of adapting to distribution shifts.
The motivation behind existing fine-tuning methods is to fit the new data while also preserving the information obtained during the pre-training phase. Such information preservation critical for successful transfer learning, especially in scenarios where the source and target distributions share a lot of information despite the distribution shift. To reduce overfitting during fine-tuning, existing works have proposed using smaller lcarning rate compared to initial pretraining (Kornblith et al., 2019; Li et al., 2020), freezing the early backbone layers and gradually unfreezing (Howard & Ruder, 2018; Mukherjee & Awadallah, 2019; Romero et al., 2020), or using different learning rate for each layer (Ro& & Choi, 2021; Shen et al., 2021).
We present result in which preserving information in non-standard way results in better perfor- mance. Contrary to conventional wisdom that one should fine-tune the last few layers to re-use the learned features, we observe that fine-tuning only the early layers of the network results in better performance on image corruption datasets such as CIFAR-10-4 (Hendrycks 2019). More specifically, as an initial finding, when transferring model pretrained on CIFAR-10 to CIFAR-1O-C by fine-tuning on a small amount of labeled corrupted images, fine-tuning only the first block of layers and freezing the others outperforms full fine-tuning all parameters by almost 3% on average on unseen corrupted images.
Dietterich,
"Equal contribution. Correspondence to yoonho @stanford.cdd and asc& @ stanford.edu.

---

Published as a conference paper at ICLR 2023
Input-Level Shift: CIFAR-C
Feature-Level Shift: Entity-30
Output-Level Shift: CelebA
Image corruption
Subgroup shift
Spurious correlation
Source data
Target data
Full Fine-Tuning
79.9%
79.3%
82.2% 86.2% (+4.0)
Surgical Fine-Tuning
82.8% (+2.9)
81.2%(+2.)
HHH
First block
Middle/later block
Last layer
Figure 1: Surgical fine-tuning, where tune only one block of parameters and freeze the remaining parameters. outperforms full fine-tuning range of distribution shifts. Morcover, find that tuning different blocks performs best for diffcrent types of distribution shifts. Fine-tuning the first block works best for input-level shifts such as CIFAR-C (image corruption), later blocks work best for feature-level shifts such as Entity-30 (shift in entity subgroup), and tuning the last layer works best for output-level shifts such as CelebA (spurious correlation between gender and hair color).
To better understand this counterintuitive result, we study a general class of fine-tuning algorithms which we call surgical fine fite-tunin, defined fine-tuning only small contiguous subset of all layers in the pre-trained neural network. Equivalently, we could define surgical fine-tuning as freezing all but a few layers during fine-tuning. Parameter freezing can be beneficial because, depending on the relationship between the source and target tasks, some layer parameters trained on the source task may be close minima for the target distribution. Therefore, freezing these layers can facilitate generalization to the target distribution. We evaluate the performance of surgical fine-tuning with various layer choices 7 different distribution shift scenarios, which we categorize into input-level, feature-level, and output-level shifts. As shown in Figure 1, fine-tuning only the first block of layers, the middle block, the last layer can perform best in different distribution shift conditions, with the best such subset consistently outperforming fine-tuning all parameters.
outperfooree
To support empirical results, theoretically analyze why different types of distribution shifts require fine-tuning different layers. For two-layer neural networks, we show why fine-tuning the first layer is better for input perturbations but fine-tur tuning the last layer is better for label perturbations. We then present a setting where surgical fine-tuning on the first layer provably outperforms fine-tuning all parameters. If the target distribution contains only few new "directions" (inputs outside the span of the source distribution), we show that tuning only the first layer can learn these new directions with very few target examples, while preserving all the information learned from the source distribution. However, we show that full fine-tuning forgets information learned from the source distribution the last layer changes to accommodate the target directions, but now performs poorly examples outside the span of the training data. Motivated by the theoretical insight that freezing some layers can help generalization, we empirically analyze two criteria for automatically selecting layers to tune based on loss gradients. Tuning the layers selected by such criteria can also outperform full fine-tuning, though this procedure does not outperform manually choosing the best layers tune.
Our main contribution is the empirical observation that fine-tuning only a small contiguous subset of layers can outperform full fine-tuning on range of distribution shifts. Intriguingly, the best layers to tune differ for different distribution shift types (Figure 1). This finding is validated empirically across seven real-world datasets and three types of distribution shifts, and theoretically in idealized two-layer neural network setup. We additionally empirically analyze two criteria for automatically selecting which layers tune and find that fine-tuning only the layers with higher relative gradient norm outperforms full fine-tuning -unitu.
SURGICAL FINE- TUNING: FREEZING PARAMETERS DURING ADAPTATION
Our problem setting assumes two datasets from different distributions: a large dataset following the source distribution Pare. and relatively smaller dataset following the target distribution Pigt- The objective is to achieve high accuracy on target data by leveraging the different but closely related source distribution, common scenario in real-world applications that require adaptation. For example, the source dataset can be the 50, 000 training images in CIFAR- 10 (Krizhevsky et al., 2009) while the target dataset is smaller set of 1000 corrupted CIFAR datapoints with the same image corruption (Hendrycks & Dietterich, 2019); see Figure for more examples of source-target

---

Published as a conference paper at ICLR 2023
dataset pairs that consider. To achieve high performance the target distribution, model should broadly fit the large source dataset and make minor adjustments based on the smaller target dataset.
We empirically evaluate transfer learning performance with a two-stage training procedure consisting of pre-training and fine-tuning. First, we pre-train a network to minimize the loss on the source dataset to obtain fere. which has high accuracy in the source distribution. The fine-tuning stage starts from pre-trained model parameters and minimizes the loss on the labeled target data, resulting in the model figt We evaluate two fine-tuning settings in this section: supervised fine-tuning (Section 2.1) and unsupervised adaptation (Section 2.2). In all experiments, we perform early stopping on held-out target data according to the fine-tuning loss. Finally, we evaluate the performance of the fine-tuned model on held-out data from the target distribution, i.e. Ligt (f (xr..))PP.
( (x), v)]-
Our main focus is analyzing surgical fine-tuning, in which we fine-tune only a subset of layers of the pre-trained model while keeping the others frozen. Denote the pre-trained model as f = fn o...o fi (a)). where each layer fi has parameters d,, and the empirical target loss as Ligt. Formally, surgical fine-tune fine-tuning with respect to subset S _ {I....,n, n) of layers is defined as solving the optimization problem
arg min Ligo (f(11..... 0n)). ViES
(1)
where all non-surgery parameters (0, for & S) are fixed to their pre-trained values. Typical choices of parameters to optimize are fine-tuning all (S = last few layers (S = (n k, ... n))) The main novelty of the surgical fine-tuning framework is that it additionally considers tuning earlier layers while keeping later layers frozen. For example, surgical tnneiuning on the first layer (S (1)) updates only 01, resulting in the fine-tuned model f(e = fare .... o f (x).
n.), last (S = {n.)), or the
Intuitively, surgical fine-tuning can outperform full fine-tuning when some layers in f are already near-optimal for the target distribution. As a hypothetical example, consider a scenario where there exist first-layer parameters 0f such that changing only the first layer of fxc to 0f achieves zero target loss. Here, first-layer fne-tuning (S = {1}) can find 0j with a small amount of target data, while full fine-tuning (S = f1.... n)) may needlessly update the other layers and thus underperform on held-out target data due to overfitting. We note that the efficacy of parameter freezing a consequence of having limited target data, and choosing bigger S will be beneficial in settings where target data plentiful. Now that we have introduced the problem set-up, we will next empirically investigate how surgical fine-tuning with different choices of S performs on real datasets.
2.1 SURGICAL FINE-TUNING: EXPERIMENTS ON REAL DATA
In this subsection, we aim empirically answer the following question: how does surgical parameter fine-tuning compare to full fine-tuning in terms of sample efficiency and performance on real-world datasets?
Datasets. We run experiments on nine real-world distribution shifts, categorized into input-level, feature-level, output-level, and natural shifts, with examples shown in Figure 1. For more details about these datasets, see Appendix B.3.
Input-level shift: (1) CIFAR-C (Hendrycks & Dietterich, 2019), (2) ImageNet--C (Kar et al., 2022). The source distributions correspond to the original CIFAR- 10 and ImageNet datasets (Krizhevsky et al., 2009; Deng et al., 2009), respectively. The task is to classify images from the target datasets, which consist of corrupted images.
Feature-level shift: (3) Living-17 and (4) Entity-30 (Santurkar al., 2020): While the source and target distributions consist of the same classes, they contain different subpopulations of those classes. For example, in Entity-30, for the class "vegetables", Pare and Pigt will contain different subclasses of vegetables.
Output-level shift: (5) CIFAR-Flip, (6) Waterbirds, and (7) CelebA (Sagawa et al., 2019). CIFAR-Flij is a synthetic task where the Pare consists of the original CIFAR-10 dataset and the target distribution is the same dataset where each label has been flipped to be 9 - y. e.g. the label 0 is now label 9 and vice versa. For Waterbirds and CelebA, the task labels are spuriously correlated with an attribute. The source distribution Pare is the training set while the target distribution Pigs is a balanced subset with equal amounts of each of the four (label, spurious attribute) groups

---

Published as a conference paper at ICLR 2023
Surgical Fine-Tuning on Different Distribution Shifts
Relative Accuracy
5.0
2.5
0.0
Block
Block
Block
5.0
Block 4
-75
Last Layer
CHAAC
ImageNet
Living-17
Entity-30
FFFR
Waterbirds
CelobA
Input-level shifts
Feature-level shifts
Output-level shifts
We present plots of relative accuracy, ie. (surgical fine-tuning accuracy) (full fine-tuning accuracy), along with standard errors across three runs. Fine-tuning single parameter block can outperform full fine-tuning and more importantly, different blocks are best for different distribution shifts. The location of the best block reflects the nature of the shift: tuning earlier layers performs best for input-level shifts while tuning later layers best for output-level shifts.
Figure 2:
Natural shift: (8) Camelyon17 (Bandi et al., 2018) and (9) FMow (Christie et al., 2018), part of the WILDS (Koh et al., 2021) benchmark. Camelyon17 contains medical images from different hospitals where variations in data collection and processing produces naturally occurring distribution shifts across data from different hospitals. FMoW contains satellite imagery of types of buildings or land use, with both temporal and sub-population shifts resulting from geographical diversity.
Model architectures and pre-training. For each task, before pre-training on Pare. we initialize with a model with ResNet-50 architecture pre-trained on ImageNet, except for experiments with CIFAR-C and CIFAR-Flip, which both use ResNet-26 trained from scratch, and experiments on Camelyon17 1 and FMoW, which use a pre-trained CLIP ViT-B/16. After initialization, we pre-train on the source small amount of data from the target domain. We fine-tune with
domain Pare and then fine-tune the Adam optimizer, sweeping over 3 learning rates. We choose the best hy hyperpameters and early stop based accuracy on held-out target data. We report results across seeds for all experiments. See Appendix B.4 for more fine-tuning details.
Surgical fine-tuning, The models used consist of three (for ResNet-266) or four (for ResNet-50) convolutional blocks followed by a final fully connected layer. We denote these blocks as "Block 1", "Block 2", etc in the order that they process the input, and the fully connected layer "Last Layer". CLIP ViT-B/16 has embedding layer, 11 attention blocks and a linear classifier as the last layer. For each experimental setting, report the relative target distribution accuracy and standard error across three runs after surgical fine-tuning on each block of the network, fine-tuning only that block while freezing all other parameters. We compare against full fine-tuning, i.e. tuning all parameters to minimize target loss.
Experimental results. Results in Figure 2 show that every domain, surgically fine-tuning one block of the network outperforms tuning all parameters on the target distribution. We note that matching full fine-tuning performance with surgical fine-tuning would indicate that ignoring some gradients is harmless; these results show that ignoring some gradicnts has a positive effect. Furthermore, we find that the best block to fine-tune is dif- ferent across settings, depending on the nature of the distri- bution shift between source and target data. Datasets with input-level shift are best handled by tuning the first network block, and similarly for feature- level shifts with middle block and output-level shifts with the last layer. We see the similar phenomenon in the natural shifts: Camelyon17 is closer to an input-level shift due to the lighting difference in different hospitals, while the shift between different regions in FMoW can be seen as close to a feature-level shift be-
Parameters
Camelyon17
FMoW
No fine-tuning
86.2
35.5
All
92.3 (1.7)
38,9 (0.5)
Embedding First three
95.6 (0.4)
36.0 (0.1)
92.5 (0.5)
39.8 (1.0)
Last three
84.1
44.9 (2.6)
Last layer
90.1 (1.5)
36.9 (5.5)
Table 1: OOD set accuracies after surgically fine-tuning different parameters in CLIP ViT-B/16 model for two WILDS datasets. Bold numbers represent superior results for a dataset, and we also report the standard deviation from runs with different seeds.
cause building shape and spacing most salient in satellite
imagery. Quantitative results in in Table 1 are in agreement with this intuition, where tuning the earliest embedding layer
works best for Camelyon 17 and tuning later attention blocks
works best for FMoW. Following Kumar al. (2022b). we also evaluate tnn-ttunng performance with the Adam W optimizer; results in Table 8 show a similar tendency but with smaller performance gaps. In Figure 4, we find that on CIFAR-C, fine-tuning the first block matches and even outperforms full fine-tuning as well as tuning with other individual blocks when given varying amounts of data for tuning, although the gap between Block 1 and All decreases as the number of training points increases.

---

Published as a conference paper at ICLR 2023
CIFAR-C Supervised
CIFAR-C Unsupervised
Parumeters
Episodic
Online
90
No aduptation
67.8
66-
Target Accuracy (%)
80-
et Accuracy (96)
All
71.7
69.7
60
02-
Layer
Layer
Layer 1-2
69.0
69.0
75.5
75,4
60
Layer
Block 1-2
69.0
75.3
Last Layer
Last
67.9
67.8
Block
56-
All
65
60-
Block
Block
Last Layer
54
Table 2: Unsupervised adaptation
accuracy on CIFAR-IO-C, aver- aged across 10 corruptions.
55
All
50
Target Set Size
Target Set Size
Parumeters
Episodic
Online
Figure 4; Online unsupervised adaptation with surgical fine-tuning for the Gaussian corruption in the CIFAR- C dataset. Adding data in an online fashion to full or last-layer tuning results in worse performance, whereas more data helps for first- layer adaptation. Error bars indicate the standard error across three runs.
No adaptation
46.5
Figure 3: Surgical fine-tuning results on CIFAR-IO-C with varying amounts of target data. Fine-tuning the early layers is beneficial even in the few- shot setting, given as few as 1 image per class for tuning. Error bars indi- cate the average standard error over 14 coruutiions.
All
47.8
1.8
Layer Block
46.6
46.4
46.7
49.0
Last
46.5
46.5
Table 3: Unsupervised adaptation accuracy ImageNet-C, aver- aged across 14 corruptions.
Intuitively, why might surgical fine-tuning match even outperform full fine-tuning on distribution shifts? For each type of shift we consider (input-level, feature-level and output-level), there is a sense in which one aspect of the distribution changes while everything else is kept the same, therefore requiring modification of only a small part of information learned during pre-training. For example, in image corruptions (categorized as input-level shift), pixel-wise local features are shifted while the underlying structure of the data is the same in the source and target distributions. On the other hand, in a label shift (categorized as output-level shift), the pixel-wise features remain the same in the source and target distributions while the mapping from final features to labels is shifted. This intuition is also in line with the independent causal mechanisms (ICM) principle (Schölkopf et al., 2012; Peters et al., 2017), which states that the causal generative process of a system's 's variables is composed of autonomous modules that do not inform or influence one another. From this viewpoint, distribution shifts should correspond to local changes in the causal generative process. Because discriminative models learn to invert the generative process from label to datapoint, it suffices to fine-tune only the region of the network that corresponds to the change in the causal process. We formalize this intuition more concretely in our theoretical analysis in Section 3.
2.2 UNSUPERVISED ADAPTATION WITH PARAMETER FREEZING
In this subsection, we aim to validate whether the findings from Section 2.1 hold in the unsupervised test-time adaptation setting, where adapt model trained on source data to the target distribution using only unlabeled target data. We experiment with variants of a representative state-of-the-art unsupervised adaptation method (Zhang al., 2021a, MEMO), which minimizes marginal entropy of averaged predictions for a single image. We consider two settings: online, where the model retains updates from past test images, and episodic, where we reset the model back the pre-trained weights after every test image.
Results in Table 2 and Table 3 show that the highest accuracy achieved by adapting the first two layers and first block in the online setting for CIFAR- and ImageNet-C respectively, and doing so outperforms fine-tuning all parameters. With full fine-tuning, online MEMO performance deteriorates as the test set size increases due to distortion of pre-trained features, as shown graphically in Figure4 4. In contrast, surgical fine-tuning mitigates this effect. These results are consistent with the supervised learning experiments in Section 2.1, where adapting the early parameters was best for image corruption datasets. We show detailed results in Appendix B.6.
ANALYSIS OF SURGICAL TUNING
We now present theoretical and empirical analysis idealized examples of distribution shifts, to better understand the role of surgical parameter tuning in our previous experimental results. In Section 3.1, we present setting with two-layer neural networks where tuning only the first layer can obtain zero loss on the target task while tuning only the last layer cannot and vice versa. Then, in Section 3.2, study setting in which tuning only the first layer provably achieves zero loss while full fine-tuning overfits and gets non-zero loss due to limited data. Finally, in Section 3.3, to support this theoretical analysis, we construct distribution shifts where localized subsets of parameters substantially better suited for adaptation than tuning all or other parameters.

---

Published as a conference paper at ICLR 2023
We focus on regression, where our goal is to map inputs R to outputs y R. and (z. 5) = (y - -))² is the squared loss. We consider two-layer networks (z) = v Φ( where V E R2 B E Be and @ is an elementwise activation function such as ReLU. Let Tare y/rcc ~ Pare and Ztrg: Mtrg Pirg be the inputs and outputs in the source and target distributions. We assume Yerc = fewe Ba. (Ierc) for some Usrc: Bsc. Note that Terc: Vsrci Ztrg: Vtrg are all random variables, and expectations are taken over all random variables if not specified. We define the population losses for source and target as ( (1, B) = E [[[[F,, (Tare), Vsre)] and Lurg (v, B) = B B|[(fe, B (Ferg) Murg)].
Theoretical setup.
3.1 LAYER CHOICE AND EXPRESSIVITY: WHYY FINE-TUNING THE RIGHT LAYER MATTERS
First note that for two-layer neural networks, we have two choices for surgical fine-tuning: the first layer and the last layer. We show by construction that if the distribution shift is closer to the input then first-layer tuning is better, but if the shift is closer to the output then last-layer tuning is better. In this section, we assume that ø is the elementwise ReLU function: 6(x); = max( r., 0). Recall that we first train on lots of source data suppose this gives us pretrained parameters Ue Bare which achieve minimum source loss: Togre ( (6rc. Bayc)= = 0.
Input perturbation. Suppose that the target input is a "perturbed" or "corrupted" version of the source input: Wing Are for some invertible matrix A E RR*x, where the corresponding label is unchanged: Yrg = Usrc- We note that this simplified perturbation class includes some common image corruptions as brightness shift and Gaussian blur as special cases, while others such as pixelation are similarly linear projections but ion-invertible. Proposition 1 shows that for this distribution shift, tuning only the first layer can minimize the target loss but only changing the last layer may not.
Proposition 1. For all A, Parc: Pung with Ttrg ATrrr for invertible A and yvg Ysue there exists first-layer B that can minimize the target loss: ming Lirg (rrr( B) = O. However, changing the last layer may not be sufficient: there exists such A, Pere, Pung such that the target loss is non-zero for any choice of last layer V: for all i, min, Lug( Bacc)= > O.
Intuitively, the first-layer can learn to "undo" the perturbation by selecting B A-1 However, if we freeze the first-layer then the representations Φ ((BecTrrg) may miss important input directions in the target, so no last-layer u can produce the correct output. For a full statement and proof, see Appendix B.1.
Label perturbation. the target output is perturbed from the source output: Birg = tyyee for some t. Proposition shows that tuning only the first layer may not achieve non-zero target loss for this distribution shift while tuning the last layer will do so.
Now suppose that the source and target inputs are the same Ftrg = Zare. but
Proposition 2. For all 1, Pares Purg with Tirg = Tarc. and ytrg = Lyse there exists a last-layer o that can minimize the target loss: min, Lug(U, Bstc) = O. However, changing the first layer may not be cefficient-thery exists such t, Parc: Pirg such that the target loss is non-zero for any choice of first layer B: ming Lirg( (Bre: B)- =
Similarly to Proposition 1, the last layer can adapt to the label shift by "reversing" the multiplication by f. In contrast, when the last layer is frozen and only the first layer is tuned, we may lack expressivity due to the information destroyed by the ReLU activation <(-). For full statement and proof, see Appendix B.I.
3.2
CAN SURGICAL FINE- TUNING OUTPERFORM FULL FINE-TUNINGS
In this section, we show that first-layer fine-tuning can provably outperform full fine-tuning when we have an insufficient amount of target data. We show that this can happen even tuning two-layer linear networks (Kumar et al., 2022a), where Φ is the identity map: for all i, o(x); = 24. Our analysis suggests perhaps a more general principle underlying the benefits of surgical fine-tuning over full fine-tuning: by fine-tuning more parameters than necessary. the model can overfit to the small target dataset while forgetting relevant information learned during pre-training.
We first start with Vo. Bo which are initialized randomly: Vo: (0, and Boy NO, o for all j.j. We then run gradient descent on Lsre B) to obtain Dsre, Bacs which we assume minimizes the source loss: V Base) O.
Pretraining.

---

Published as a conference paper at ICLR 2023
Added Noise
First block
Second block
Third block
Last layer
Tuned Layers
First block
92.5 (0.3)
27.7 (1.2)
27.2 (1.7)
37.4 (9.0)
Second block
68.8 (2.3)
92.6 (0.3)
46.6 (2.6)
61.0 (4.4)
Third block
48.0 (1.3)
59.7 (0.6)
92.1 (0.2)
92.0 (0.5)
Last layer
33.7 (0.4)
34.8 (0.4)
62.0
91.8 (0.6)
All
(1.8 91.9 (0.2)
92.0 (D.1)
91.3 (0.3)
Figure 5: In this setting. we construct distribution shifts such that particular block of parameters is substantially suited for adaptation. We find that tuning only the subset of parameters that responsible for the shift performs better than uning any other block of parameters all parameters. Darker blue indicates higher accuracy while darker red indicates lower accuracy.
Fine-tuning data. Suppose we have target datapoints sampled from Purg: (Teg yerg (6) (n)) (wriig) Mtrg ). The empirical fine-tuning loss is given by: Logg( B) Ei= ((,,n( (Fir) Verg
(1) ())
(9) (5)
Fine-tuning algorithms. We study two different gradient flows each corresponding to fine-tuning methods: first-layer tuning (fl) and full fine-tuning (ft).
B. p (on (t))
(t
(t) --V (t), Bn (t))
(ten(t) = -V, v Lurg (vn(t), (t)),
with initial conditions (0 = (0) = Use and Ba(O) = Ba) = Bure. We denote the limit points of these gradient flows BA lim Un (t), etc.
The following theorem shows that there exists a shift, where if we have a small target dataset, full fine-tuning does worse than first-layer tuning.
Theorem 1. For any O, there exists d, k, Prres Pargi n such that with probability at least first-layer tuning gets 0 loss convergence, but fine fine-tuning gets higher (non-zero) loss throughout the fine-tum teeettningg trajectory:
Lirg (Un (1), (i) Lirg(va² B 0 Vt.
Intuitively, if Pood contains few additional directions in the input that are not present in Pia, then first ttyeing can quickly learn those new directions. Full fine-tuning changes both the head and feature extractor B to fit these new be incompatible with B in some directions not seen in the finite training set, thus "forgetting" some knowledge present in the source data. The full proof is in Appendix B.2.
however, because the head has changed may
3.3 SURGICAL FINE- TUNING ON SYNTHETIC DISTRIBUTION SHIFTS
To better illustrate how specific subsets of parameters are better suited depending on the distribution shift, model distribution shifts by adding noise to individual blocks of layers. More specifically, we initialize with a ResNet-26 model pretrained on the CIFAR-10 10 (Krizhevsky et al., 2009) training dataset. We then add noise to each of the three blocks the last layer, simulating distribution shifts localized to those parameters, and then tune each of the different blocks of the network while freezing all other parameters the CIFAR-10 10 test dataset.
In Figure 5, we find that only tuning the subset parameters that responsible for the shift performs better than tuning any other subset of parameters and even outperforms tuning all layers, indicating that when tuning the parameters responsible for the shift, tuning other parameters may actually hurt performance.
4 AUTOMATICALLY SELECTING WHICH LAYERS TO TUNE
In this section, we investigate three criteria for automatically finding an adequate subset of layers to perform surgical fine-tuning on. We evaluate their fine-tuning performance versus full fine-tuning and another prior method on the 7 real-data domains introduced Section 2.1. We also analyze performance on the synthetic distribution shifts introduced in Section 3.3.
4.1 CRITERIA FOR SELECTING LAYERS
We consider three metries for automatically choosing which layers freeze.
Cross-Val After running surgical fine-tuning for all blocks, we select the best block based on a held-out validation set from the target distribution. While quite effective, this method requires as many fine-tuning runs as there are blocks inside the network.

---

Published as a conference paper at ICLR 2023
Inpur-level Shifts
Feature-level Shifts
Outpur-level Shifts
Avg Rank
Method
CIFAR-C
IN-C
Living-17
Entity-30
CIFAR-F
Waterbirds
CelehA
No Adaptation
52.6 (0)
18.2 (0)
80.7 (1.8)
(1.1)
(0)
31.7 (0.3)
27.8 (1.9)
Cross-Val
82.8 (0.6)
51.6
93.2 (0.3)
81.2 (0.6)
93.8 (0.1)
89.9 (1.2)
86.2 (0.8)
Fine- Tuning (All)
(0.7)
50.7 (0.1)
92.8 (0.7)
79.3 (0.6)
85.9 (0.4)
88.0 (1.2)
82.2 (1.3)
2.71
Gradual Unfreeze (First Last)
(0.7)
50.7 50.7 (0.1)
90.1 (0.1)
69.9 (3.0)
81.9 (0.6)
87.5 (0.2)
71,9 (0.7)
4.71
Gradual Unfreeze (Last First)
(0.8)
49.8 (0.2)
90.6 (0.2)
(0.5)
84.3
87.8 (0.1)
78.5 (2.9)
4.0
L, Regularize (Xuhong al., 2018)
(0.6)
48.8 (0.3)
93.4 (0.5)
78.4 (0.1)
(1.2
87.6 (1.9)
82.6
3.28
SNR
(0.7)
49.9 (0.2)
93.5 (0.2)
77.3 (0.3)
(0.7)
86.3 (0.7)
(1.8)
4.14
Auto-RON
(0.6)
51.2 (0.2)
93.5 (0.3)
80.6
87.7 (2.8)
88.0 (0.7)
82.2 (2.7)
1.29
Table 4: We report the average accuracy and standard error achieved on the target distribution real-data tasks. Cross-Vo Val, which requires as surgical fine-tuning rum for each block, performs the best, but we find that Auto-RGN performs the hest out of all methods that require only fine-tuning run, outperforming Full Fine-tuning, Gradual Unfreezing, Li Regularize, and Auto-SNR. The best overall method for each shift is underlined, and the best among methods that use fine-tuning run is bolded.
Relative Gradient Norm (Auto-RGN), Within each layer, we measure the ratio of gradient norm to parameter norm, and select layers that have relatively larger gradients. Intuitively, our hypothesis is that layers with large gradient magnitudes may carry more information about the target task than others and can therefore be more useful. Formally, denote gradients at layer as Jr. We define the relative gradient norm of this layer as RGN(O.) = || Then to alter fine-tuning with this
(9.)
criterion, at each epoch, we normalize the RGNs for each layer between 0 and 1 and then multiply the learning rate for each layer by its RGN. Using this criterion for fine-tuning requires no additional hyperparameters over tuning all layers and only one fine-tuning run.
Nigoiee Ratio (Auto-SNR). For each layer i, this criterion is defined as SNR(g;) Avg(g:1
Var(g:)
with average and variance computed across (target) datapoints. Intuitively, SNR measures how noisy the gradient of each layer is and thus how much it may contribute to distorting the function learned during pre-training. This gradient-based criterion has been shown to be useful for early stopping (Mahsereci et al., 2017). During fine-tuning, we normalize the SNR for each layer between and and then freeze all layers that have SNR under a threshold that is tuned as an additional hyperparameter.
As points of comparison, we compare the three criteria for layer selection above to existing methods for regularizing fine-tuning. We consider two variations of gradual unfreezing: Gradual Unfreeze (First -> Last) and Gradual Unfreeze (Last- First) (Howard & Ruder, 2018; Romero et al., 2020; Kumar et al., 2022a), in addition to L1 Regularize (Xuhong et al., 2018). These methods are similar in spirit to surgical fine-tuning in that they aim to minimize changes to parameters. We additionally experimented with regularizing the La norm, but found that L₁ consistently performs better.
4.2 RESULTS ON REAL WORLD DATASETS
In Table 4, we compare Cross-Var with 6 methods (Full Fine-Tuning, Gradual Unfreeze (First + Last), Gradual Unfreeze (Last + First), L1 Regularize, Auto-SNR, and Auto-RGN) that require only 1 tine-tuning run. We find that auto-tuning with relative grad norm (Auto-RGN) matches or outperforms fine-tuning all parameters on all domains and is the most competitive method that requires only 1 fine-tuning run although it does not quite match the performance of Cross-Val. We find that Cross- Val corresponds in performance to the best surgical fine-tuning result for each dataset, which expected, as the validation and test sets are both held-out subsets of the same target distribution. Auto-SNR struggles to extract the most effective layers for tuning and hence does worse on most shifts than All and Auto-RGN. While Gradual Unfreeze fails to consistently outperform full fine-tuning, the directionality of results is consistent with surgical fine-tuning; unfreezing first layers best in input-level shifts and unfreezing last layers is best in output-level shifts. L2 Regularize performs slightly better than tne-tuning all, but performs worse than Auto-RGN on all datasets except CIFAR-C. All methods outperform no adaptation.
4.3 AUTOMATIC SELECTIVE PINE-TUNING IN SYNTHETIC DISTRIBUTION SHIFTS
As Auto-RGi is the best performing method that requires only one fine-tuning run, and in particular outperforms fine-tuning all layers, we further analyze what layers Auto-RGN chooses to fine-tune and see to what extent they correlate with our experiments in Section 2. To do so, we evaluate Auto-RGN on the synthetic distribution shifts introduced in Section 3.3, where we model distribution shifts by adding noise to blocks of parameters, and plot the weights that Auto-RGN gives to the layers.

---

Published as a conference paper at ICLR 2023
Block
Block
Blockk
Tuned Layers
Full Fine Tuning
Auto RGN
Auto-RGN Weights
23
Block 1
87.8 (1.2)
92.0 (0.4)
Block 2
91.9 (0.2)
92.7 (0.4)
Block
92.0 (0.1)
92.5 (0.4)
15
Last Layer
91.3 (0.3)
92.8 (0.3)
10-
Block 1
Table 5: Automatically choosing which layers to tune using the relative gradient norms (RGN) of each layer outperforms fine-tume tuning on distribution shifts con- structed by adding noise different blocks of layers.
Block 2
Block 3
12
16
20 Layer
24
26
12
Figure 6: Auto-RGN consistently gives higher weights for the layers in the block responsible for the distribution shift than for the other layers.
We find that Auto-RGN is able to ascertain which parameters may be responsible for the shift and weight the learning of those parameters to higher than the others, resulting in an informative signal that matches the performance of tuning only the noisy subset of parameters and outperforms full fine-tuning, as seen in Table 5. Figure shows the accumulated weights given by Auto-RGN over the course of training for each layer, colored by block. The weights for the layers responsible for the distribution shifts are higher than the weights for the other layers.
RELATED WORK
Parameter frreziing Freezing parameters to preserve previously learned information has been shown to be effective strategy in diverse set of domains: domain adaptation (Sener al., 2016; Long et al., 2016), early stopping (Mahsereci et al., 2017), generative models (Mo et al., 2020), and gradient-based meta-learning (Zintgraf et al., 2019; Raghu al., 2019; Triantafillou al., 2021), A highly effective approach to fast adaptation of large language models is prompt tuning (Li & Liang, al., 2021; Lester et al., 2021; Wei et al., 2021), which similarly be seen as an extreme special case of freezing where we only fine-tune the inputs to the neural network. Our surgical fine-tuning framework contains many such previous works special cases, and our experiments highlight the value of carefully choosing the subset of parameters to freeze.
2021; Hambardzumyan
Transfer learning. Prior works in transfer learning have studied how fine-tuning may be used to adapt pretrained features to a target distribution (Oquab et al., 2014; Yosinski et al., 2014; Sharif Razavian et al., 2014). To preserve information obtained during pre-training, many works propose methods of regularizing the fine-tuning process (Zhang et al., 2020; Xuhong et al., 2018; Lee et al., 201 Jiang al., 2019; Li et al., 2020; Aghajanyan et al., 2020; Gouk al., 2021; Shen et al., 2021; Karani et al., 2021). In particular, many works show that freezing some parameters in the pre-trained model can reduce overfitting during fine-tuning (Kirkpatrick 2019; Ramasesh et al., 2020; Liu et al., 2021b; Royer & Lampert, 2020; Eastwood et al., 2021; 1; Evci et al. 2022: Eastwood et al., 2022: Cohen et al., 2022: Touvron et al., 2022), and we build on such observations. Module criticality (Zhang et al., 2019; Chatterji et al., 2019; Neyshabur et al., 2020), which independently examines each layers' loss surface, is also closely related to our analysis. In contrast to existing works, we make the counterintuitive observation that freezing the later layers, or equivalently performing surgical fine-tuning the early layers, can perform best in some settings. Furthermore, we study the relationship between the best subset of layers to tune and the nature the distribution shift between the source and target distributions.
al., 2017; Lee al., 2019b; Guo et al.,
Distribution shifts. Many existing works have studied adaptation and robustness to various distribu- tion shifts (Tzeng al., 2014; Byrd & Lipton, 2019; Hendrycks al., 2019; Arjovsky al., 2019; Salman et al., 2020; Liu et al., 2021a; Wiles et al., 2021; Andreassen et al., 2021; Miller et al., 2021; Creager al., 2021; Lee et al., 2022: Kumar et al., 2022a). Such works typically frame robustness to distribution shift as zero-shot generalization problem, where the model is trained on source and evaluated on target. We consider different problem setting where the model is allowed to adapt to some labeled target data available. Some recent works have proposed methods for model adaptation at test time (Sun et al., 2020; Varsavsky al., 2020; Iwasawa & Matsuo, 2021; Wang et al., 2020; Zhang et al., 2021a;b; Gandelsman et al., 2022). Recent works (Rosenfeld et al., 2022; Kirichenko et al., 2022) study problem setting close to ours, showing that fine-tuning the last layer is sufficient for adapting to datasets with a spuriously correlated attribute. Our experiments in Section 2 confirm these results, and we further evaluate a broader set of distribution shifts including image corruptions and shifts at the level of intermediate features. We find that fine-tuning different subsets of layers performs best for different types of distribution shifts, and also present theoretical analysis on the relationship between surgical fine-tuning and the type of distribution shift.

---

Published as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2088.03/56 2020.
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning. arXiv preprint arXiv:2106.1583/ 2021.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Pazz Invariant risk minimization. arXiv preprint arXiv: 0000000009,.... 2019.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. 872-881. PMLR, 2019.
Niladri S Chatterji, Behnam Neyshabur, and Hanie Sedghi. The intriguing role of module criticality in the generalization of deep networks. arXiv preprint arXiv: /912v0/2 2019.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. " this my unicorn, fluffy": Personalizing frozen vision-language representations. arXiv preprint arXiv:2204.0169) 2022.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference Machine Learning, pp. 2189-2200, PMLR, 2021.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthiss Hein. Robusthench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in Neural Information Processing Systems, 31, 2018.
Cian Eastwood, Ian Mason, Christopher KI Williams, and Bernhard Schölkopf. Source-free adaptation to measurement shift via bottom-up feature restoration. arXiv preprint arXiv:2107.05446, 2021.
Cian Eastwood, Ian Mason, and Christopher KI Williams. Unit-level surprise in neural networks. In I (Still) Can't Believe It's Not Better! Workshop al NeurIPS 2021, pp. 33-40. PMLR, 2022.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. In International Conference on Machine Learning, pp. 6009-6033 PMLR, 2022.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A Efros. Test-time training with masked autoencoders. arXiv preprini arXiv:2209.05522, 2022.
Henry Gouk, Timothy Hospedales, and massimiliano pontil. Distance-based regularisation of deep networks for fine-tuning. In International Conference Learning Representations. 2021.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 0805-4814, 2019.

---

Published as a conference paper at ICLR 2023
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgrammingg In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pp. 4921-4933, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 acl-Iong.381
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/15 2015.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: AXXiv:8000006455 2018
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. arXiv preprint arXiv: 191 1.03437, 2019.
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18963-18974, 2022.
Neerav Karani, Ertune Erdil, Krishna Chaitanya, and Ender Konukoglu. Test-time adaptable neural networks for robust medical image segmentation. Medical Image Analysis, 68: 101101,, 2021. ISSN 1361-8415.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. arXiv preprint 2eXii:2204.02937, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-355 3526, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-2677 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022a.
Ananya Kumar, Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. How to fine-tune vision models with sgd, 2022b.
Cheolhyoung Lec, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. arXiv preprint arXiv: 2019a.
Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv: 1.03090, 2019b.

---

Published as a conference paper at ICLR 2023
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspeci- fied data. arXiv preprint arXiv:2202.03418, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint urXiv:2 (at.::2000.0699/1 2021.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint (eXiv:2111.00//0 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021a.
Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman, Autofreeze: Automatically freezing model blocks to accelerate fine-tuning. arXiv preprint arXi::2102.01386, 2021b.
Mingsheng Long. Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances Neural Information Processing Systems, 2016.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig. Early stopping without a validation set. arXiv preprint arXiv:1703.09550 2017.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between out-of-distribution and in-distribution generalization. International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans. arXiv preprint wrXiv:2002.10944 2020.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. Distilling transformers into simple neural networks with unlabeled transfer data. arXiv preprint arXiv:1910.01769, 1, 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivie. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724, 2014.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7845):947-1012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint 17X1v:1909009157, 2019.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400 2020.
12

---

Published as a conference paper at ICLR 2023
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400 PMLR, 2019.
Youngmin Ro and Jin Young Choi. Autolr: Layer-wise pruning and auto-tuning of learning rates in fine-tuning of deep networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2486-2494, 2021.
Miguel Romero, Yannet Interian, Timothy Solberg, and Gilmer Valdes. Targeted transfer learning to improve performance in small medical physics datasets. Medical Physics, 47(11))6246-06 6256, 2020.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint orXiv:2202.068555 2022.
Amélie Royer and Christoph Lampert. A flexible selection scheme for minimum-effort transfer learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2191-2200, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. 2019.
arXiv preprint
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversari- ally robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-3545, 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008. 04859, 2020.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In International Conference on Machine Learning. 2012.
Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable represen- tations for unsupervised domain adaptation. Advances in neural information processing systems, 29, 2016.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding bascline for recognition. In Proceedings the IEEE conference computer vision and pattern recognition workshops, pp. 806-813, 2014.
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9594-9602, 2021.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with superviuprv for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9248. PMLR, 2020.
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. arXiv preprint arXi::2203.09795 2022.
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning a universal template for feot dataset generalization. In International Conference on Machine Learning. pp. 10424-10433, PMLR, 2021.
Eric Tzeng, Judy Hoffmann Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412.344, 2014.
Thomas Varsavsky, Mauricio Orbes-Arteage Carole H Sudre, Mark Graham, Parashkev Nachev, and M Jorge Cardoso. Test-time unsupervised domain adaptation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 428-436. Springer, 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint 20X.::00066102266, 2020.
13

---

Published as a conference paper at ICLR 2023
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 21t.:2200900162, 2021.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise- Rebuffi, Ira Ktena, Taylan Cemgil, et al. A fine- fine-grained analysis on distribution shift. arXiv preprint 10. 1328, 2021.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang, In- N-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. In International Conference on Learning Representations (ICLR), 2021.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. International Conference on Machine Learning, pp. 2825-2834. PMLR, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? 2019.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks, In European Conference Computer Vision, pp. 698-714. Springer, 2020.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. arXiv preprint a 2021a.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems, 34:23664-23678, 2021b.
Luisa Zintgraf. Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning In International Conference on Machine Learning, pp. 7693-7702.
PMLR, 2019.

---

Published as a conference paper at ICLR 2023
A EXTENDED DISCUSSION
In this paper, we empirically find that when fine-tuning on new target distribution, it is often best to perform surgical fine-tuning, ie. to adapt only small contiguous subset of parameters. More importantly. which subset is more effective to tune depends on the type of distribution shift. For example, on input-level shifts like image corruption, only tuning earlier layers fine-tuning all or only later layers. These results support our intuition from the independent causal mechanisms (ICM) principle: many distribution shifts can be explained by shift in one module of the prediction mechanism and can thus be adapted to by tuning only a small subset the network. Our empirical findings supported by theoretical results, which show by construction that first- tuning may outperform full fine-tuning in an idealized two-layer neural network setting.
outperform
Additionally, manually choosing which layers freeze with the framework of surgical fine-tuning requires more fine-tuning runs than fine-tuning all layers, so we analyze two criteria for automatically selecting which layers to tune. While Auto-RGN consistently improves over full fine-tuning, its performance does not match the best surgical fine-tuning approach. Future work may close this gap by investigating more effective criteria for automatic selection. More generally, potentially fruitful direction for future work in better understanding when a distribution shift would prefer a certain layer, potentially shedding light on the nature of different distribution shifts.
B APPENDIX
B.1 PROOFS FOR SECTION 3.1
For all A, Paece Pug with Ftrg A.rare invertible A and Yug Ysc. there exists first-layer B that can minimize the target loss: ming Lirg (frg( B) = 0. However, changing the last layer may not be sufficient: there exists such A, Pares Purg such that the target loss is non-zero for any choice of last layer u: for all i, min, Lug(U, Barc) > 0.
Proposition 1.
Proof. Let Bare, Uwe be minimum loss solutions so that Yorcc Obsretbl BercWare) for all Tarc, Merc Denoting B A-1 we have for all Tirg
0 (Brr)) = C A Adtare) = Osee Brre = Msrc = ytrg.
(3)
Therefore, this pair of parameters B achieves Ling( B) =0 O.
We construct counterexample showing the impossibility of last-layer tuning follows. Recall that <(-) is the elementwise ReLU function. Let A = -1, an invertible diagonal matrix with all entries 11. Let the source distribution be so that Bire has only positive entries for all Tarc in its support. Then for any v. we have v Burc.eegg) vol Bwettsre) 0, so the expected loss is positive. Therefore, min, Ling (u, Bsrc) O.
For all I, Pares Purg with Itrg = Jsrc, and yhrg Uyee there exists last-layer that can minimize the target loss: min, Lug(u, Bssc) O. However, changing the first layer may not be sufficient-ther exists such t, Parcs Pirg such that the target loss is non-zero for any choice of first layer B: ming p Lerg(Uere; B) >
Proposition 2.
Proof. Let Bare, Usrc be minimum loss solutions so that Yarc = -frrepp( (Brc for all Tare, Varc. Let tu Then for all trr
vo vo (BscTtrg) ti (ercttrr) ty/scc Vtrg.
(4)
Therefore, this pair of parameters Bare achieves Lirg(v (v, Bare) = O.
We next construct counterexample showing that tuning only the first layer may not be sufficient.
Recall that o(-) is the elementwise ReLU function. Let = -1 Let the source distribution be so that
both Basse and ferc consist only of positive entries for all &'re its support. For any B, both ées and
Φ( (Btrrg) consist only of positive entries, so Oxre BBrgg) cannot express Ytrg Usrr < O. so the expected loss positive for any B. Therefore, ming Larg (Uore, B) O.
B.2 PROOF OF THEOREM I IN SECTION 3.2
We introduce some additional setup, prove key lemmas which bound the loss of first-layer tuning and full fine-tuning respectively, and then this immediately implies Theorem 1.
15

---

Published as a conference paper at ICLR 2023
Defining the label distribution. We assume that P(y 2)) is the same in both the source and the target. That is, we assume there exists some ve,, B. such that y = w.! B,æ for both P and Perg- Let w. B. U..
Defining the covariate distributions. Now, we define the distribution over the inputs x. Let the source distribution Parc have density on a dimeniinnsi subspace, where dare < d (recall that the imput dimension is d). Formally, this means that there exists some Secc E Rddddrr with linearly independent columns, and some distribution P(ee with density on Rue such that Pare has the distribution Sare where 2 P.c.
We assume a non-degeneracy condition that the optimal model w. does not map (non-zero) source examples to O: for all source examples E colspace(Sard) if of O, then w. + 0. If w. were random or had some noise then this would hold with probability 1.
Suppose we have an orthogonal distribution Porth which has density on dorth dimensional subspace. Formally, this means that there exists some Sorth E dxddt with linearly independent columns,
and some distribution p(=) with density on IRdorsh, such that Porth has the distribution Sorh where p(*) We assume that the support of Porth and Pare are orthogonal -tht is, the columns of
Sorth and the columns of Sare are all orthogonal.
The target distribution Pirg is an equal mixture of the source distribution Parc and the orthogonal distribution Parch:
1 (Parc + Porth)
Pug = This means that sample from Parg. with probability 0.5 pick sample from Pare and with probability 0.5 we pick a sample from Porth-
(5)
2
First layer tuning gets 0 target loss. We first show that first-layer tuning gets 0 loss on the target distribution:
Lemma 1. For any > 0, suppose
> 10dorth log 3. Then with probability at least 1 - 8, first-layer Lirg( (un BR =
tuning gets 0 loss at convergence:
Proof. We first note that first-layer tuning does not update the head v, so we have in Usrc-
Convex so converges. Note that the loss Lug is convex in B. To see this, note that we can write:
Log(v, Σ (v Br 1=1
Virg )2 (Trratngo B) Vtrg (
Tr(ing B) is a linear function of B, so this is simply a least squares regression problem and is convex. This means that gradient flow converges a minimizer of the frain loss:
(4)
Larg(fure B - Lere( B) YB.
(8)
However, since bere + O, there exists B such that Lerg( B) = 0, so since the loss is non-negative, this implies that:
Legg BR 0.
Define ID and orthogonal training examples. We note that every example sampled from Perg comes from exactly one of Porth or Pare"" We group examples based on which distribution they come from. Let X and Xorth denote the source and orthogonal examples respectively, where X, denotes
Xar: = (mrr
(t) Firg E colspace(Ss.r.)) Xmrth = {>:10
Ferg E colspace(Sorth.)
(10)
Enough to get basis correct. Since are working with linear models, it suffices to get all examples in a basis correct to get the entire subspace correct. Stated formally, if e Br = U_ B. and 8. Bx. v._ B,x', then the equality holds for any linear combination as well: v B(ar 3x' - (.x + Bx') for all a, 3. So to show that Lirg B = 0 suffices show that 0g. B x v. B.x for some set of x that spans the support of Pare and Porth-
Since the distributions have density on some subspace, x: is almost surely non-zero.
16

---

Published as a conference paper at ICLR 2023
Xorth spans orthogonal subspace. A standard application of Hoeffding gives us that with proba- bility 2 21-6/2, 6/2. we have at least dorth examples from the orthogonal distribution: [Xorth| 2 dorth- Since Porth has density these examples will span the orthogonal subspace almost surely: span (Monnt) = colspace (rtth)) Intuitively, since Porth has density we will sample points in different directions.
dorth dimensional subspace, from e.g., Lemma 3 in Xie et al. (2021)
Get all examples in orthogonal subspace correct. Since we have 0 training loss, and Xorth spans the support of Porth. this means that we get all examples the orthogonal subspace correct that is, for all x in the support of P Porth, we have: fg Bi r 0, B.
Get all examples in source subspace correct. For the source subspace we will split into two cases. First, we define the region of the source subspace (alspace(Sare) that is orthogonal to all source training examples Xare
srC = fz E colspace(Ssro) Va' E X 1 x'}.
(11)
Since we have 0 training loss, we get all examples in the span of the source training examples correct: for all x E span(Xare) we have: B x v. B.z.
For all x E X from Lemma A.3 in Kumar et al. (2022a) we have that B x = Bn = B
But this means that fiyrc BA x = dggc BuC.. Since we assumed that we pretrained to get 0 loss on the source have: fg B U. B,z.
Combining these two cases, we get all examples in the support of Pare correct.
Pun is a mixture of Pare and Porth- Since we get 0 loss on the support of Pare and support of Poeth, and Pin is mixture of the two, we get loss on Pung as well:
Lurg(Uwe, B = 0
(12)
Lemma 2. Suppose the representation dimension is I-dimensional (k- 1) and dare >> n. Then with probability least full fine-tuning gets positive (non-zero) loss at all times
Lerg (t), Bre (t))
(13)
First, we note that since n < dgres and we only have n training examples, our training examples do not span the source distribution. That is, there exists some source example x, E colspace(SS.r)
Proof.
which is in the support of Psrc, Ix + O, which is orthogonal to all the training examples: v, Firg for all 1 < S n. Choose such an Zs.-
(3)
We note that since k (the representation dimension is 1), (f) IR xd are row vectors. For notational convenience, let b(t) = (t) b = BT, and byc Byc. so we have for example Virg x.b.... Zerg
scalar, and B Ba (t), B,
(a)
Un cannot change for zero loss. First, show that if uge (t) + êre then Lurg (Ux (t), Ba (t) 0. Since V, is orthogonal to the training examples, from Lemma A.3 in Kumar et al. (2022a), we have b(t) T = b5 Since pretraining gave us O loss on the source distribution, we have TT v. x.. Recall that we assumed that W x. + if + O, which implies that &T. 2. A O since pretraining gets all the source examples right, and the ground truth label for all non-zero source examples is non-zero. But then en (t) # Ósrc. we have: un (t)b(t) T Tx * 0grc = v.b x. Since Pare has density, can construct small ball B of non-zero probability around x, such that for all x E B, Un (t)b(tt) T x + v.bl x. This implies that Lung (vn (t), (t)) >
en (t) must change for zero loss. Next, suppose that (n Bt (t)) 0. We will show Un (t) # Egrc- Suppose for the sake of contradiction that en (t) = Usrc.
From Lemma A.4 in Kumar al. (2022a) (also see Theorem 2.2 in Du et al. (2018)), we have:
222
iT B = (t)( b(t) b(t).
(14)
17

---

Published as a conference paper at ICLR 2023
Since Usre (t), this gives us:
b b(t) b(t).
(15)
Let R colspace(Ssre) be the source subspace. Since Par is a subset of Pirg. we have that
(vn b()) also gets loss on the source distribution, and we have:
[lx(v.b.) = 11 (Usreburr))) = [v (t))tt)).
(16)
Since vo = Un (t), we have:
IT (t) = (R((((()((()))))
(17)
Since Un (t) + (otherwise we would get source examples wrong):
(brre) = ll)()(()).
(18)
Let = colspace(Sortt) be the orthogonal subspace. From Equation 15 and Equation 18, we have:
(b (blt)
(19)
But to get O loss on T, we have: (t)(((t)))((t) = [1 b.). Which implies:
v²
(b(()))
11 (b.))
(20)
(en (t))
From Equation 19 and since Usec = un (t), we have:
[ (u (1))
II (b.))))l
(21)
Recall that the way we got byc was we initialized by = B, ~ NO, Ol We then ran gradient descent on the source distribution However, from Lemma A.3 in Kumar et al. (2022a) this does not change the projection onto components orthogonal to the source distribution. In other words, [[()7(bbr))) )13 = [bo)r(b)) I. However, this is a random variable with density, so the probability that this is exactly equal the RHS of Equation 21 which is fixed number, is O. This is contradiction.
Wrap Either ways, if un (t) = Usrc or (t) * Srec we have: Ling (Un (f), Ba > 0.
Proof of Theurem I. The proof follows directly because Lemma 2 shows that full fine-tuning gets positive (non-zero) loss but Lemma 1 gets zero loss.
B.3 ADDITIONAL DATASET DETAILS
Below, we provide additional information the datasets used in our experiments.
CIFAR-10 + CIFAR-10-C (Krizhevsky et al., 2009; Hendrycks & Dietterich, 2019): The task is classify images into 10 classes, where the target distribution contains severely corrupted images. We run experiments over 14 of the corruptions (frost, gaussian blur, gaussian noise, glass blur, impulse noise, jpeg compression, motion blur, pixelate, saturate, shot noise, snow, spatter, speckle noise, and zoom blur). For the main experiments, we tune on 1000 images from CIFAR-10-C and evaluate corrupted images from each of the corruptions. We use the data loading code from (Croce et al.. 2020), which has 5 levels of severity, and we evaluate with the most severe level. In our main experiments, we report the accuracies averaged across all corruptions and the average error for all corruptions.
ImageNet -> ImageNet-C (Deng et al., 2009; Hendrycks & Dietterich, 2019): The task is to classify images into 1000 classes, where the target distribution contains severely corrupted images. We run experiments over 15 of the corruptions (brightness, contrast, defocus blur, elastic transform, fog, frost, Gaussian noise, glass blur, impulse noise, jpeg compression, motion blur, pixelate, shot noise, snow, zoom blur). For the main experiments, we tune on 5000 images from ImageNet-C. evenly split between classes, giving 5 corrupted images per class and evaluate corrupted images from each of the corruptions. Similar to CIFAR- 10-C, we evaluate with the most severe level. We also report the accuracies averaged across all corruptions and the average std error for all corruptions.
18

---

Published as a conference paper at ICLR 2023
Living-17 and Entity-30 (Santurkar et al., 2020): The task is to classify images into of 17 animal categories or one of 30 entities. These datasets present subpopulation shifts, in that while the ID and OOD distributions have the same overall classes, they contain different subpopulations of those classes. For Living-17, 17, we tune on 850 images from the target distribution, evenly split between the 17 classes, giving 50 images per class. For Entity-30, tune 1500 images from the target distribution, evenly split between the 30 classes, giving 50 images per class.
Waterbirds (Sagawa et al., 2019): The task is classify images as being a "waterbird" or "landbird". The label is spuriously correlated with the image background, which is either "land" or "water." The source distribution is the training set while the target distribution is a balanced subset with equal amounts of each bird on each background. In the training data, 95 % of the waterbirds appear on water backgrounds, and 95% of the landbirds appear on land backgrounds, so the minority groups contain far fewer examples than the majority groups. We tune on 400 images from the target distribution, evenly split between the groups of (bird, background) pairs, giving 100 images per group.
CelebA (Sagawa et al., 2019): The task is to classify the hair color in images as "blond" or "not blond", and the label is spuriously correlated with the Male attribute. The source distribution is the training set while the target distribution is a balanced subset with equal amounts of each of the four (hair color, gender) groups. We tune on 400 images from the target distribution, evenly split between the 4 groups of (hair color, gender) pairs, giving 100 images per group.
(Bandi al., 2018): This dataset part of the WILDS (Koh et al., 2021) datasets and contains roughly 450,000 images in the source distribution (Train) and 84,000 images in the target distribution (OOD test) of size 96 x 96. comprises of medical images collected from 5 hospitals where difference in sttcce//data-processiii
Camelyon17
between different hospitals produces natural distribution shift. We pre-train on the 450,000 images of the source distribution and use 100 label-balanced images (50 per class) from the target distribution for fine-tuning. We use another 100 babel-blaanced target distribution images for tuning hyper- parameters, and report the performance of the fine-tuned model on the rest of the images of the target distribution.
FMoW (Christie et al., 2018): This is also part of the WILDS (Koh et al., 2021) datasets and its source distribution contains 520,000 satellite images of size 224x224 from 5 geographic regions. The task is to classify one of 62 building or land use types. For target distribution, we use Africa test, i.e., the subset of the OOD test data belonging to Africa region, which has roughly 2,500 images. We use 62 label-balanced images from target distribution for fine-tuning and report the accuracy on the rest of the target distribution images.
B.4 ADDITIONAL DETAILS FOR SUPERVISED TRANSFER LEARNING EXPERIMENTS
Below, we provide additional details for our experiments real data, including tuning details. For all datasets and experiments, we early stop according to the best accuracy on a held-out validation subset of the labeled target data.
Flip: We use the Standard (Krizhevsky et al., 2009; Hendrycks & Dietterich, 2019) and CIFAR- model from (Croce al., 2020), which is trained on the source CIFAR-10 distribution. We fine-tune on the labeled target data for 15 total epochs. We tune over the 3 learning rates [1e-3, 1e,4, le-5) for all methods except last-layer fine-tuning, where we tune over [1e-1, 1e-2, 1e-3), and we use a weight decay of 0.0001 for all methods.
CIFAR-10 + CIEAR-10----
ImageNet + ImageNet-C (Deng et al., 2009; Hendrycks & Dietterich, 2019): We use the Standard pre-trained model from (Croce et al., 2020), which is trained the source ImageNet distribution. We then fine-tune on the labeled target data for 10 total epochs. We tune over the 3 learning rates (le-3, 1e-4, 5) for all methods, and use weight decay of 0.0001 for all methods.
Living- 17 and Entity-30 (Santurkar et al., 2020): We first train on the source data for 5 epochs, tuning only the head for 3 epochs and then fine-tuning all for 2 more epochs, following LP-FT (Kumar et al., 2022a) and using the Adam optimizer. We then fine-tune on the labeled target data for 15 epochs. We tune over the learning rates (0.0005, 0.0001, 0.00001) } for all methods and do not use any weight decay.
Waterbirds (Sagawa et al., 2019): We first start with a ResNet-50 pretrained on ImageNet and train the source distribution for 300 epochs, taking the best checkpoint based on early stopping and using the Adam optimizer. We then fine-tune on the labeled target data for 100 total epochs. We tune over the 3 learning rates (0.005, 0.001, 0.0005) for all methods and use weight decay of 0.0001.
19

---

Published as a conference paper at ICLR 2023
Method
mCE (%)
Vanilla ResNet-50 (Hendrycks & Dietterich, 2019)
76.7
Cross-Var Val
61.7
Full Fine-tuning
62.8
Gradual (First + Last)
62.4
Gradual (Last - First)
63.6
LI Regularize (Xuhong et al., 2018)
64.7
Auto-SNR
62.9
Auto-RGN
61.9
Table 6: For completeness, we report the mean corruption error (mCE) on ImageNet-C, which weights the target distribution error by the difficulty of the corruption. We report the average accuracies on the target distribution in Figure 2 and Table 4, and that the metrics are correlated for experiments, as the best performing methods according average accuracy also the best for mCE.
First Layers
Block
Block 2
Block
Block 4
Last
All
No Tuning
74.6 (1.9) 75.3 (1.9) 81.6 (1.2)
86.1 (0.7) 82.9 (1.5) 72.6 (1.3) 77.3 (0.4)
65.6 (0)
Table 7: Surgical fine-tuning results on Living-17 17 (% accuracy) initialized with a CLIP ViT-B/16. We find that similar to the results using a ResNet-50 architecture, fine-tuning a single parameter block with this vision transformer architecture outperforms full fine-tuning, and in particular, middle block still performs best for this feature-level shift.
CelebA (Sagawa et al., 2019): We first start with ResNet-50 pretrained on ImageNet and train on the source distribution for epochs, taking the best checkpoint based on early stopping and using the Adam optimizer. We then fine-tune on the labeled target data for 50 total epochs. We tune over the learning rates (0.001, 0.0005, 0.0001] } for all methods and use a weight decay of 0.0001.
Camelyon17 (Bandi et al., 2018; Koh et al., 2021): We start with vision transformer, CLIP ViT-B/16 (Radford et al., 2021), pre-trained on the CLIP datasets. Next we fine-tune the model on Camelyon 77 train dataset using an SGD optimizer with initial learning rate 0.0001 for 3 epochs. We use a cosine annealing learning rate scheduler (Loshchilov Hutter, 2017) and batch size 32. Finally, we fine-tune on the labeled target data for 10 total epochs with the same setting before, except we tune over learning rates (10 10- x 10-5
10 10 6, 3. x 10 10-7, 10-5)
FMoW (Christie et al., 2018; Koh et al., 2021): 1): Similar to Camelyon17, we start with vision transformer, CLIP ViT-B/16 (Radford et al., 2021), pre-trained on the CLIP datasets. Next we fine- tune the model on FMoW train dataset using an SGD optimizer with initial learning rate 0.0003 for 5 epochs. We use cosine annealing learning rate scheduler (Loshchilov Hutter, 2017) and batch size 32. Finally, we fine-tune on the labeled target data for 10 total epochs with the same setting as before, except we tune over learning rates (10 10 0.0003, O. 0.0001, 0.001, 0.01, 0.25).
In Table 6, for completeness, we additionally report the mean corruption error (mCE), which weights the target distribution error by the difficulty of the corruption, ImageNet-C since that is common metric used for the dataset. We find that these results give similar conclusions as the average accuracies reported in Table 4, with Cross-Va Val and Auto-RGN performing the best.
We additionally include an ablation where we use a CLIP VIT-B/16 (Vision Transformer) as our initial model pretrained on the Weblmage'Text dataset. This model consists of 2 first layers followed by 4 transformer blocks and last layers. We analyze surgical fine-tuning with this model architecture on the Living-17 17 dataset, which has feature-level shift. In Table 7, we find that our results in Section hold similarly using this vision transformer, as tuning only a middle block outperforms full fine-tuning or tuning any other block of layers.
B.5
MORE LARGE VISION TRANSFORMER EXPERIMENTS
Prior work has shown that when fine-tuning vision transformers, task accuracy on held-out data is substantially higher when using the AdamW optimizer rather than SGD (Kumar al., 2022b). We evaluate the performance of surgical fine-tuning on two large pre-trained vision transformer models (CLIP VIT-B/16 and ViT-I/114 while fine-tuning with AdamW. We follow the experimental setting of Kumar et al. (2022b) as closely as possible:
20

---

Published as a conference paper at ICLR 2023
Dataset
Camelyon17 ViT-B/16
Camelyon17 ViT-B/16
FMoW
FMoW
FMoW
Architecture
ViT-B/16
VIT-B/16
ViT-I/14
Optimizer
SGD
AdamW
SGD
AdamW
AdamW
No fine-tuning
86.2
96.2
35.5
41.7
50.3
All
92.3 (1.7)
96.4 (0.4)
38.9 (0.5)
29.2 (2.6)
52.9 (1.2)
Embedding
95.6 (0.4)
96.9 (0.1)
36.0 (0.1)
41.8 (0.1)
50.1 (1.6)
First three blocks
92.5 (0.5)
95.0 (0.2)
(1.0)
26.7 (1.3)
51.6 (0.9)
Last three blocks
87.5 (4.1)
96.6 (0.1)
(2.6)
46.1 (2.3)
52.4 (0.6)
Last layer
90.1 (1.5)
96.7 (0.1)
36.9 (5.5)
42.6 (0.1)
52.3 (0.3)
Table 8: OOD set accuracies after surgically fine-tuning different parameters in vision transformer models for two WILDS datasets. Bold numbers represent superior results for a dataset, and we also report the standard deviation from runs with different seeds.
Camelyon17 (Bandi et al., 2018; Koh et al., 2021). We first train a pre-trained CLIP ViT-B/16 (Rad- ford et al., 2021) model the train split of the Camelyon17 dataset using A optimizer with initial learning rate 10 for 3 epochs. We use a cosine annealing learning rate sched- uler (Loshchilov & Hutter, 2017) and batch size 32. Finally, we fine-tune the labeled target data for 10 total epochs with the same setting as before, except we tune over learning rates (10 10 3 10 5,-5,,5 10TH 10 6, 10 10-? 10 10.
FMoW (Christic et al., 2018; Koh et al., 2021). We similarly train a pre-trained CLIP ViT-B/16 or ViT-L/14 model on the train split of the FMoW dataset with initial learning rate 10 for 5 epochs. We use a cosine annealing learning rate scheduler (Loshchilov Hutter, 2017) and batch size 32. For the VIT-L/14 setting,
larger 336 x 336 images. Finally, fine-tune labeled target data for 10 total with the same setting as before, tuning over learning rates {10 10 10-b, 5, x 10 10-5, 7 10 10-6 3 10-6 10-7 10 10-8)
We show AdamW fine-tuning results in Table While surgically fine-tuning the right layer continues to improve over no fine-tuning, the relative advantage compared to fine-tuning all layers is smaller than what observed with SGD in Table 7. We observe instability in fine-tuning all layers of a ViT-B/16 network for FMoW target distributions. Fine-tuning later layers seems to consistently improve performance without running into such instability issues. We leave further investigation of such properties of fine-tuning ViT models to future work.
B.6 COMPLETE UNSUPERVISED ADAPTATION RESULTS
Method. We experiment with MEMO (Zhang et al., 2021 la) as our unsupervised adaptation method. Given test image MEMO first takes "adapt" stage, where it minimizes the marginal entropy over standard augmentations of z, then it takes a "test" step, where the network predicts a label for æ. Note that MEMO tests single image at a time, i.e., the test batch size is 1. We also consider the two following variations.
This version is discussed the origin work. Here after predicting the labels, we reset the weights of the network the pre-trained ones, i.e., undo the "adapt" stage.
Episodic:
We also consider the online variation of MEMO, where we do not reset the weights after each test image, i.e., we accumulate the "adaptation" changes over test images.
Online:
We have also experimented with TENT (Wang et al., 2020), but since TENT only updates the batchnorm modules (whereas MEMO updates all parameters), freezing parameters with TENT did not produce expected results and we did not pursue it further.
Dataset and Network. We use the CIFAR-1O-C and ImageNet-C corruption datasets for our experiments. For CIFAR-IO-C, we use the same ResNet-26 (He et al., 2015) pre-trained model used by MEMO (Zhang et al., 2021a), which is available in their GitHub repository. For ImageNet-C, we -mall architecture and pre-trained weights used by Zhang et al. (2021a).
For CIFAR-IO-C, we use 1000 corrupted test image for hyper-parameter tuning and report the test accuracy on the held out 9000 examples. We consider the following hyper-parameter grid:
Sypper-parameter.
21

---

Published as a conference paper at ICLR 2023
Corruption
Settings
Tuned Layers
gauss
impulse
shot
fog
frost
snow
elast
contr
defoc
No Adaptation
51.6
49.7
55.2
72.0
66.9
75.9
74.4
85.9
70.3
75.9
Episodic
All
56.31
56.39
59.94
77.42
71.93
78.63
78.99
88.15
72.66
76.32
First layer
53.11
51.29
56.53
73.27
68.41
77.16
76.46
86.53
70.82
76.11
First layers
53.20
$1.21
56.58
73.48
77.06
76.41
86,47
70.67
76.15
First 2 blocks
53,20
51.36
56.63
73.37
68.61
77.22
76.45
86,56
70.73
76.09
Last
$1.69
49.82
$5.24
72.01
67.05
85.98
70.28
75.98
Online
All
51.48
49.53
55.85
75.69
72.66 64
77.36
76.16
86.83
70.88
79.89
First layer
68.14
59.21
71.10
79.61
76.20
79.29
76.86
82.18
First 2 layers
68.00
60.92
73.07
79.22
76.16
79.34
76.18
86.65
73.97
81.97
First 2 blocks
67.84
59.30
70.96
79.11
75.91
79.44
76.46 46
86,91
74.67
82.12
Last
51.61
49.71
55.21
71.99
66.92
75.87
74.49
86.00
70.27
75.91
Table 9: MEMO (Zhang al., 2021a) with parameter freeziig results on CIFAR-10-C 10-C on 10 representative corruptions and severity level 5. Bold numbers represent superior results for a particular corruption.
Learning rate: 10-3, 10-1, 10-5 and 10-6, then 2.5x, 5x. 0.5z of the best learning rate from before.
Steps: 1,
Weight decay: 0, 10 10-2, 10-1
Number of augmentations per image: 32
For ImageNet-C, we do not do any hyper- hyper-paramen parameter tuning and simply use the best described by Zhang et al. (2021a), which are:
Learning rate: 0.00001
Weight decay: 0.01
Steps:
Number of augmentations per image: 32 (This is 64 Zhang et al. (2021a), but we use 32 for computational cost)
Moreover, for ImageNet-C, the experiments are done over 2000 test images (the first 2 test image per class for each of the 1000 classes) instead of the entire reasons. This produces numbers that are slightly different from the original work, hence we include the adaptation baselines well for fair comparison.
of 50, 000 images, for computational
Finally, in practice, we saw that AdamW and SGD optimizers work better for the episodic and online setting, respectively.
Layers. We use the following naming convention for the layers of ResNet-26:
First: Only the first conv layer.
First layers:
The first layer of the entire network, and the first conv layer within the first
block.
First blocks:
The first conv layer, and the first block.
Last: The last fully-connected (FC) layer.
For RVTl-small:
First layer: First conv layer inside the first transformer block.
First block: First transformer block.
Last: Head or the final fully connected layer.
Results. Table 9 and Table 10 show results for MEMO with parameter freezing for CIFAR- and ImageNet-C respectively on severity level 5 and on representative set of corruptions.
22

---

Published as a conference paper at ICLR 2023
Corraption
Settings
Tuned Layers
impul
sbot
Tog
Trost
clast
plass
motion
pisel
No Adapt
43.80
45.90
42.40
55.20
45.95
50.05
73.95
53.25
35.00
19:35
40.35
32.05
$2.60
60.60
Episadic
45.35
47.40
44.60
55.35
47.05
50.25
74.35
54.85
36.25
20.55
.40
33.95
55.55
61.70
layer
First block
44.00
46.30
42.60
55.40
16.00
50.15
74.10
53.20
35.15
19.75
40.75
32.25
53.20
60.75
Last
43.75
45.95
42.40
55.25
46.0
50.05
73.95
53.30
35.05
19:35
40:40
32.05
52.60
60.60
Online
1.05
0.90
1.40
0.35
3.40
1.60
0.80
0.80
2.75
1.05
2.50
2.25
2.70
3.40
layer
First block
46.50
49.65
46.75
55.80
47.40
$2.55
74,80
54.15
40.50
24.75
41,45
34.15
56.85
61.05
Last
43.80
45.90
42.10
55.15
15.95
50.00
73.90
53,20
35.05
19.35
10.35
32.05
52.60
60.60
Table MEMO (Zhang et al., 2021a) with parameter freezing results on ImageNet-C on representative corruptions and severity level 5, Bold numbers represent superior results for a particular corruption.
23


Reviews:
[{"review_id": "f6fUMhEX8s", "reviewer": "Reviewer_QAba", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows fine-tuning a (contiguous) subset of layers matches or outperforms commonly used fine-tuning approaches including full fine-tuning or fine-tuning the last few layers. The authors evaluated the performance of surgical fine-tuning diverse layers of choice on 7 different distribution shift scenarios (input, feature, and output-level shifts). In addition to these empirical results, they tried to validate theoretically in an idealized two-layer neural network setup to explain why the earlier and later layer fine-tuning achieve best performance for input and output-level shifts, respectively. At last, the authors investigated several criteria for automatic selection of which layers to tune for better performance, and they empirically demonstrated that the automatic fine-tuning outperforms full fine-tuning.", "summary_of_the_review": "The authors investigated how fine-tuning can be adapted to diverse distribution shifts by showing empirical results and theoretical verification, followed by proposing a new type of regularization (several criteria to control extent of tuning per layer) to achieve a better performance than a commonly used approach (full fine-tuning). But, the paper can be more interesting to read by the potential readers if they can show results for real-world scenarios that are likely to have more than two types of distribution shifts.", "strength_and_weaknesses": "Strength\n• For empirical studies, they categorized distributional shifts into three main scenarios including input, feature, and output levels to analyze optimal selection of layers to tune for each scenario with plausible explanations. \n• This type of study usually reports experimental results only without theoretical verification. But this paper somehow provides theoretical verification using a simplified two-layer neural network. \n• Diverse experiments have been conducted to make their claims. \n• Beyond the explanations of the empirical results, they leveraged the findings to develop a training pipeline with automatically selecting layers to tune, and demonstrated the proposed approach achieves better performance than simply full fine-tuning which has been mostly used by researchers without any doubts. \n• In addition to supervised learning, they analyzed unsupervised learning that otherwise would be requested by reviewers. \nWeakness\n• Can you explain the rationale about why only contagious subset of layers need to be tuned instead of discrete subset of layers? \n• In this study, distribution shifts are categorized into three levels including input, feature, and outputs. However, it would be interesting to show results for combination of scenarios like input+feature, feature+outputs, inputs+outputs, and input+feature+outputs shifts. Especially, when we want to fine-tune ImageNet pretrained models on medical images (which is commonly used by biomedical researchers), do we expect full fine-tuning achieves the best performance instead of subset of layers to tune as natural images to medical images have distributional shifts of input, feature, and outputs levels. These results could be much more interesting because large distributional shift scenarios are more like real word situation compared to what’s shown in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "REIvUH8XFKl", "reviewer": "Reviewer_Y4ag", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, an interesting finds are demonstrated: instead of always finetuning the last layer of the model, tuning different layers works best for different types of distribution shifts. 3 types of shifts are discussed: (1) Input-level shift, first-layer finetuning is better; (2) Feature-level shift, mid-later blocks are better; (3) Output-level shift, last layer is better;", "summary_of_the_review": "This paper present a counter-intuitive finding, with detailed experimental and theoretical analysis. However, the major concern is that the finding is validated on small datasets, which might not be able to generalize to larger datasets finetuning.", "strength_and_weaknesses": "## Strength\n- This paper present a counter-intuitive finding, with detailed experimental and theoretical analysis\n## Weakness\n- Datasets used are relatively small. It is fine that this work focus on \"fine-tune the pre-trained model on a small target dataset\". However, it would be more persuasive to show some results on large model + mid-size dataset. In industry, this is a more common setting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NlCWvztnieO", "reviewer": "Reviewer_eWpB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper performs an analysis of what the fine-tuning strategy should be when encountering various distribution shifts. They classify the different distribution shifts into three categories and show that for each category of distribution shift, different layers of the model should be surgically fine-tuned. This analysis results in interesting outcomes, for example, for shifts where the inputs are changing (like CIFAR-C), contrary to the conventional wisdom of fine-tuning the last few layers to re-use the learned features, it is better to fine-tune only the early layers of the network. The paper also discusses and compares some ways to automatically decide which layers to fine-tune. ", "summary_of_the_review": "Overall, although the work has some interesting analyses and outcomes/results, it is still unclear how the method of surgically fine-tuning a network would compare to existing methods for domain adaptation or methods that perform test-time adaptation (adjusting the covariate shift, etc.). Therefore, my recommendation for this paper would be marginally below the acceptance threshold (5).", "strength_and_weaknesses": "Strength(s):\n1. The paper nicely and cleanly classifies the distribution shifts into three types and empirically shows how different fine-tuning strategies are optimal for different types of distribution shifts.\n2. The synthetic experiment in the paper is clever and successfully demonstrates that surgically fine-tuning the relevant layers/blocks can result in a significant performance gain.\n3. New strategies (Auto-SNR and Auto-RGN) for automatically selecting which layers to fine-tune were a nice outcome of the theoretical analysis performed by the authors.\n4. The empirical observations to fine-tune only a subset of layers for different distribution shifts are well supported by multiple datasets having different types of distribution shifts. \n\nWeakness(es)/Suggestion(s):\n1. For table 3, in addition to the L1 regularization method, it would be nice to include other methods like gradually unfreezing or different learning rates for different layers, etc. \n\n(some methods from the existing work: \nA. Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661–2671, 2019. [page 1]\nB. Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum id=B1g8VkHFPH. [page 1, 9]\nC. Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9594–9602, 2021. [page 1, 9]\nD. Youngmin Ro and Jin Young Choi. Autolr: Layer-wise pruning and auto-tuning of learning rates in fine-tuning of deep networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2486–2494, 2021. [page 1]\nE. Miguel Romero, Yannet Interian, Timothy Solberg, and Gilmer Valdes. Targeted transfer learning to improve performance in small medical physics datasets. Medical Physics, 47(12):6246–6256, 2020. [page 1])\n\n2.  Although the observation of fine-tuning different subsets of layers for different distribution shifts is an interesting one, it is not clear how the method of surgically fine-tuning a network would compare to existing methods for domain adaptation or methods that perform test-time adaptation (adjusting the covariate shift, etc.) [F,G,H]. It would strengthen the paper if authors can show that surgical fine-tuning is a better and simpler option to tackle distribution shifts as compared to training exotic models that enable robustness or allow for domain adaptation. \n\nF. Improving robustness against common corruption by covariate shift adaptation, NeurIPS 2020\nG. Adaptive Denoising via GainTuning, NeurIPS 2021\nH. Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization, 2020\n\n3. Is the reported metric for CIFAR-C and IN-C (1- mean corruption error (mCE)), in Tables 1,2, and 3, and Figures 2, 3 and 4? Because mCE is a standard metric to track when dealing with corruption datasets.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster