How to fine-tune vision models with SGD
Authors: Ananya Kumar
Ruoqi Shen
Sebastien Bubeck
Suriya Gunasekar
Venue: Submitted to ICLR 2023 (2023)
Keywords: fine-tuning
SGD
freezing layers
distribution shift

Abstract: SGD (with momentum) and AdamW are the two most commonly used optimizers for fine-tuning large neural networks in computer vision. When the two methods perform the same, SGD is preferable because it uses less memory and is more efficient than AdamW. However, when evaluating on  downstream tasks that differ significantly from pretraining, we find that across five popular benchmarks SGD fine-tuning gets substantially lower accuracies than AdamW on many modern vision models such as Vision Transformers and ConvNeXts---especially out-of-distribution (OOD). We find that such large gaps arise in instances where the fine-tuning gradients in the first (``embedding'') layer are much larger than the rest of the model. Our analysis suggests an easy fix: if we simply freeze the embedding layer (0.7\% of the parameters), SGD performs competitively with AdamW while using less memory across a suite of benchmarks. Our insights lead to state-of-the-art accuracies on popular distribution shift benchmarks: WILDS-FMoW, WILDS-Camelyon, BREEDS-Living-17, Waterbirds, and DomainNet.

Under review as a conference paper at ICLR 2023
How TO FINE-TUNE VISION MODELS WITH SGD
Anonymous authors
Paper under double-blind review
ABSTRACT
SGD (with momentum) and Adam are the two most used optimizers for fine- tuning large neural networks in computer vision. When the two methods perform the same, SGD is preferable because it uses less memory (12 bytes/parameter) than AdamW (16 bytes/parameter). However, on a suite of downstream tasks, especially those with distribution shifts, we show that fine-tuning with Adam W performs substantially better than SGD on modern Vision Transformer and Con- vNeXt models. We find that large gaps in performance between SGD and Adam occur when the fine-tuning gradients in the first "embedding" layer are much larger than in the rest of the model. Our analysis suggests an easy fix that works consistently across datasets and models: merely freezing the embedding layer (less than 1% of the parameters) leads to SGD performing competitively with A while using less memory. Our insights result in state-of-the-ar accu- racies on five popular distribution shift benchmarks: WILDS-FMoW, WILDS- Camelyon, Living- 17, Waterbirds, and DomainNet.
INTRODUCTION
Fine-tuning large pretrained models on downstream tasks has become a dominant approach in deep learning (Komblith et al. 2019; Chen et al.] 2020 Zhai et al.] (020)). The two most commonly used optimizers in current practice are SGD and Adam (Kingma & Ba 2015 Loshchilov & Hut- ter 2019 While most modern vision architectures (ViT', ConvNeXts, and variants) increasingly use Adam W for pretraining. is still common to use SGD for fine-tuning. Part of the appeal is that SGD more memory and compute efficient: Adam maintains 4 states/parameter, while SGD only maintains 3 states/parameter (Ginsburg et al. 2019 Dettmers et al.] 2022). In training ultra-large models, the additional memory from even extra state/parameter can be costly. At the same time, in terms of fine-tuning accuracies, prior work (Dosovitskiy et al. 2021: Steiner et al.) 2021: Ku- mar al. 2022) report similar performance between Adam and SGD on ImageNet like domains that are closer to pretraining data. In contrast, we reach different conclusions when fine-tuning on datasets that are far from pretraining data or have substantial distribution shifts.
We examine 7 popular models, including vision transformers (Dosovitskiy et al. 2021; Caron al. 2021 Radford et al. (02T). ConvNeXts (Liu et al. 2022), and ResNets (Kolesnikov et al.) 2020 He et al. 2016), of different sizes and pretraining modalities. When pretrained on a large corpus and then fine-tuned, these models achieve near state-of-the art performance on downstream benchmarks. In addition to good transfer learning, we also want our fine-tuned models to handle practical distri- bution shifts gracefully. So we focus on 5 distribution shift datasets that have both in-distribution (ID) and out-of-distributtinn (OOD) evaluations: WILDS-FMOW. WILDS-Camelyon BREEDS-Living-I1 17, DomainNet. These were selected to capture different types of data shifts (sub- population shifts, spurious correlations, style shifts), including two real world shifts in medical imaging and satellite remote sensing from the WILDS benchmark (Koh et al. 202T).
Waterbirds,
We find that on newer models like ViTs and ConvNeXt, AdamW can have significantly higher accuracies, especially OOD. For example, averaged across the datasets, fine-tuning CLIP ViT- B/16 model with AdamW gets 2.1 % higher accuracy ID and 8. 1% higher accuracy OOD compared to SGD (Figure lb). These gains are consistent across models too-averaged across all models and datasets, Adam W gets 1. 1.2% higher accuracy ID and 4.0% higher accuracy OOD (Tables [
We use SGD to refer to its usage in deep learning minihatch stochastic gradient descent with momentum.

---

Under review as a conference paper at ICLR 2023
random init.
Head
finetune
12 MSAHHN
ID accuracy
OOD accuracy
prerriined
SGD
90.0%
67.9%
Trans, 12 MSA+FFN
init.
Adamm SGD (frezze-embed)
(+2.1%)
(+8.1%)
(+2.1%)
(+8.0%)
SGD (frceze-embed no mom.)
(+2.2%
(49.0%)
freeze-
Embedding
Gradual Unfreeee
(+2.3%)
(+8.3%%
(a) Simplified schematic of ViT illustrating how we do freeze-embed.
(b) Performance of different fine-tuning methods on CLIP
ViT-B/16 averaged over distribution shift datasets.
We fine-tune 7 models including ViTs, DINO, CLIP. ConvNeXt, ResNet, on 5 distribution shift datasets (Living-17, Waterbirds, DomainNet, WILDS-Camelyon WILDS-FMoW) Finc-tuning with SGD gets lower accuracics than AdamW modern architectures (transformers and ConwNeXt), especially OOD. In- terestingly, a minor tweak SGD where we freeze the first "embedding" layer (< 1% of parameters see Figure Ta] is competitive with AdamW while using lower GPU memory. Further, dropping the momentum state in SGD gives additional gains in accuracy at even lower memory cost. Gradually unfreezing a model's layers using carefully annealed learning rate can further improves accuracies, while reducing computation.
Figure 1:
A key difference between AdamW and SGD, is that AdamW normalizes the gradient update of each parameter using an estimate of their second moments. Thus, parameters with consistently high gradients will change less when using AdamW than with SGD. Towards understanding these dynamics better, we examine the gradients at each layer of our pretrained models. We find that for the models where Adam significantly outperforms SGD, the gradients at pretrained initialization of the first "embedding" layer are much larger than the gradients of the other layers.
To test if over-training of the embedding layer in fact why SGD performs worse than Adam W, we consider a minor modification where we freeze the embedding layer and tune the rest of the model with SGD (Figure la) we call this SGD (reezd. In vision transformer models, the embedding layers are only small fraction (around 0.7% for ViT-B/16) of the total parameters of the model, so priori we might not expect a substantial difference in accuracies. However, surprisingly this simple freezing of the embedding layer consistently improves SGD performance across most models and datasets and achieved ID and OOD accuracies that are competitive with or better than AdamW (Figure [bp. Averaged across all datasets and models, SGD (freezo-embed) gets 76.7% accuracy OOD (vs. 72.0% for SGD and 76.0% for AdamW). The analogous Adam (freeze- embed) gets 76.5%, which does not improve over SGD (freeze-embed), supporting that freeze- embed may be the reason Adam outperforms SGD (it is not an independent axis of improvement).
Inspired by our results from SGD (freeze-embed), we explored further variations of our method on CLIP ViTs. First, we tried a more memory efficient variation, SGD (freeze-embed, no momentum), which drops the momentum state in SGD. At least CLIP models, we see that this variation gets better accuracy than SGD (freeze-embed) while giving additional memory gains. Second, we revisit the layerwise unfreezing technique proposed in prior work like ULMFiT (Howard & Ruder 2018) and related methods (Mukherjee & Awadallah 2019 Romero et al. 2020). We find that with certain exponential decay in learning rate, a gradual unfreezing procedure in some cases can lead to further gains in ID and OOD accuracies (Figure lb]. However, a large fraction of the gain over SGD appears to come from simply freezing the embedding layer, which suggests that the embedding layer plays a key role in modern vision architectures and merits further investigation.
In terms of resource comparison, our profiling on Titan-X GPU shows that on ViT-B/16, AdamW uses 16% and 36% more memory than SGD (freeze-embed) and SGD (freeze-embed, no momen- tum), respectively. The memory overhead of Adam W increases with the size of the models. On a VIT-L/14 the memory overheads of AdamW are 18%, and 49%, respectively. Gradual-unfreezing provides a gain in computational time of about 30% over Adam
These methods and insights, while simple, lead to state-of-the-an accuracies all five datasets: WILDS- -ans--Camelyon, WILDS-FMoW DomainNet, Waterbirds, and BREEDS Living-17, while being more compute and memory efficient than AdamW.
2

---

Under review as a conference paper at ICLR 2023
2 SCOPE AND SETUP
We use the following notation: a network map fo X - y is represented as a composition of ((L) layers fo @ o f8L o... f/. BBmbes ggambed) where 9 (Ohead: fL..... 01. embedd denote all p/ombed) and f(haad) to denote blocks that can conceptually be
the parameters of the model. We use faymbdd considered the "embedding" layer and the "head", respectively.
Fine-tuning. Consider networks that have been pretrained get an initialization Apretrain We focus on fine-tuning on a labeled dataset D ~ Piinetune from a new task. Concretely, given a loss function L y x y + Rzo- we minimize the training loss L(0) Prriil ((aa)) {(f(())
using iterative optimization algorithms starting from the initialization gpretrain
We evaluate the accuracy of fine-tuned models on held-out in-distribution (ID) and out-of- distribution (OOD) test datasets. For ID evaluation, we samples D Pinettune from the same distribution as the fine-tuning training dataset. To examine whether we have learned a robust model, we consider benchmarks that also provide OOD test examples Dood Pood, which differs from the fine-tuning distribution Prinetune in practically meaningful ways.
2.1 OPTIMIZERS: SGD (WITH MOMENTUM) AND ADAMW
The two most common optimizers for minimizing the fine-tuning loss L(0) from pretrained ini- tialization are SGD (with momentum) and AdamW. We will introduce other optimizers as needed. Compared to vanilla SGD (no momentum), which only stores the parameters and gradients as opti- mizer states, SGD (with momentum) stores extra state per parameter (to track the first moment), while A stores extra states per parameter (to track the first and second moments)- see Ap- pendix B for exact updates. This corresponds to a difference between Adam W and SGD of 4GB GPU memory per 1B parameter model during training?] With the current scale of the models 100s of billions parameters and increasing, such memory overheads are very costly (Dettmers et al., [022). Thus, understanding when and how we can use the cheaper SGD compared to Adam W can signifi- cantly improve training of ultra-large scale models.
2.2 DATASETS AND MODEL ARCHITECTURES
Datasets. We choose five fine-tuning benchmarks that capture different types of data shifts (sub- population shifts, spurious correlations, style shifts), including two real world shifts in medical imaging and satellite remote sensing from the WILDS benchmark. The datasets we consider are:
1. Living-17 (Santurkar et al. 2020) is a sub- pobblaoutttion shift dataset from the BREEDS benchmark. The goal is classify an image as one of 17 animal categories with ID and OOD data from different sub-categories. For example, in the "bear" category, the ID dataset contains black bears and sloth bears and the OOD dataset has brown bears and polar bears.
2. Waterbirds (Sagawa et al.) 2020) is spurious correlation dataset where the goal is to classify an image as a "waterbird" or "landbird".. In the ID dataset, "water" backgrounds are typically correlated with "waterbird" labels, but are uncorrelated in the OOD dataset.
3. DomainNet (Peng et al. 2019)) is a domain adaptation benchmark. The ID dataset contains sketch images (e.g.. drawings of apples, elephants, etc), and the OOD dataset contains real images of the same categories. We use the version of the dataset from Tan et al. (2020).
4. WILDS-FMoW (Christie al.) 2018 Koh al. 2021) consists of remote sensing satellite images. The goal is to classify a satellite image into one of 62 categories such as "impov- erished settlement" or "hospital". The II) dataset contains satellite images from across the world between 2002 and 2012, and the OOD dataset contains images from Africa in 2017.
5. WILDS- WILDS-Camelyon (Bandi et al. 20118] Koh al. [021] medical images dataset. The task here is to classify whether a tissue slide has tumor or not. The ID and OOD datasets contain slides from different hospitals.
"The bottleneck for memory in older RssNe(X)ts is typically the number of activations, which is often much higher than the number of parameters. However, in modern large transformer training. memory requirements are of the same scale the number of parameters. Further, techniques such as gradient accumulation and gra- dient checkpointing can make the activation memory small leaving the optimizer states as the main bottleneck.
3

---

Under review as a conference paper at ICLR 2023
Model Architectures. We consider seven popular pretrained models listed below that span differ- ent architectures (vision transformers and convolutional networks), network sizes, and pretraining objectives (multi-modal, supervised, self-supervised).
(1-2) CLIP ViT-B/16 and CLIP ViT-L/14 (Radford et al. 202T): CLIP vision transformers of two sizes pretrained custom multi-modi Weblmage'Texx (WTT) dataset.
(3) ViT-B/16 (Dosovitskiy et al.] 2021):
vision transformer pretrained on Imagenet-211
(4) DINO ViT-B/I6 (Caron al. [022
self-supervised ViT pretrained on ImageNet-IK.
(5) Con-NeXt-B (Liu et al. 2022
modernized convolutional network pretrained on
ImageNet-21k using advanced data augmentations and MixUp as in Touvron et al. (202I).
(6-7) BiT ResNet-50 and BiT ResNet- 101 (Kolesnikov et al. 2020): ResNetV2 models of two
sizes pretrained on ImageNet-21k.
Bented and (head)) For vision transformer models, we consider the patch-to-token embedding layer along with its layer norm if present as the embedding layer. For convolutional networks, the embedding layer refers to the "stem" block along with the first stage: the "stem" in ResNetV2 is convolution with stride followed by a overlapping 4 x 4 convolution with stride 4. For each downstream task, we replace the final layer of all the pretrained models with randomly initialized classifier head J (
(embed)
MaxPool; while in ConvNeXt it is non-
INITIAL OBSERVATIONS, SGD, ADAMW, AND LAYER GRADIENTS
Modern deep learning models increasingly use Adam W for pretraining, where it has been repeatedly shown to produce better features for downstream tasks than SGD (Dosovitskiy et al. 2021 Liu et al. 2021 2022). For fine-tuning, on the other hand, there are no systematic studies or a definitive answer al., 2021 Touvron
as whether AdamW or SGD should be used (Dosovitskiy The ablation study in Touvron et al. (202T) even found that there is no difference in performance between AdamW and SGD when fine-tuning on ImageNet-IKK ConvNext (Liu al.) (022) and Swin transformers (Liu et al., 2021) papers report using Adam W for fine-tuning, but they do not
al. 1021),
mention a comparison with SGD. In this work we focus on better understanding the dynamics of Adam W and SGD during the fine-tuning phase. Detailed results are discussed in Section 4 We first highlight some initial observations below.
3.1 ADAMW VS SGD
Adam W outperforms SGD. We find that, generally speaking, fine-tuning with AdamW produces better results than with SGD, especially for more recent models like ViT variants and ConvNeXt. The gaps are more dramatic on out-of-distribution (OOD) test accuracies compared to in-distribution and Table for the full OOD and ID results, respectively. Averaged across the models and 5 datasets, A gets an OOD accuracy of 76.0% (vs. 72.0% for SGD), and an ID accuracy of 91.5% (vs. 90.3% for SGD). We emphasize that this happens even though we sweep over 6 learning rates and early stop. Note also that these averages slightly underestimate the gaps in newer architectures as they also include the two older ResNet models, where we did not find significant differences.
(ID). See Table and
AdamW W x SGD on BiT ResNets. Breaking down the results by model type, we find that Adam W and SGD perform comparably for older convolutional networks, namely BiT ResNet-50 and BiT ResNet- 101. For example, on a BiT ResNet-101 101, Adam W gets an average OOD accuracy of 74.5% (vs. 74.6% for SGD). However, for modern architectures including the modernized convolutional network (ConyNeXt), AdamW gets significantly higher accuracies.
3.2 EXAMINING LAYER GRADIENTS
A key operational difference between AdamW and SGD is that AdamW divides the gradient update for each parameter by a weighted running average of its second moments. So parameters with consistently high gradients will change less when using AdamW than with SGD. This suggests examining the gradients across different components of the neural network if they vary a lot, then Adam W and SGD will behave very differently.
4

---

Under review as a conference paper at ICLR 2023
40
middle layers
head layer
35
embedding layer
25
15
10
cip_b
clip_l
vit_b
dino_b
convnnxt
resnet50
resnet1.01
Figure 2:
We visualize the layer- wise gradient norms our models on DomainNet at the pretrained initialization. For each layer, we plot the average minibatch gradient norm G computed in (3.)). We highlight two special layers: gradient norms of the "embedding" layer parameters G those of the classifier "head"s Gieit are show green-triangles. The middle layer gradient norms shown as black-circles. For transformer models, we have separate (black) points for the MLP and the attention layers.
shown red-squares, and
We measure the average gradient norm at cach layer, across minibatches of the training set. More for- mally, recall our notation for network layers as So ((head) ooo oo..o,0, 102 o //embed) Given a mini- batch = {(11, v1), .... (Ir, yo)} random samples from the training data, the stochastic gradient of layer é E (embed, 1, . L, head} is given by: ge(B) TET EE VoLlSonnenn (z:), vs). The norm of the stochastic gradient, ( |2, roughly measures how much layer will change after one step of SGD from pretrained initialization. We use the average gradient norm across all minibatches Bl...., B,, in the training set Drrain as a measure of movement in the first SGD step.
1 Σ (lge(B))l2, where E (embed, 1.2, L, head].
G
(3.1)
Note that measure all the gradients the pretrained initialization any gradient updates. In Figure 2 we plot the layer-wise gradient norms Gi on DomainNet. Similar plots for other datasets and with alternative normalization are provided in Appendix G
performing
First-layer gradient is an outlier. Apriori, we expect the gradients of the classifier "head" (green- triangles) to be large as they are randomly initialized while the rest of the model is pretrained (Kumar al. 2022). The interesting observation we see in Figure (see also Appendix G is that for the recent architectures (Vision Transformers and ConvNeXt), gradient norms of the "embedding" layers rred-suuares) stand out outliers with much larger gradients than the other layers. These also the models where see big gaps between Adam W and SGD in Tables 1 This suggests that the embedding layer plays distinctive role in the transfer learning from modern architectures.
We emphasize again that Figure shows gradient norms at pretrained initialization before any up- dates have been performed. Thus, the large gradients of the embedding layer vs AdamW fine-tuning. Rather, these trends hint at potential explanation for the differences be- tween them. Since SGD uses the same learning rate for all layers, we will end up make substantially larger updates to the embedding layer compared to other loyerr-leading to either over-tuning the embedding layer or under-tuning the remaining layers (which also includes the randomly initialized head). Over-training of the embedding layer is undesirable given the common wisdom that lower layers ought to be tuned less as they learn more transferable features (Kumar al. 2022). Adam on the other hand adaptively normalizes the movement of cach parameter.
the effect of SGD
How can we avoid the above issues with SGD? One possibility is to use different learning rates for different layers, but this might end up requiring extensive hyperparameter tuning. Another option is to use optimizers with layerwise normalization techniques like LARS (You et al. 2017a) and LAMB You al.) | (020). In our initial experiments (see Table while these methods improved over SGD, they did not close the gap with AdamW. Instead, we found much simpler modification to SGD that consistently leads to accuracies competitive with or better than AdamW while using lower memory.
5

---

Under review as a conference paper at ICLR 2023
3.3 FREEZE- EMBEDDIG
While the presence of large embedding layer gradients in Figure 2 correlates with the observed performance gaps between AdamW and SGD, it is not clear that this observation its potential issues discussed above are definitive causes for SGD performing worse than AdamW. To further test the intuition, in the next section, we consider a "freeze-embed" variation of fine-tuning. where we simply freeze the embedding layer to its pretrained initialization, and then fine-tune the rest of the model as usual. The embedding layer only consists of small fraction of the model parameters (e.g., 0.7% in a base vision transformer), so apriori freezing the embedding layer is very tiny tweak- and we would not expect a large change in accuracy. However, the hypothesis above holds merit, then we would expect the modification to aid SGD but not Adam
4 DETAILED EXPERIMENTS ON FREEZE-EMBEDDING
We consider a simple variation of SGD fine-tuning where we freeze the embedding layer and only perform SGD updates on the rest of the network-we call this method "SGD (freeze-embed)". In this section we discuss the results of fine-tuning our 7 models on 5 distribution shift benchmarks mentioned in Section We compare three methods: SGD, AdamW, and our proposed SGD (freeze- embed). We use the implementations of SGD and A in PyTorch. For each method, we train for the same number of epochs using a cosine learning rate schedule, and sweep over 6 starting learning rates (ensuring that the optimal learning rate is in the middle of the sweep). For all datasets we follow prior work (Kumar et al., 2022) and pick the best learning rate and early stop based on the ID validation accuracy. See Appendix B for additional details.
Table T and Table 2 show the OOD and ID accuracies, respectively, of the three methods across our models and datasets. For each model and dataset, we also highlight the relative gain/loss of AdamW and SGD (freeze-embed) from regular SGD in green/red. We discuss the main observations below.
1. A outperforms SGD. We see that Adam largely outperforms SGD, often by substantial margins on ViT and ConvNeXt models. Particularly in OOD evaluation, the gaps are remarkable. In the few instances where SGD performs better the gaps much smaller. The differences between SGD and Adam W are generally more modest for older ResNet models.
SGD (freeze-embed) competitive with or better than Adam W. For each individual model, averaged across the datasets SGD (freeze-embed) is consistently the best or tied-best method on OOD accuracy, and only minimally worse than A on ID accuracy (right columns of Table D2. Averaged across all the models and datasets, SGD (freeze-embed) performs the best OOD, getting an average OOD accuracy of 76.7% (vs. 71.9% for SGD and 76.0% for AdamW). On ID data, SGD (Freeze-embed) closes 85% of the gap between SGD and AdamW, getting an average accuracy of 91.3% (vs. 90.2% for SGD and 91.5% for AdamW).
Larger CLIP model gains more from SGD (freeze-embed) and AdamW. On a CLIP-ViT-B/16 we see that SGD (freeze-embed) and Adam W get about 8% higher OOD accuracy than SGD. Upon scaling
larger CLIP-ViT-L/14, which is also our best model for OOD performance, we see even higher gains over SGD, On LLIP-VIT-/11,, SGD (freeze-embed) gets 14.3% higher OOD accuracy than SGD. and 0.7% higher OOD accuracy than AdamW. This suggests that our findings might be even more relevant for larger models.
SGD (freeze-embed) is red when AdamW is red. Across all our models and datasets, we see that in instances where SGD (freeze-embed) is worse than SGD (i.e., highlighted as red), AdamW W is also worse than SGD. This is not always the case the other way around. For example, on VIT B-/16 fine-tuned on DomainNet and Camelyon, OOD performance of AdamW is worse than SGD, but SGD (freeze-embed) is competitive to the best of the two. At the same time, on Li OOD evaluation, multiple models have both AdamW and SGD (freeze-embed) perform significantly worse than SGD.
SGD performs well when fine-tuning data is closer to pretraining. Breaking down the results by datasets, the main trend we see is that with models that were pretrained on ImageNet-211 k (all models here except CLIP) and fine-tuned on Living-17) 17, the performance of SGD is typically higher than AdamW and SGD (freeze-embed). Images in Living-17 17 derived from ImageNet- IK and hence arguably, in this case the fine-tuning distribution is closest to pretraining. This suggests that SGD works better in instances where fine-tuning and pretraining data are similar.

---

Under review as a conference paper at ICLR 2023
6. Adam W (freeze-embedd does not improve significantly over AdamW. SGD (freeze-embed) was motivated by our hypothesis that AdamW would inherently avoid over-tuning of the embed- ding layer. If our hypothesis is true, unlike for SGD, freezing the embedding for AdamW would not be beneficial. In the Appendix, we expand the Tables 1 to include AdamW (freeze-embed) and indeed we see that freeze-embed does not provide complementary gains on top of AdamW.
Table I: Out-of- cutrof-distribution (OOD) accuracies for fine-tuning 7 pretrained models on 5 datascts. We compare fine-tuning with SGD, AdamW, and SGD (freeze-embed). For each model and dataset, we also highlight the relative gain or loss of AdamW and SGD (freeze-embed) from SGD, in green (for positive) and red (for negative). Lighter shades of green/red correspond to < 0.2 gain/loss. Key observations are: AdamW gets higher accuracies than SGD for modern ViT and ConvNeXt models. Averaged over the datascts (right column), SGD (eeeze--mbbed is consistently the best tied- best method for all 7 models.
Living-17
Waterbirds
DomainNet
FMoW
Camelyon
Avg.
CLIP VIT-B/16
SGD
80.0
62.5
72.8"
37,3
86.8
67.9
CLIP VIT-B/16
AdamW
82.8 (+2.8)
71.9 (+9.4)
89.2 (+16.4)
40.7 (43.4)
95.7 (48.9)
76.0 (+8.1)
CLIP VIT-B/16
SGD (trezze-embed)
83,2 (+3.2)
73.7 (+11.2)
88.2 (+15:4)
40,2 (+2.9)
94.3 (+7.5)
75,9 (+8.0)
CLIP VIT-//14
SGD
84,2
65.0
60.8
41.0
83.2
66.8
CLIP VIT-//14
AdamW
88.0 (+3.8)
85.2 (+:0.22)
93.8 (+:33.0)
48.3 (47.3)
95.9 (+12.7)
82.2 (+15.44
CLIP WTT-//14
SGD (frezzz-ombe))
90.5 (+6.3)
84.7 (+19.7)
93.1 (+32.33
49,9 (48.9)
96.5 (+13.3)
83.0 (+16.2)
Sup V/T-B/16
SGD
89.5
77.4
86.3
33,5
92.6
75.8
Sup VTT-B/16
AdamW
88.3 (-1.2)
81.6 (+4.2)
84.4
(-1.9)
35,9 (42.4)
87.9 (-4.7)
75.6 (-0.2)
Sup V/T-B/16
SGD (freoze-ombed)
88.0 (-1.5)
82.4 (+5.0)
86,3
(+0.0)
34.4 (40.9)
93.7 (+1.1)
77.0
(+1.2)
DINO ViT-B/16
SGD
88.2
56.1
76.0
33.6
86.9
68.2
DINO VIT-B/16
AdamW
87.4 (-0.8)
61.2 (+5.1)
77.4
(+1.4)
35,8 (+2.2)
91.9 (+5.0)
70.7 (+2.5)
DINO ViT-B/16
SGD (freezo-embed)
86.7 (-1.5)
67.9 (+111.8
78.4
(+2.4)
35.9 (42.3)
90.6 (43.7)
71.9
(+3.7)
ConsNext- Base
SGD
94.0
80.2
89.8
39.7
83.0
77.3
ConvNext-Bas
Adam'V
90.3 (-3.7)
89.8 (+9.6)
89.5
(-0.3)
38.4 (-1.3)
89.5 (46.5)
79.5
(+2.2)
ConsNext-
SGD (frezze-emeed)
92.6 (-1.4)
86.9 (+6.7)
91,2
(+1.4)
38.2 (-1.5)
88.1 (+5.1)
79.4
(+2.1)
BiT ResNet-50
SGD
84,3
76.5
80.0
34.1
90.4
73.1
BiT ResNet-50
AdamW
83.1 (-1.2)
74.8 (-1.7)
84.0 (+4.0)
33.8 (-0.3)
92.4 (42.0)
73.6 (+0.5)
BiT ResNet-50
SGD (freeze embed)
84.1 (.0.2)
75.5 (-1.0)
82,3
(+2.3)
35.0 (+0.9)
95,2 (+4,8)
74,4 (+1.3)
BiT ResNet-101 101
SGD
82.8
76.9
86.2
38.0
89.3
74.6
RIT ResNet-101
AdamW
82,9 (+0.1)
79.4 (+2.5)
83.5
(-2.7)
37.0 (-1.0)
89.7 (+0.4)
74.5
(-0.1)
BIT ResNet- 101
SGD (frezzz-ombed)
83.1 (+0.3)
77.3 (+0.4)
86.0
(-0.2)
36.0 (-2.0)
95.5 (+6.2)
75.6
(+1.0)
'SGD tuning DomainNet with CLIP unstable leading to high variance in the results. The variance for other dalasets small.
Table 2: In-distribution (ID) accuracies presented in the same format as the OOD accuracies in Table I We see similar high level trends: AdamW gets higher accuracies than SGD on modern vision models, but the gaps are smaller at higher accuracies; and SGD (freeze-embed) is consistently competitively with AdamW.
Living-17
Waterbirds
DomainNet
FMoW
Camelyon
Avg.
VIT-B/16
SGD
97.8
97.2
88.8t
67.0
99.4
CLIP VIT-B/16
AdamW
98.1 (+0.3)
97.7 (+0.5)
95.0 (+6.2)
70.1 (43.1)
99.5 (+0.1)
92.1
(+2.1)
CLIP VIT-B/16
(trezze-embed)
98.2 (40.4)
97.8 (40.6)
94.9 (+6.1)
70.0 (+3.0)
99.5 (+0.1)
92.1
(42.1)
CLIP VIT-L/14
SGD
98.4
97.3
84.31
69.0
99.4
89.7
CLIP VIT-L/14
AdamW
98.9 (+U.5)
98,8 (+1.5)
96.9 (+12.6)
74.5 (+5.5)
99.6 (+0.2)
93.7
(+4.0)
CLIP VIT-L/14
SGD (frezeeembed)
98.7 (40.3)
98.9 (+1.6)
97.1 (+12.8)
74.5 (+5.5)
99.6 (+0.2)
93.7
(+4.0)
Sup V/T-B/16
SGD
98.6
99.1
91.7
64.1
99.4
90.6
Sup VTT-B/16
AdamW
98.7 (+0.1)
99.0 (-0.1)
91.7
(-0.0)
66.4 (+2.3)
99.5 (+0.1)
91.1
(+0.5)
Sup VIT-B/16
SGD (frezzo-ombed)
98.7 (+0.1)
99.2 (40.1)
91.5
(-0.2)
65.0 (+0.9)
99.6 (++.22
90.8
(+0,2)
DINO ViT-B/16
SGD
98.4
97.0
88.2
62.4
99.4
89.1
DINO VIT-B/16
AdamW
98.5 (40.1)
97.9 (+0.9)
89.4 (+1.2)
66.0 (43.6)
99.6 (+0.2)
90.3
(+1.2)
DINO VIT-B/16
SGD (freeze-ombed)
98.4 (+0.0)
97.5 (+0.5)
89.0 (+0.8)
63.5 (+1.1)
99.5 (+0.1)
89.6
(+0.5)
Com Base
SGD
98.7
99.0
94.8
66.3
99.4
91.6
ConsNext-Bae Base
AdamW
98.6 (-0.1)
99.5 (+0.5)
94,5 (-0.3)
68.8 (42.5)
99.7 (+0.3)
92.2
(+0.6)
ComNNext-Bvee
SGD (frezze-emeed)
98.6 (.0.))
(40.4)
95.1
(+0.3)
67.4 (+1.1)
99.5 (+0.1)
(+0.4)
BiT RcsNet-50
SGD
97.4
98.4
89,3
64.6
99.5
89.8
BiT ResNet-50
AdamW
97.2 (-0.2)
98.5 (+0.1)
89.2
(.0.1)
63.1 (+0.5)
99.5 (+0.0)
89.9
(+0.0)
BiT ResNet-50
SGD (frezze-cmbed)
97.6 (40.2)
98.5 (40.1)
89.2
(40.1)
64.8 (+0.2)
99.5 (+0.0)
89.9
(40.1)
BIT ResNet-101
SGD
98.3
98.9
92.0
66.0
99.4
90.9
ResNet-
AdamW
98.4 (40.1)
98.6 (-0.3)
91.1
(-0.9)
67.0 (+1.0)
99.6 (+0022
90.9
(-0.0)
B/T 101
SGD (frezzo-ombed)
98.4
(+0.1)
98.8
(00.1)
91.5
(-0.5)
65.9 (40.1)
99.5 (+0.1)
90.8
(-0.1)
"SD tuning on DomainNet with CLIP unstable leading to high variance in the results. variance
datasets were small.
7

---

Under review as a conference paper at ICLR 2023
4.1 SOTA EXPERIMENTS AND RESULTS
Our experiments get new state-of-the-an art results for OOD accuracy on all 5 datasets. On 3/5 datasets (Living-17) 17, WILDS-FMoW, and WILDS-Camelyon) our proposed SGD (freeze-embed) does the best, while in other 2, Adam W has a small edge. Here, state-of-the-art means that the numbers we get are better than, to our knowledge. any reported number and all numbers on the official leaderboard, and are better than standard full fine-tuning with SGD. We show the best results from our paper in Table with a comparison to the previous tatte-of-theaar
Table 3: Our OOD accuracy results compared with the best reported numbers in prior work these datasets.
restrict to methods that do not use OOD data hyperparameter selection or early stopping. To our knowledge, the previous ofte-off-heeart results are from Wortsman et al. (2022) for FMo [Robey et al. (2021 for Camelyon, Kumar et al. (2022) for Living- 17 and the version of DomainNet et introduced by Tan et al. (2020)
and Ghosal et al. al. (2022) for Waterbirds. The WILDS numbers and references are taken from the official WILDS
leaderboard (as of 28 Sep 2022), and for Waterbirds consider all methods that do use group labels. For
Living-1's 17, we omit models pretrained with ImageNet- IK as Living- 17 is a subset of ImageNet-
Living-17
Waterbirds
DomainNet
FMoW
Camelyon
Best prior result
87.6
89.3
87.2
47.6
93.3
Best result from paper
90.5
89.8
93.8
49.9
96.5
SGD
SGD
SGD
Optimizer for best result
Adam
Adam
(frezze-embed) CLIPViT-L/14
(frezze-embed) CLIP ViT-L/14
(freeze-embed) CLIP VIT-L/14
Model for best result
ConvNeXt-B
CLIP VIT-L/14
SGD (freeze-embed) result
90.5
86.9
93.1
49.9
96.5
DETAILED ANALYSIS OF CLIP.
CLIP models have strong transfer learning performance and robustness amann our models, CLIP models had the best ID and OOD accuracies averaged across datasets. So we did more detailed analysis of the CLIP ViT-B/16 with other optimizer which are summarized in Table
Table 4: CLIP VYT-B/16 performance with new optimizers and confidence intervals. For addition SGD, Adam and SGD (frozze-embed), we provide 90% confidence intervals based on 3 runs of cach hyperparam- eter configuration. In addition, we show accuracies for two of our variant methods SGD (freeze-embed, no momentum) and
as well as comparison to two other optimizers, LARS You et al. 2017b and LAMB You et al. 2020 which use layer-wise normalization in their updates.
Living-17
Watcshidss
DomsiisNe
FMol
Canclyon
Arg:
ID
0OD
ID
00D
ID
00D
ID
0OD
ID
00D
ID
00D
SGD
97.8 (0.2)
(1.0))3))
(0.1
62.5 (5.0)
(7.1
(185
(7.0008)
37.3(1.1)
99.4 (0.0)
86.8 (1.1)
90.0
67.9
Adan'W
98.1 (0.1)
(1.2)
97.7 (0.00
71.9 (2.4)
(0.1
(1.1)
70.1 (0.2)
40.7 (0.3)
99.5 (0.I)
95.7 (0.4)
92.1
76.9
(frezz--mbbed)
SGD
98.2 (0.3)
83.2 (6.8)
(0.8
73.7 (1.1)
(0.9
(0.)
70.0 (0.2)
40.2 (0.7)
99.5 (0.0)
94.3 (0.3)
92.1
75.9
(freneeem, no mom) )
SGD
98.2
83.1
97.9
80.4
95.2
89.0
70.1
38.8
99.5
93.3
92.2
76.9
Guddul-anfiezzing
98.3
81.9
98.3
69.1
96.3
93.2
69.2
40.5
90.3
96.5
92.3
76.2
LAMB LARS
98.2
79.5
97.8
64.0
05.1
90.4
67.9
38.8
99.5
93.4
91.7
73.2
97.7
83.9
97.1
48.6
93.2
8388
67.0
38.6
99.3
93.3
90.9
69.6
SGD (freeze-embed, no momentum). Across our datasets and models, SGD (freeze-embed) per- formed competitively with AdamW, while using one less optimizer state/parameter and hence less memory. We can potentially gain additional memory savings by using a more basic SGD without momentum. Note that momentum has been shown to be important when training from scratch, but its benefit while fine-tuning has not been explored much. So for our SGD (freeze-embed, no momentum) method, all our datasets, we fine-tune CLIP ViT-B/16 model with SGD, freeze the embedding layer, and do not use momentum. In Table 4 we see that SGD (freeze-embed,, no momentum) is competitive with SGD (freeze-embed) and Adam W, even having edge on some
Generating Table involved over 600 fine-tuning runs, so we weren't able repeat this for every model.
8

---

Under review as a conference paper at ICLR 2023
datasets. This is promising, but since we only evaluated on CLIP ViT-B/16, more experiments needed to derive stronger conclusions about the accuracy of SGD (freeze-embed, no momentum).
GPU memory profiling. We profiled all methods on a Titan-X GPU, On ViT-B/16, AdamW uses 16% and 36% more memory than SGD (freeze-embed) and SGD (freeze-embed, no momentum) respectively. The gains are better for larger models: on a ViT-L/14, Adam W uses 18% and 48% more memory respectively, On ViT-H/14, AdamW W runs out of memory, while SGD (freeze-ember SGD (freeze-ember embed, no momentum) used 10.1 GB and 7.6 GB respectively. out of 12.2 GB. Note that the gains from SGD (freeze-embed) would be even higher with gradient checkpointing, which reduces activation memory. See more details in Appendix [
and
Gradual-unfrrezing The success of freeze-embed for SGD is inline with our hypothesis that it is beneficial to change the lower layers less during fine-tuning. We could further refine this idea- instead of just freezing the first "embedding" layer, we could start by freezing the entire pretrained model (except for the head) and gradually unfreeze the rest of the model over training. Strategies like this have been explored by ULMFiT (Howard & Ruder 2018) and others (Mukherjee & Awadallah, 2019 Romero et al. 2020). An advantage of we tune a smaller fraction of the model, which speeds up training- for example, for WILDS-FMoW on an A100 GPU, the average runtime decreases by over 30%.
is that in earlier stages of training
For gradual-unfreezing unfreezing, we found that naively using single learning rate throughout this process or even a cosine learning rate schedule is not sufficient to achieve good performance. Instead, we fit an exponential learning rate decay law for our experiments (see details in Appendix B). The performance of cradual-unfreezing gradual-unfreezing performed particularly well. For example, it got an OOD accuracy of 93.2% on DomainNet and 96.5% on Camelyon-which are both competitive with SGD (freeze-embed) and Adam W on a much larger CLIP ViT-L/14 model. On average, gradual-unfreezing gets slightly better results both ID and OOD, while being 30% faster. Overall though, we find that the most consistent improvements come from simply freezing the embedding layer, which suggests that the embedding layer plays a particularly prominent role in modern vision architectures.
CLIP ViT-B/16 is shown in Table 4 On some datasets,
SGD (freeze-embed) and A outperform LAMB and LARS. We also ran two alternative adaptive gradient methods, LARS (You 20176) and LAMB (You et al. 2020)- also sweeping over 6 learning rates and early stopping. These are alternate methods with layerwise normalization that can avoid over-training of large gradient layers. Moreover, like SGD and SGD (freeze-embed), LARS also has a lower memory footprint than AdamW. In Table 4 we see that while LARS and LAMB get higher accuracies than SGD, they do worse than SGD (ireeze-embed) and Adam both ID and OOD. In this case, our modification with freeze-embed appears to be more effective.
CIFAR-10 results. As a proof of concept for standard (non- OOD) transfer learning, we fine-tune CLIP ViT models on CIFAR-10. In Table 5 we see that even at high accuracies, AdamW and SGD (freeze-embed) manage to improve perfor- mance. SGD (freeze-embed) gets 20% and 30% lower error than SGD on CLIP ViT-B/16 and CLIP ViT-L/14, respectively.
Table 5: Test accuracy on CIFAR-10.
CLI viFii/t
CUP
SGD AdamW
98.0
99.0
SCD (freze-embed)
98.4
DISCUSSION.
We note that the methods we consider are not complex. We showed that a minor tweak of freezing the embedding layer overwhelmingly improves the performance across the board when fine-tuning with SGD. We clarify that we do not claim that SGD (freeze-embed) is a substantially better method than AdamW in terms of accuracy. Rather, it is remarkable that with its simplicity, we can already achieve comparable or even slightly better accuracy than AdamW across a wide range of models and benchmarks, all while retaining the memory and computational edge of SGD over A. The broader point of our work is that pretrained models have a lot of useful information, but naively fine-tuning these models can lead sub-optimal performance. Exploring simple properties of these models such as the gradients of different layers, and designing simple modifications such as freezing just small part of the model, can lead state-of-the-att accuracies.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta, Better fine-tuning by reducing representational collapse. International Conference on Learning Representations (ICLR), 2021.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYONI7 challenge. IEEE Trunsuctions on Medical Imaging, 38(2)5550-560, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference Computer Vision (ICCV), 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Computer Vision and Pattern Recognition (CVPR), 2018.
Tim Dettmees, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block- quantization. 9th International Conference on Learning Representations, ICLR, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In Computer Vision and Pattern Recognition (CVPR), 2017.
Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations? arXiv preprint NXXi:22008.02990 2022.
Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, Yang Zhang, and Jonathan M. Cohen. Training deep networks with stochastic gradient normalized by layerwise adaptive second moments. 2019.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: Transfer learning through adaptive fine-tuning. In Computer Vision and Pattern Recog- nition (CVPR), 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Computer Vision and Pattern Recognition (CVPR), 2016.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Association for Computational Linguistics (ACL), 2018
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https::////// If you use this software, please cite it below.
Ren.odo.51333
Haoming Jiang, Pengcheng He. Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. In International Conference on Learning Representations (ICLR), 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
10

---

Under review as a conference paper at ICLR 2023
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning (ICML), 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer better? In Computer Vision and Pultern Recognition (CVPR), 2019.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of distrbbbbiitt,.nn.. In International Conference on Learning Representations (ICLR), 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffiient prompt tuning, arXiv preprint arXiv:2104.0869) 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Computer Vision and Putlern Recognition (CVPR), 2022.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision, pp. 2200-2207, 2013.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. Distilling transformers into simple neural networks with unlabeled transfer data. arXiv preprint arXiv:1910.01700 1, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In International Conference on Computer Vision (ICCV), 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Interna- tional Conference on Machine Learning (ICML), volume 139, pp. 8748-8763, 2021.
Alexander Robey, George J. Pappas, and Hamed Hassani. Model-based domain generalization. In NeurIPS, 2021.
Miguel Romero, Yannet Interian, Timothy Solberg, and Gilmer Valdes. Targeted transfer learning to improve performance in small medical physics datasets. Medical Physics, 47(12):6246-6256, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations (ICLR), 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv, 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning (ICML), 2018.
11

---

Under review as a conference paper at ICLR 2023
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. ArXiv, abs/2106.10270 2021.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey, arXiv preprint TrX1:: 2020.
Hugo Touvron, Matthieu Cord, Maithijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.009900 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lpess Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022.
Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon. Deep gaussian process for crop yield prediction based remote sensing data. In Association for the Advancement of Artificial Intelligence (AAAI), 2017a.
Yang You, Igor Gitman, and Boris Ginsburg, Large batch training of convolutional networks. arXiv: Computer Vision and Pattern Recognition, 2017b.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations (ICLR), 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv, 2020.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: bascline for network adaptation via additive side networks. In European Conference Computer Vision (ECCV), 2020.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. FreeLB: Enhanced adversarial training for natural language understanding. In International Conference Learning Representations (ICLR), 2020.
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORKS
Many works transfer learning propose freezing parameters while fine-tuning to preserve pre- trained information. For example, linear probing, which freezes the entire model except the head (Kumar et al.] 2022] Wortsman et al. 2021) and zero-shot models (Radford et al., [021] have been shown improve OOD performance over standard fine-tuning. In NLP, methods such as prefix-tuning (Li & Liang. 202T) and prompt-tuning (Lester et al.] 2021) have been shown to im- prove OOD accuracy. Other works propose regularizing parameters towards initialization, freezing the first several layers, using different learning rates for different layers. or tuning different layers for different examples (Long et al. 2013 Ge & Yu 2017 Howard Ruder 2018 Guo al. 2019] Zhang et al. 2020 Zhu et al. 2020 Jiang et al. | Aghajanyan et al. 2021) Typically, large fraction the model is frozen to preserve the pretrained information- a key difference in this work is that we find that freezing very small fraction of the model (<1% of the parameters) can lead to
substantial and consistent improvements accuracy.
Other optimizers have been proposed to reduce AdamW's memory footprint, including LARS YYou et al., 2017b) and AdaFactor (Shazeer Stern 2018). Our method is simpler and achieves better accuracies with same or better memory gains. A complementary line of work Dettmers et al. (2022) study quantization mechanisms for optimizer states. These tools, although developed for Adam W, can also be used with SGD (freeze-embed) to get additional gains in memory.
B ADDITIONAL TRAINING DETAILS
SGD and A updates. We start with initialization ((0) (pretrain, Given hyperparam- eters, minibatch size batch_size and number of epochs numn_epochs, our algorithms run for we select a randomly shuffled minibatch B. from Derain (reshuffled at end of each epoch) and compute the minibatch stochastic gradient [B, EORDE Vol(Jo(z).v). The parameters ((0) are updated as follows.
T num peipeppohhsspaae steps. At steps = 0, ,,,,, T, batch size
For SGD, in addition to gradients gr and weights 000, we maintain first order momentum estimate my as optimizer state, initialized as m_1 0. The SGD(), (4, X) update with vi learning rate, momentum, and À weight decay, is given by
Jt It + (B(1)
pmm + St
(first moment)
((+11) g(0)
m
For AdamW, we maintain two additional optimizer states: the first moment estimate m, and second moment estimate ut, initialized as m-1 v-1 0. The AdamW(n 31, 32, X) update with n learning rate, (31, (82) betas, and À weight decay is given by
m Bim=-1 + 11)gt
(first moment)
Ut= B2 + (1 82)
(second moment)
m4
m. (1 - 81))
02 (1 1--
@(++1) = (1 1// 1)2))(())
n+
1)t
Vor +
Hyperparameter details
Learning rate. We sweep over 6 learning rates for SGD ([3e-5, le-4, 3e-4, le-3, 3e-3, le-2)) and for AdamW ([3e-7, le-6, 3e-6, le-5, 3e-5, le-4]). As standard, we use smaller learning rates for AdamW because they work better. We check that the best learning rate is in the middle of this grid for each model and dataset. The best learning rate is chosen based on ID validation accuracy.
Other optimizer parameters. We use all the default options for SGD and Adam W in pytorch except the SGD momentum parameter to 0.9. The other default options are: for
13

---

Under review as a conference paper at ICLR 2023
SGD, the weight decay is 0 and momentum without dampening, and for Adam W, the weight decay is 0.01, betas are (0.9,0.999) and epsilon is 1e-08.
Number of epochs. We train for 20 epochs on Living- 11, 20 epochs on Waterbirds, 50 epochs DomainNet, 5 epochs on WILDS-FMoW, and 3 epochs on Camelyon. We use the same number of training epochs for all the optimizer methods we report results on.
Learning rate schedule. With the exception of Gradual -usfreziing Table f we use cosine learning rate schedule and decay the starting learning rate to 0 over the course of T training steps. Learning rates schedule for cradual-unfreezzn
described below.
Learning rate schedule for gradual unfreezing. For gradual unfreezing, we do not use a cosine learning rate scheduler. Instead, at epoch t (0-indexed) out of T. we multiply the base learning rate by exp(3.73 (1.0 t/(T" learning rate by exp(3.73) ~ 41.7 in epoch 0. and by in the last epoch. The intuition is that when we are tuning a smaller number of layers we need a higher learning rate than for full fine-tuning- the optimal learning rate for full fine-tuning. The exact constant (3.73) was selected by comparing the optimal learning rate for head tuning with full fine-tuning (the optimal learning rate for head-tuning was approximately exp(3.73) larger for head- tuning). Without this decay schedule, for example using vanilla cosine learning rate scheduler, gradual unfreezing worked similarly or worse than vanilla full fine-tuning.
1))). This means we multiply the
example, the optimal learning rates for head tuning is higher than
Waterbirds
Stopping criteria. The results presented the paper are from models early stopped based ID validation accuracy. We sanity checked that the conclusions similar if we the last epoch checkpoint.
Data augmentation and input preprocessing. and augmentations on our input images. We use very basic augmentations (e.g., only horizontal flips for WILDS, and standard augmentations from past work), including for our state-of-the-ar results:
Additionally, we use the following preprocessing
1. For WILDS-FMoW, the images are 224 x 224, so we do not resize, and only perform a random horizontal flip augmentation. We do not perform any augmentations test time.
2. For WILDS- Camelyon, we resize the images to 224 224 with bilinear interpolation (standard in pyorch), and only perform a random horizontal flip augmentation. We do not perform any augmentations test time, just the resize.
3. For Living- 17, we follow (Kumar al. 2022) and perform a RandomResizedCrron to 224 x 224 sized images (using the default options in pyorch), and then a random horizontal flip augmentation while training. At test-time resize the image 256 256 and then take a centercrop of size 224 x 224.
4. For DomainNet, we follow (Kumar et al. 2022) and first resize the image to 256 256 with bicubic interpolation, then take RandomCrop of size 224 224 (using the default options in pyorch), and then random horizontal flip augmentation while training. At test-time we simply resize the image 224 224 with bicubic interpolation.
5. For Waterbirds, we resize the image 224 224 with bicubic interpolation and then take a centercrop of size 224 x 224. We apply the same transformation at test time.
Embedding layer. For SGD (freeze-embed), the exact layers we freeze are follows:
1. CLIP ViTs:
We freeze the patch-to-token embedding layer and layernorm.
2. Supervised and DINO ViTs: We freeze the patch-to-token embedding layer (there is no layernorm after this)
3. BiT-ResNets: We freeze the 'stem' and the first convolution block of the model. We tried freezing less and more of the model in our initial experiments, but it did not seem to help.
4. Con-NeXt-B: We freeze the 'stem' and the first stage of the model.
14

---

Under review as a conference paper at ICLR 2023
C ABLATIONS ON CLIP
In Tables wc show additional ablations for the CLIP ViT-B/16 model on all datasets
f We tried:
1. SGD (freeze-ember not layer-norm); For the CLIP model our freeze-embed variation freezes the bottom embedding layer along with the layernorm right after that. We ran an ablation where we only freeze the bottom linear embedding layer, but not the layer norm. This performs comparably with SGD (freeze-embed), which suggests that freezing the input layer is what's important, and the layer norm does not matter much.
SGD (no momentum): Since SGD (freeze-embed, no momentum) performed very well in our experiments, also tried fine-tuning with full SGD (no freezing), but without momentum. We found that SGD (no momentum) and SGD perform comparable.
SGD (weight decay): Vanilla SGD is done without weight decay, but AdamW incorporates weight decay. We ran this ablation to confirm that the gains of AdamW are not because of weight decay. We used the torch SGD optimizer, and the weight decay argument 0.01. Indeed, we found that SGD and SGD (weight decay) perform comparably, which suggests that decght_decay is not the reason for the improved performance of AdamW.
4. Linear probing: We freeze the pretrained model, and only train linear probe the features of the CLIP ViT-B/16 model. We train a logistic regression classifier using the sklearn library, sweeping over 50 regularization values in np.logssacee-7.22
Table 6: -ft-of-distribution (OOD) accuracy of more optimizers on CLIP ViT-B/16. We find that weight decay, momentum, and unfreczing the layer norm at the bottom of the model do not make much of a difference.
Living-17
Waterbirds
DomainNet
FMoW
Camelyon
Avg.
SGD Adam
80.0
62.5
72.8
37.3
86.8
67.9
82.8
40.7
95.7
76.0
Our methods
SGD ((rezze-mbedd)
83.2
73.7
88.2
40.2
94.3
75.9
SGD (freeze-embed, no momentum)
83.1
80.4
89.0
38.8
93.3
76,9
Gradual-unfreezze
81.9
69.1
93.2
40.5
96.5
76.2
Other layerwise normalization methods
LAMB
79.5
90.4
38.8
93.4
73.2
LARS
83,9
48.6
83.8
38.6
93.3
69.6
Variations SGD (frezz-embed)
SGD (freeze-embed, not layer-norm)
83.6
74.3
89.1
39.3
92.9
75.9
Other freezing: Linear probing
86.2
60.4
89.1
29.0
92.6
71.5
Variations of SGD without any freezing
SGD (no momentum)
81.4
76.7
37.9
84.3
67.9
SGD (weight decay)
83,9
65.1
67.5
37.1
85.6
67.8
D PROFILING GPU MEMORY
We profiled the GPU memory consumption of 4 fine-tuning methods, on 3 models on a Titan-X GPU with 12.2 GB of GPU memory. The profiling was done using Weights and Biases and measures the total GPU memory utilization (so includes weights, activations, etc.). The four methods we profiled are: SGD, AdamW, SGD (freeze-embed), SGD (freeze-embed, no momentum). We profiled these on three models: CLIP-ViT B/16, CLIP ViT-L/14, and OpenCLIP ViT-H/14 the original CLIP model only scales up to ViT-L/14, so we used the OpenCLIP (Ihaacco et al. 2021 for ViT-H/14. We used a batch size of 32 and do gradient accumulation with micro-batcl size of Ti..e. we calculate the gradients one sample at a time and accumulate them before calling the optimizer update.
We show results below in Table 8 On ViT-B/16, AdamW uses 16% and 36% more memory than SGD (freeze-embed) and SGD (freeze-embed, no momentum), respectively. The gains are better
Running each ablation for all models all datascts too computationally expensive.
15

---

Under review as a conference paper at ICLR 2023
Table 7: In-distribution (ID) accuracy of more optimizers CLIP ViT-B/16. We find that weight decay, momentum, and unfreezing the layer norm at the bottom of the model not make much a difference.
Living 17
Waterhirds
DomainNet
FMoW
Camelyon
Avg.
SGD AdamW
97.8
97.2 97.7
88.8
67.0
99.4
90.0
98.1
95.0
70.1
99.5
92.1
Our methods
SGD (frecze-embed)
98.2
97.8
94.9
70.0
99.5
92.1
SGD (frezze-embed, no momentum)
98.2
97.9
95.2
70.1
99.5
92.2
Gradual unfreezing
98.3
98.3
96.3
69.2
99.3
92.3
Other layerwise normalization methods
LAMB LARS
98.2
97.8
95.1
67.9
99.5
91.7
97.7
97.1
93.2
67.0
99.3
90.9
Variations of SGD (freeze-emeed)
SGD (freeze-embed, not layer-norm)
98.0
98.0
95.4
70.2
99.5
92.2
Other freezing: Lineur probing
97.8
96.6
94.5
47.2
96.1
86.4
Voriations SGD without any freezing
SGD (no momentum)
98.0
97.1
89.5
66.4
99.3
90.1
SGDD (weight decay)
97.6
97.2
87.9
66.4
99.3
89.7
Table 8: We profile the GPU memory consumption of four optimization methods on three CLIP models of varying sizes, on Titan-X GPU. SGD ffreeze-embed gives practical gains over AdamW, especially if we drop the momentum parameter. The largest CLIP ViT-H/14 does not fit in GPU memory when using AdamW, but fits in memory with other optimizers. Note that the freeze-embed methods perform competitively better than Adam W on accuracy, as shown Table
CLIP ViT-B/16
CLIP ViT-L/I4
CLIP ViT-I1/14
AdamW SGD
2.7
7.1 GB
Out-of-Memory
2.3 GB
6.1 GB
10.1 GB
SGD (freeze-embed) SGD (freeze-embed, no momentum)
2.3 GB
6.1 GB
10.1 GB
2.0 GB
4.8 GB
7.6 GB
for larger models: ViT-L/14, AdamW uses 18% and 48% more memory respectively. On ViT-H/14, AdamW runs out of memory, while SGD (freeze-embed) and SGD (freeze-embed, no momentum) are able to run, showing that the gains are at least 20% and 60% respectively.
For large models, is common to use additional tricks like gradient checkpointing to reduce the activation memory. That is, for speed penalty (at most 2x), only need to store activations in VL of L layers when doing backpropagation. Gradient checkpointing would further increase our gains over AdamW since they do not change the memory consumed by the weights but can substantially decrease the memory consumed by the model's activations.
E RESULTS FOR ADAMW (FREEZE-EMBEL)
In our paper, we hypothesized that AdamW and SGD (freeze-embed) improve on SGD for the same reason-they change the embedding layer less. Based on this hypothesis, we would expect that the gains of AdamW and freeze-embed are not complementary. Indeed, we find that the AdamW (freeze-embedl variation performs similarly to AdamW and SGD (freeze-embed).
Tables 9 & [0]] expand on the OOD and ID accuracy results in Tables & [T respectively, in- clude Adam W (frecze-embed). Overall, AdamW (freeze-embed) and SGD (freeze-embed) perform comparably for all the recent vision models, both fairly close Adam W allthough there are differ- ences in some models and datasets. Averaged across all the datasets and models, AdamW, Adam W (freeze-embed), and SGD (freeze-embed) get 76%, 76.5%, and 76.7% accuracy, respectively, com- pared to 72.0% for SGD, Averaged across all the in-distribution datasets and models, Adam W, AdamW W (freeze-embed), and SGD tively, compared to 90.3% for SGD. Overall the performances of the three models are similar enough to suggest that these methods work well for the same reason that they tune the embedding layers less, and that freeze-embed is not independent axis of improvement.
get 91.5%, 11.5% and 91.3% accuracy, respec-
16

---

Under review as a conference paper at ICLR 2023
Table 9: Out-of-distribution (OOD) accuracies with Adam (freeze-embed), This is expansion of Table ! to include AdamW (frezze-embed) OOD results for fine-tuning 7 popular models across benchmark datasets. On OOD performance averaged across all models and datasets, AdamW (freeze-embed) gets slightly better accuracy than AdamW but slightly worse than SGD (frezze-embed).
Living-17
Watcrbirds
DomainNet
FMoW
Camelyon
Avg.
CLIP VTT-B/16
SGD
80.0
62.5
72.8
37.3
86.8
67.9
CLIP VTT-B/16
Adam W
82.8 (+2.8)
71.9 (49.4)
89,2 (+16.4)
40.7 (43.4)
95.7 (48.9)
76.0 (48.1)
CLIP VTT-B/16
SGD (frezze-mmedd)
83.2 (+3.2)
73.7 (=11.2)
88.2 (+15.4)
40.2 (+2.9)
94.3 (+7.5)
75.9 (+8.0)
CLIP VTT-B/66
AdamW (freeze-emhed)
82.4 (42.4)
69.5 (47.0)
88,2 (+15,4)
40.7 (43.4)
96.7 (49.9)
75.5 (47.6)
CLIP VIT-L/14
SGD
84,2
65.0
60.8
41.0
83.2
66.8
CLIP VIT-L/14
AdamW
88.0 (+3.8)
85.2 (+20.2)
93.8 (+33.0)
48.3 (+7.3)
95.9 (+12.7)
82.2 (+15.4)
CLIP WTT-L/14
SGD (frezz-embeed)
90.5 (+6.3)
84.7 (+19.7)
93.1 (+32.3)
49.9 (+8.9)
96,5 (+13,3)
83.0 (-16.2)
CLIP VIT-L/14
AdamW (frezzo-embed)
88.0 (43.8)
84.6 (+19.6)
93.8 (+33.0)
44.6 (+3.6)
96.4 (+13.2)
81.5 (+14.7)
Sup VIT-B/16
SGD
89.5
77.4
86.3
33.5
92.6
75.8
Sup V/T-B/16
AdamW
88.3 (-1.2)
81.6 (+4.2)
84,4 (-1.9)
35,9 (+2.4)
87.9 (-4.7)
75.6 (-0.2)
Sup VIT-R/16
SGD (frezze-embed)
88.0 (-1.5)
82.4 (45.0)
86.3 (+0.0)
34.4 (+0.9)
93.7 (+1.1)
77.0 (+1.2)
Sup VIT-B/16
Adam (freezo-cmbed)
88.1 (-1.4)
82.4 (+5.0)
82.3 (-4.0)
35.7 (+2.2)
93.0 (+0.4)
76.3 (40.5)
DINO ViT-B/16
SGD
88.2
56.1
76.0
33.6
86.9
68.2
DINO VIT-B/16
AdamW
87.4 (-0.8)
61.2 (+5.1)
77.4 (+1.4)
35.8 (+2.2)
91.9 (+5.0)
70.7 (+2.5)
DINO VIT-B/16
SGD ffrezze-mbbdd
86.7 (-1.5)
67.9 (+11.8)
78.4 (+2.4)
35,9 (+2.3)
90.6 (43.7)
71,9 (43.7)
DINO VIT-B/16
AdamW (freeze- embhd)
86.8 (-1.4)
64.5 (+8.4)
76.6 (+0.6)
35.4 (+ (+1.8)
93.5 (+6.6)
71.4 (+3.2)
Con-Next-Bal -Bsc
SGD
94.0
80.2
89.8
39.7
83.0
77.3
ConvNext-Bose
Adam
90.3 (-3.7)
89.8 (49.6)
89,5 (-0.3)
38.4 (-1.3)
89.5 (46.5)
79.5 (+2.2)
Con-Next-Base
SGD (freeze-embed)
92.6 (-1.4)
86.9 (+6.7)
91.2 (+1.4)
38.2 (-1.5)
88.1 (+5.1)
79.4 (+2.1)
ConyNext- Base
AdamW (freeze-embed)
88.1 (-5.9)
91.6 (.4)
89.8 (+0.0)
41,6 (+1.9)
87.7 (+4.7)
79.8 (42.5)
BiT ResNet-50
SGD
84.3
76.5
80.0
34.1
90.4
73.1
BiT RcsNet-50
AdamW
83.1 (-1.2)
74.8 (-1.7)
84.0 (+4.0)
33.8 (-0.3)
92.4 (42.0)
73.6 (40.5)
BiT ResNet-50
SGD (freeze-
84.1 (-0.2)
75.5 (-1.0)
82.3 (+2.3)
35.0 (+0.9)
95.2 (+4.8)
74.4 (+1.3)
BiT ResNet-50
AdamW (freezo-embed)
82.9 (-1.4)
77.3 (40.8)
83.3 (+3.3)
36.3 (42.2)
93.7 (43.3)
74.7 (+1.6)
BIT ResNet-101
SGD
82.8
76.9
86.2
38,0
89.3
74.6
ResNet-101 101
AdamW
82.9 (+0.1)
(+2.5)
83.5 (:2.7)
37.0 (-1.0)
(+0.4)
74.5 ((0.))
BIT ResNet- 101
SGD (frezzo-embed)
83.1 (40.3)
77.3 (40.4)
86.0 (-0.2)
36.0 (-2.0)
95.5 (46.2)
75.6 (+1.0)
ResNet-101
AdamW (freeze- embed)
84.8 (+2.0)
78.3 (+1.4)
84.3 (-1.9)
38.2 (+0.2)
(+5.7)
76.1 (+1.5)
F WHAT COULD CAUSE LARGE "EMBEDDING" LAYER GRADIENTS?
Generally speaking, we find that Adam fine-tuning leads to better performing and more robust models than SGD fine-tuning, specially in modern architectures. Our algorithms to close this gap were inspired by the observation that on modern vision architectures, the embedding layers have substantially larger gradients at pretrained initialization compared to other layers. This could lead to over-training of the embedding layer when using SGD, which we hypothesized would be bad for robust fine-tuning. The success of SGD (frezze-embed) in our experiments adds further evidence to our hypothesis. But, why are the gradients of embedding layers at pretrained initialization high in the first place? More generally, why does AdamW do better than SGD during tine-tuning? We do not have definitive answers to these questions as we are far from understanding the precise dynamics optimization algorithms on large neural networks. Below we discuss some plausible reasons.
Algorithmic aspects: pretraining algorithm? Among our 7 models, the more recent models like vision transformers and ConvNeXt, which were pretrained with AdamW are also the ones with largest gaps in performance between AdamW and SGD fine-tuning. BiT ResNets that were pre- trained with SGD had much smaller differences between AdamW and SGD. This strongly suggests that the discrepancy between the algorithms in pretraining and fine-tuning might be cause for the performance gaps. For example, it is possible that pretraining with AdamW implicitly biases to- wards configurations that most benefit from AdamW updates. This would also explain why other adaptive algorithms like LARS and LAMB are not competitive even though they perform some form of layer-wire wise normalization. On the other hand does not explain why such effects would distinc- tively impact the "embedding" layer and not the other layers. In a related manner, it is also unclear why SGD (freeze-embed) would be able to overcome such implicit biases from Adam W pretraining-
Architectural aspects: "patchifying" first layer? There are substantial architectural differences between the newer transformer and ConvNeXt models and the older ResNets which could also contribute to the newer models working better with AdamW. Most notably, vision transformer is
17

---

Under review as a conference paper at ICLR 2023
Table 10: In-distribution (ID) accuracies with AdamW (freeze-embed). This is an expansion of Table 2 to include AdamW (freeze-embed) ID results for our 7 models and datasets, AdamW, AdamW (freeze-embed), and SGD (freeze-embed) all perform comparably on ID accuracies.
Living 17
Waterhirds
DomainNet
FMoW
Camelyon
Avg.
CLIP VIT-B/16
SGD
97.8
97.2
88.8
67.0
99.4
90.0
CLIP VYT-B/16
Adam
98.1 (+0.3)
97.7 (+0.5)
95.0 (+6.2)
70.1 (+3.1)
99.5 (+0.1)
92.1 (+2.1)
CLIP B/16
SGD (freeze embed)
98.2 (+0.4)
97.8 (+0.6)
94.9 (+6.1)
70.0 (+3.0)
99.5 (+0.1)
92.1 (+2.1)
CLIP V/T
Adam (fresre-emeed)
98.3 (+0.5)
97.8 (+0.6)
95.3 (+6.5)
70.5 (+3.5)
99.6 (+0.2)
92.3 (+2.3)
CLIP VIT-L/14
SGD Adam
98.4
97.3
84.3
69.0
99.4
89.7
CLIP VIT-L/144
98.9 (+0.5)
98.8 (+1.5)
96.9 (+12.6)
74.5 (+5.5)
99,6 (+0.2)
93.7 (+4.0)
CLIP VIT-L/14
SGD (freeze embed)
98.7 (+0.3)
98.9 (+1.6
97.1 (+12.8)
74.5 (+5.5)
99.6 (+0.2)
93.7 (+4.0)
CLII VIT-1/14
Adum (frezze-emhen
98.7 (+0.3)
98.8 (+1.5)
96.7 (+12.4)
75,1 (+6.1)
99.6 (+0.2)
93.8 (+4.1)
Sup VIT-B/16
SGD
98.6
99.1
91.7
64.1
99,4
90.6
Sup VIT-B/16
Adam
98.7 (+0.1)
99.0 (-0.1)
91.7 (-0.0)
66.4 (+2.3)
99.5 (+0.1)
91.1 (+0.5)
Sup VTT-B/16
SGD (freeze embed)
98.7 (+0.1)
99.2 (+0.1)
91.5 (-0.2)
65.0 (+0.9)
99.6 (+0.2)
90.8 (+0.2)
Sup VTT-R/16
Adam (frezre-embed)
98.6 (+0.0)
99.0 (10.1)
90.9 (-0.8)
66.8 (+2.7)
99.5 (+0.1)
91.0 (+0.4)
DINO ViT B/16
SGD AdamW
98.4
97.0
88.2
62.4
99.4
89.1
DINO VIT-B/16
98.5 (+0.1)
97.9 (+0.9)
89 19.4 (+1.2)
66.0 (+3.6)
99.6 (+0.2)
90.3 (+1.2)
DINO ViT-B/16
SGD (freezo-embed)
98.4 (+0.0)
97.5 (+0.5)
89.0 (+0.8)
63.5 (+1.1)
99,5 (+0.1)
89.6 (+0.5)
DINO VII-B/16
Adam (freeze-embed)
98.4 (+0.0)
97.8 (+0.8)
89.7 (+1.5)
65.7 (+3.3)
99.6 (+0.2)
90.3 (+1.2)
Con-Next- Base
SGD
98.7
99.0
94.8
66.3
99.4
91.6
ConyNext-
AdamW
98.6 (40.1)
99,5 (+0.5)
94.5 (-0.3)
68.8 (+2.5)
99.7 (+0.3)
92.2 (+0.6)
ConyNext-Bast
SGD (froczo-embed)
98.6 (-0.1)
99.4 (+0.4)
95.1 (+0.3)
67.4 (+1.1)
99.5 (+0.1)
92.0 (+0.4)
Conv Nexi-Base
Adam (frezze-embed)
98.7 (+0.0)
99.5 (+0.5)
94.7
(-0.1)
69.2 (+2.9)
99.6 (+0.2)
92.4 (+0.8)
BIT R5
SGD
97.4
98.4
89.3
64.6
99.5
89.8
BIT ResNet-50
AdamW
97.2 (-0.2)
98.5 (+0.1)
89.2 (-0.1)
65.1 (+0.5)
99.5 (+0.0)
89.9 (+0.1)
BiT ResNet-50
SGD (frezze-embed)
97.6 (+0.2)
98.5 (+0.1)
89.2
(-0.1)
64.8 (+0.2)
99.5 (+0.0)
89.9 (+0.1)
BiT ResNet-50
Adam ((rezze-embed)
97.4 (+0.0)
98.4 (-0.0)
89.1 (-0.2)
64.9 (+0.3)
99.6 (+0.1)
89.9 (+0.1)
BIT ResNet-101
SGD
98.3
98,9
92.0
66.0
99.4
90.9
ResNet-101
Adam
98.4 (+0.1)
98.6 (-0.3)
(-0.9)
(+1.0)
99.6 (+0.2)
90.9 (-0.0)
BiT ResNet-101 101
SGD (freeze embed))
98.4 (+0.1)
98.8 (:0.1)
91.5 (-0.5)
65.9 (:0.1)
99.5 (+0.1)
90.8 (:0.1)
BiT ResNet-101
AdamW (frezze-embed)
98.2 (-0.1)
98.7 (-0.2)
91.1 (-0.9)
66.6 (40.6)
99.6 (+0.2)
90.9 (-0.0)
fundamentally different architecture from convolutional networks. At the same time, the biggest differences between the architectures- self-attention, softmax non-linearity, or fully connected layers-are not the likely contributors to the differences between AdamW and SGD in fine-tuning. This is because, we also see gaps between these methods with the CenvNeXt models which lack the major transformer components. Rather, it is the possible that some of the designs that were adapted from transformers to ConvNeXt contributes to the differences between Adam and SGD fine- tuning. Among these, many primarily affect the higher layers such as heavy third stage, depthwise convolution, inverted bottleneck, and larger kernels, and are unlikely to cause the lower layer gradi- ents to be high. The key design change we believe is important here is the use of "patchify stem" that could cause distinctive changes in the gradients for lower blocks.
The "stem" layer primarily controls how the input is processed for the rest of the network. ViTs and ConvNext down-samples the input images from -veeraaalap patch tokens, compared to ResNets that use overlapped convolution followed by max- maxx-pool. The coarser non-overlap might lead the embedding layer to attune more closely to the patch distribution in pretraining. This might not be an issue by itself as pixel/patch level information is noisy anyways and the higher layers can extract more robust features from across the patches. A possible dynamics in fine-tuning is as follows: On new datasets where the patch distributions are very different from pretraining, the embedding layers might have large gradients. In standard SGD, this might cause the embedding layer moving too quickly to fit the new patches before the higher layers can adapt to the changes. Instead, freezing the embedding layer would pass along the raw patches with minimal processing and lets the model adapt to the new distribution based on higher level features, which are likely more robust.
Summary. In conclusion, we hypothesize that the differences between AdamW and SGD fine-tuning arise from combination of the above reasons- with AdamW and large "patchify" embedding layer. Additionally, the use of GeLU activation and layer normalization might also change the optimizer dynamics although these are minor changes from ReLU activa- tion and group normalization used in BiT ResNets. is of interest to systematically explore these reasons further in future work.
18

---

Under review as a conference paper at ICLR 2023
G ADDITIONAL PLOTS ON GRADIENT NORMS AT THE PRETRAINED INITIALIZATION
In Figure 3 show plots for the gradients norms different layers for Living-17 and Waterbirds. These are analogous to Figure 2 for DomainNet in the main paper. We again see that, among the pretrained layers, the embedding layer has much higher gradients in the modern architectures.
We also consider two ways of normalizing the gradient magnitude. In the first method, we divide the of the gradient by the norm of the parameters in order capture the relative "movement" in the first SGD step as opposed to the absolute "movement" which is captured by gradient norm itself. In the second method, we divide the norm of the gradient by the square root of the number of parameters. This is to check that a layer does not simply have larger gradient because it has more parameters. The reason we use square root as follows: suppose each parameter has gradient z e, then the layerwise gradient norm scales with the square root of the number of parameters. Also, the first step of Adam update essentially a signed gradient descent step, wherein we ignore weight decay, the per-layer "movement" is the square root of the number of parameters. So this normalization can be thought of as relative size of SGD update compared to Adam W in each layer initialization. For visualization purposes, they often much larger than the others so the plots become hard to see if we include the 'head' layer. Note that expect the head layer to have higher gradients because is randomly initialized (Kumar et al. (022).. For VIT models, we omit gradients of the cls token, position embedding, and layer norm after the embedding layer.
exclude the 'head' layer gradient in these plots as
middle layers
head layer embedding layer
15
10
5
clip_b b
clip_1
vit_b
dino_b
convnext_p
resne:50
resnet101
(a) Layer-wise gradient norms on Living-17 17 pretrained initialization
60
middle layers
head layer
embedding layer
40
10
clip.b
clip_1
vit_b
dino_b
convnext_
resner50
resnet101
(b) Layer- wise gradient norms on Waterbirds at pretrained initialization
Figure 3: We visualize the layer-wise gradient norms our models (a) Living-17 17 and (b) Waterbirds, at the pretrained initialization. The format is the same Figure 2 gradient norms of "embedding". "head", and "middle" layers shown red-squares, green-triangles and black-circles, respectively.
19

---

Under review as a conference paper at ICLR 2023
middle layers
2.0
embedding layer
1.5
1.0
0.0
clip
clip_l
vit
dino_h
convnext
resine50p
resnet101
(a) Layer-wise gradient norms divided by parameter norm, on DomainNet al pretrained initialization
middle layers
embedding layer
2.0
1.0
0.5
0.0
clip
clip
vit.
dino__
convnext
resine500
resnet101
(b) Layer-wise gradient norms divided by parameter norm, on Living- at pretrained initialization
middle layers
embedding layer
2.0
1.5
1.0
0.5
0.0
clip_b
clip_]
vit
dino_b
convnext_b
resnet50
resnet101
(c) Layer-wise gradient norms divided by parameter norm, on Waterbirds at pretrained initialization
Figure 4: We visualize the layer-wise gradient norm, divided by the norm of the parameters on (a) Domain- Net, (b) Living-17, 17, and (c) Waterbirds, the pretrained initialization. For better visualization, we omit the head from the plot, which predictably has much larger gradients than the others (since it randomly initial- ized). The format is the same as Figure [ gradient norms of "embedding" and "middle" layers are shown as red-square, and black-circles. respectively. Under this normalization scheme, the embedding layer has higher gradicnts than the other layers in all models. However, the gradient is only slightly larger for ResNet models, and substantially larger for the Vision Transformer models-whiie also provides support for why freezing the embedding layer Vision Transformers might make a larger difference.
20

---

Under review as a conference paper at ICLR 2023
middle layers
0.030
embedding layer
0.025
0.020
0 0.015
0.000
clip_b
clip_l
vit_b
dinou_
convnext_
resnet50
resnet101
(a) Layer-wise gradient norms divided by /Paarmeterss DomainNet at pretrained initialization
0.035
middle layers
embedding layer
0.030
0.025
0.020
0.015
0.005
0.000
clip_1 b
clip_l
vit b
dino
convnext
resnet50
resnet101
(b) Layer-wise gradient norms divided by v /#paramcters, on Living-17 17 at pretrained initialization
middle layers
embedding layer
0.04
0.03
0.02
0.01
dip_b
vit
dino_b
convnext_s
b
resnet50
(c) Layer-wise gradient norms divided by / ##parameters, on Waterbirds pretrained initialization
Figure 5: We visualize the layer-wise gradient norm, divided by the square root of the number of parameters on (a) DomainNet, (h) Living- 17, and (c) Waterbirds, at the pretrained initialization. For better visualization, omit the head from the plot which has predictably much larger than the others (since it randomly initialized). "middle" layers are shown as red-
format the same as Figure 2 gradient norms of "embedding" squares and black-cireles, respectively. Under this normalization, we see that the gradients of the embedding layer are much larger than the other layers in all models, including ResNets.
21


Reviews:
[{"review_id": "tbJ3F1LBOfe", "reviewer": "Reviewer_Q4kq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to simply freeze the embedding layer (0.7% of the parameters) of the large neural networks in computer vision, then apply the standard SGD optimizer to finetune the parameters on the out-of-distribution datasets. It is shown that such a simple setting contributes to competitive performance with the popular AdamW. The experimental results on five benchmark datasets further support the effectiveness of the proposed method.", "summary_of_the_review": "This paper contains some interesting observations, however, the novelty may not meet the standard of the main track.", "strength_and_weaknesses": "# Strength:\n- The technique part of this submission is quite simple and easy to follow, which may help to reproduce the results reported in the paper.\n- The motivation for freezing the first (“embedding”) layer has been clearly illustrated in Figure 2. It is interesting to see that ResNets are completely opposite to Vision Transformers and ConvNeXt. \n- The authors conduct thorough ablation studies on several datasets to verify the effectiveness of SGD Freeze-Embed, which should be encouraged.\n\n# Weaknesses:\n- ### Major Concerns:\n  - According to the **ABSTRACT**, \"when the two methods perform the same, SGD is preferable because it uses less memory and is more efficient than AdamW\", however, I failed to find any comparisons on the training cost or the memory footprint between SGD Freeze-Embed and AdamW. Since activations commonly consume far more memory resources than weights, I doubt that the real memory saving could be very limited. The authors are encouraged to include more quantitative analyses of the efficiency of the proposed methods.\n  - It would be better to evaluate SGD Freeze-Embed on more downstream tasks. The current manuscript mainly focuses on the out-of-distribution (OOD) setting, which seems to weaken the importance of this discovery. I am curious to know whether the phenomenon illustrated in Figure 2 widely exists in different tasks. If it is true, any theoretical or empirical analysis may shed some light on the further understanding of why vision transformer is more difficult to train and the design of Transformer-specific optimizers.\n  - Though the authors present extensive experiments on OOD tasks, the reason why large gradients damage performance remains unsolved. One may easily clamp the gradients of the first layer with `clip_grad_norm_` to fix the outlier. Hence, I am not fully convinced that SGD Freeze-Embed should be the best choice.\n- ### Minor Issues:\n  - In the original ConvNeXt paper, they \"replace the ResNet-style stem cell with a patchify layer implemented using a 4×4, stride 4 convolutional layer\". I am not sure whether the \"patchify layer\" refers to the embedding layer. Could the authors further explain the architecture of the first layer in ConvNeXt?\n  - What is the difference between SGD and SGD (Freeze-embed) when applied to ResNets, listed in Tables 1-2?  \n  - In **Freezing layers**: I politely disagree that \"The embedding layer only consists of a small fraction of the model parameters (e.g., 0.7% in a base-vision transformer), so a priori freezing the embedding layer is a very tiny tweak—and we would not expect a large change in accuracy.\" Note that a group of \"Visual Prompt Tuning\" methods aim at introducing/fine-tuning only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the other model backbone frozen [1,2,3].\n  - Did SGD Freeze-Embed still apply weight decay to the first layers?\n\n# Reference:\n- [1] Visual Prompt Tuning. ECCV2022\n- [2] Prompt-Matched Semantic Segmentation. arXiv2022\n- [3] LPT: Long-tailed Prompt Tuning for Image Classification. arXiv2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SfVGTbK5qtF", "reviewer": "Reviewer_HQNh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how to fine-tune pre-trained vision models with SGD. It particularly focuses on out-of-distribution (OOD) tasks. It is shown that the original SGD leads to poor OOD performance compared with AdamW on ViT architectures. Authors explain the reason as the large gradients on embedding layers, and propose to fine-tune with freeze-embed SGD. A further improvement is to gradually unfreezing parameters. ", "summary_of_the_review": "The key finding of this paper is informative and useful.\nThe major drawback is the lack of comparative studies. \n\n=== after rebuttal ===\nI have read the authors' response and other reviewers' insightful comments. Though I still appreciate authors' explorations on the OOD problem, I agree with other reviewers that the paper lacks convincing analyses regarding the empirical observations. Authors' response solved some of my concerns, but the discussion with other similar approaches (i.e., freezing a part of the network) is still not comprehensive. Taking ResNet as an example, it is common in practice to freeze the lowest one of two blocks for efficient or robust fine-tuning. It is not clear whether the performance gain of the proposed method can be attributed to the particular role of the embedding/stem layer, or just generally freezing lower layers? It would be better if the analyses in Appendix F can be confirmed by solid evidences. Overall, I agree with other reviewers that this paper needs further improvements for publication, and I decrease my score to 5. ", "strength_and_weaknesses": "Strength\n1. This paper is clearly written. \n2. This paper investigates an important problem of fine-tuning vision models, with the goal of improving OOD accuracy. \n3. The proposed method is simple and effective. \n\nWeaknesses\n1. As authors mentioned in Section 6, there are other transfer learning approaches also suggest freezing a part of parameters. But the differences and advantages with this work are not fully discussed. There is also not any experimental comparison.\n2. Some questions need to be clarified. (1) What’s the embedding layer in ResNet? Is it the first convolutional layer or fist block? To my knowledge, the terminology “embedding” is particularly used in Transformer architectures. (2) Why are some numbers (Waterbirds:89.8, DomainNet:98.8) in Table 5 inconsistent with those in Table 1?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NA7v8AQD3R", "reviewer": "Reviewer_YMfd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues that SGD with freezing the first embedding layer can perform competitively with the AdamW for modern vision models, by empirically tests the method on 5 finetuning benchmarks.\n", "summary_of_the_review": "The paper shows interesting empirical results of improving SGD for fine-tuning vision models, but more analysis would be appreciated.", "strength_and_weaknesses": "### Strengths\n1. The SGD (freeze-embed) is simple but effective on all the models shown in the experiments.\n2. The method performs well on both OOD and ID datasets.\n3. The paper achieves SOTA results on multiple benchmarks.\n\n\n### Weaknesses\n1. The paper claims the first layer is an outlier of many existing models with embedding layer, by comparing the L2 norm of the gradient for each layer. It is pretty straightforward as the embedding layer has much more parameters than the others. It would be great if some analysis can be made for whether the average gradient per parameter has a difference for the first layer.\n2. A deeper analysis of why the gradient norm affects the finetuning would be more interesting. For example,  it may be helpful to understand how the freezing part of the embedding layer will affect the performance.\n3. It would be interesting to see if it will achieve better results for SGD using a smaller learning rate for the first layer instead of freezing it for non-ResNet models.\n4. The paper claims the existing gradual unfreezing can perform better than the SGD (freeze-embed) proposed in the paper. It is unclear, why should people consider SGD (freeze-embed), instead of directly using gradual unfreezing.\n5. The paper implies that SGD is more efficient during the training than AdamW. It would be great if the memory consumption for the experiments can be shown in the tables to empirically show the evidence.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CNlteHhx7aV", "reviewer": "Reviewer_iVCF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to narrow the performance gap between SGD with momentum and AdamW when evaluating on the downstream tasks. The authors find that for the models where AdamW outperforms SGD, the gradients of the first layer are much larger than the gradients of the other layers. Therefore, they try to freeze embedding layer and propose a novel training technique to improve the performance of SGD with momentum and narrow the gap. They also conduct various experiments to verify the performance of the proposed method.", "summary_of_the_review": "NA", "strength_and_weaknesses": "Strength:\n\nThis paper focus on an important research topic. If we can narrow the performance gap between SGD and AdamW, the memory cost will be reduced during the process.\nThis paper also provides an interesting experimental finding: First-layer gradient is an outlier.\nThis paper is easy to follow. \n\nWeakness:\n\nThe novelty is limited. The first layer shows a different pattern and freezing embedding layer is not novel and has been proposed in some related work. [1] [2] \nThe research area about fine-tuning vision models (in your title) is very large. The experiments in this paper mainly focus on data shift datasets. Maybe you can report the experiment results on more general datasets. (such ImageNet pretrain and CIFAR fine-tune).\nI think the large memory cost of Adam is a very important motivation of your work, maybe you should also provide some results about that. \nThe resolution of Figure 2 is too low for me. \n\n[1] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning Transferable\nFeatures with Deep Adaptation Networks. ICML 2015\n[2] Jeremy Howard, Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject