How to prepare your task head for finetuning
Authors: Yi Ren
Shangmin Guo
Wonho Bae
Danica J. Sutherland
Venue: ICLR 2023 poster (2023)
Keywords: representation learning
finetune
transfer learning

Abstract: In the era of deep learning, transferring information from a pretrained network to a downstream task by finetuning has many benefits. The choice of task head plays an important role in fine-tuning, as the pretrained and downstream tasks are usually different. Although there exist many different designs for finetuning, a full understanding of when and why these algorithms work has been elusive. We analyze how the choice of task head controls feature adaptation and hence influences the downstream performance.  By decomposing the feature's learning dynamics, we find the key aspect is the training accuracy and loss at the beginning of finetuning, which determines the "energy" available for the feature's adaptation. We identify a significant trend in the effect of changes in this initial energy on the resulting features after finetuning. Specifically, as the energy increases, the Euclidean and cosine distances between the resulting and original features increase, while their dot product (and the resulting features’ norm) first increases and then decreases. Inspired by this, we give several practical principles that lead to better downstream performance. We analytically prove this trend in an overparamterized linear setting and verify its applicability to different experimental settings.

Published as a conference paper at ICLR 2023
How TO PREPARE YOUR TASK HEAD FOR FINETUNING
Yi Ren
Shangmin Guo
University of British Columbia renyi. jos.oooshhauae gma i 1. com
University of Edinburgh
a wwwged.ac.... ac.uu
Wonho Bac
Danica J. Sutherland University of British Columbia & Amii dsuthêce, ubc. ca
University of British Columbia whbae@cs. ubc.ca
ABSTRACT
In deep learning. transferring information from pretrained network to a down- stream task by finetuming has many benefits. The choice of task head plays an important role in fine-tuning, as the pretrained and downstream tasks are usually different. Although there exist many different designs for finetuning, full under- standing of when and why these algorithms work has been elusive. We analyze how the choice of task head controls feature adaptation and hence influences the downstream performance. By decomposing the learning dynamics adaptation, we find that the key aspect is the training accuracy and loss at the beginning of finetuning, which determines the "energy" available for the feature's 's adaptation. We identify significant trend in the effect of changes in this initial energy the resulting features after finetuning. Specifically, as the energy increases, the Eu- clidean and cosine distances between the resulting and original features increase, while their dot products (and the resulting fcatures' norm) first increase then de- crease. Inspired by this, we give several practical principles that lead to better downstream performance. We analytically prove this trend in an overparamter- ized linear setting, and verify its applicability different experimental settings.
INTRODUCTION
In the era of deep learning, pretraining model on a large dataset and adapting it to downstream tasks a popular workflow. With the help of large amount of data and huge computing resources, the pre- trained model can usually provide beneficial features for the downstream tasks. Such a framework proven to be efficient and effective in many domains and tasks, e.g. natural language process- ing (Kenton & Toutanova, 2019), computer vision (Chen et al., 2020b), graph based learning (Liu et al., 2022), and so on. Although different variants of pretraining and finetuning (FT) methods are widely applied including direct finetuning, finetuning after linear probing (Kumar et al., 2022), side-tuning (Zhang et al., 2020a), using different learning rates for different layers (Zhang et al., 2021), and more a detailed understanding of how features are adapted during finetuning under different settings remains elusive.
Our work builds significantly off the analysis of Kumar al. (2022), who study the interactions between the "task head" (the final layer of the network, usually randomly initialized) and the "back- bone" (usually copied from the pretrained model). Kumar et al. claim that the standard finetuning method, randomly initializing a task head then updating all parameters of the whole network, can distort the pretrained features and hence can deteriorate the generalization ability (as they assume) the previous backbone features were optimal for the downstream task. By analyzing overparam- eterized linear model, they prove that linear probing (i.c., only updating the parameters of the task head) first, followed by finetuning the whole network, leads to better performance in their setting.
In this work, we consider less stringent assumptions than they made, and study more practical set- tings from a different perspective. First, we consider scenarios where the pretrained features not optimal for the downstream tasks, thus feature adaptation indeed beneficial. Unlike the two extreme cases studied by Kumar et al. (2022), i.e. finetuning with fully random initialization and

---

Published as a conference paper at ICLR 2023
Stapping
100
Xpr
PT-baskoone
FT head
95.7
HP
000
06.5
converge
XDs
DS hackbono
D5-head
D5-loss
M(w.y)
95.3
FT valid-acc HP train acc
880
HP stage, update ((x) only
stage, update fxx and (xt) together
Head
Figure 1: Left: general example of pretraining (PT), head probing (HP) and finetuning (FT) pro- cedure (DS is short for downstream). Right: an example showing that neither probing the head to converge nor probing is the optimum (pretrained ImageNet-1K K and finetuned on STL10). fully-pretrained parameters, we consider intermediate cases where features are mildly adapted by stopping earlier (before convergence) in the linear probing procedure. To better understand the fea- ture's ss behavior, we decompose the learning dynamics of the feature vector during finetuning based on "energy" and "direction" " of the learning. We discover a non-trivial trend how this "energy" affects the way that features change from their initialization, which can inspires to design an ap- propriate finetuning procedure. Under this framework, we demonstrate that the "unchanged feature" assumption of Kumar al. (2022) is hard achieve.
Second, our task heads are not necessarily linear. Inspired by the illustrations of Olah et al. (2020), reasonable only preserve the lower layers of the pretrained model and reinitialize the top layers, assuming that the low-level features are common across task. That is, the probed task head non-linear, and we refer this more general process as "head probing" (HP) rather than linear probing. Our analysis can also help to explain feature behavior in this setting.
Finally, following our analysis, we provide a user guide conclude when and why specific methods should be considered. Specifically, we have one basic method: stop head probing earlier, before convergence; and three advanced tricks: 1.) use label smoothing during head probing: 2.) use more complex task head design; 3.) merge and reinitialize some later layers of the backbone and attach them to the task head. In summary, in this work:
we formalize and explain feature adaptation by decomposing the learning dynamics; we find a non-trivial trend feature adaptation and verify many cases; and show how controlling feature adaptation can improve downstream performance.
2 MOTIVATION
Pretrain-then--ieetue is a popular workflow for many tasks in deep learning. One common practice to 1) randomly initialize task head, attach it to pretrained backbone, then 3) finetune the whole network together (Li et al., 2020). However, the untrained task head may distort the pre- trained features during finetuning. To solve this problem, Kumar al. (2022) propose train the head to fully converge before the finetuning. However, suppose we train the head long enough and its training accuracy (HP-train-aco)
converges 100%, then the features won't change during the finetuning stage. To sum up from the above, we can see that neither probing the head to converge nor no probing optimal, since the pretraining and downstream tasks (or datasets) are usually distinet. To verify this argument, we HP various number of epochs before finetuning, and record the cor- responding validation accuracy after finetuning (FT-valid-ace for short), and the results are shown in Figure 1. It is surprising to that stopping the head training carlier (before the convergence of HP-train-acc) brings more improvement. As the only variable among these experiments is the parameters of the head before finetuning, the following two questions emerge:
How does the task head influence the pretrained features during finetuning? How does the feature adaptation influence the generalization performance after finetuning?
3 BACKGROUND
We first clarify the scope of our analysis. We don't 't consider the details of the pretraining procedure, instead just assuming that there some well-trained checkpoints for particular dataset or task.
2

---

Published as a conference paper at ICLR 2023
Meanwhile, our formulation is not restricted to classification tasks; use of the term "label" or "target" can be any form of supervisory signals.
3.1 PROBLEM SETTING AND TWO STAGE TRAINING
When training a model for downstream task, our goal to find a predictor f oog IR R that maps high-dimensional input signal x E Rd to a task-related prediction E R As depicted by the left-bottom panel of Figure I, we split the predictor into two parts: the backbone f(x B) RR,, and the task head
Rd + Rh which maps the input signal to intermediate representations g(z.v): : R + R which gives the prediction vector E R (e.g. logits in classification tasks using cross-entropy loss). Usually, the backbone / is parameterized by B and initialized by copying from a pretrained model. The task head ! parameterized by v, on the other hand, is usually randomly initialized and might be complex non-linear function. The training has two distinct stages: 1) head probing (HP) where we fix f and only update the parameters v of for epochs; 2) finetuning (FT) where the parameters {v, B} of f g are updated together until convergence. In this work, we analyze how the FT stage is influenced by the architecture and v at the beginning of finetuning.
Following the general formulation above, we show a simple overparameterized linear regression (or equivalently, binary classification) below as a case study to illustrate more insights. Suppose the N input signals are stacked, X = [x(1,, x x(N)TT E N xd, the loss function can be written as
1 1 v v
CB. 2
where B Rxx v R², and Y e RN, That 1,z Bx. q = zJ and
and C(q. 8) (9)
3.2 WHAT EXPECT DURING ADAPTATION
Compared with training a randomly initialized model, adaptation on downstream tasks needs more care. One reason is that the pretrained parameters from [xx; B) inherits all the information from the pretraining task. even any bias and noise. Furthermore, as mentioned by Kumar et al. (2022) and Du et al. (2018), (x; B) is tied to (z; v) at each time step during FT, thus the bias and noise in f(x; B) also influence the learning of g(z v). Hence, before conducting downstream adaptation, we might consider: to what extent do we want to change the feature extractor (x; B)? Here, we list three possible cases of how much we should update fxx; B):
Strong: the pretrained features are far from the optimal ones for downstream task, so we need substantial feature adaptation.
Mild: f(x B) is reasonably good, but adaptation to the downstream domain is helpful.
Tiny: the pretrained f(; B) is near optimal and only need to be slightly adapted.
In the rest of this paper, we first analyze how f(; B) and g(z; v) interact with each other, under both the general case and the simplified case. Based on our observations, we propose several practical principles in Section for better downstream performance.
INTERACTION BETWEEN THE BACKBONE AND HEAD
Rather than linking the choice of Vo (the task head parameters at the beginning of FT) to the loss function, we sketch how f(x B) and g(z; v) interact during FT, which depicts how changes cordingly. Although our analysis cannot provide any theoretical guarantee, knowing how Z changes under different No will lead to better HP-FT design practice.
4.1 AVERAGE INITIAL ENERGY
We start from analyzing the behavior of 21") (xx Bt), i.e. the feature extractor time when the network parameters updated with samples (1) - xNN using gradient descent. When using
3

---

Published as a conference paper at ICLR 2023
spocts
Res50-IN1K(upp-S5
6.3
Random
3.3
the
HP
0.3
32
29
HP 128
1.
updates
FT updates
(a) Influence of T on energy.
(b) Influence of direction.
Figure 2: Left: histogram of Po and e, under different 7. Right: approximated change of the 'direc- tion' term in Equation (2) under different т. The titles represent the settings. HP means HP for epochs. Toy+MLP means pretrain a 4-layer MLP on full MNIST then transfer to distorted subset of MNIST. Res50-INIK(Supp) classification, then transfer to a downstream task on STL10. See Appendix B for more details.
means pretrain ResNet50 on ImageNet- 1k using supervised
cross-entropy loss, we can have the following result:
5 2+1
()
(j.m)
(vzq
eyy,
p(
0(72),
N
slow-change
direction
energy
where y is the learning rate, Kig (j.m) (V
(Vez!")) E hgg is the empirical neural tangent
E R is the gradient of the task head prediction w.r. the representation vector at time I, p Softmax (") is the predicted probability vector for input x(7), and ey.. is the one-hot vector of the label yn-
kernel (NTK) of the backbone between x(i) and x(") at time t, Vzq: (n)
In this decomposition, the first term is often understood to change only slowly during FT, i.c. the lazy-parameters setting used by Chizat al. (2019); Yang & Hu (2020). The second term determines the direction, and the last term provides "energy" for the update of z. FFrmally, we can define the
Average Initial Energy (AIE) and use it to bound the norm of 2
(3) (1)
Pa ll2 is the Average Initial Energy (AIE). Here c a constant, T'' the FT epochs, and Po the model's prediction of sample xlm) at the beginning of FT (or at the end of IIP).
Proposition Extr [//
2
lla I c. Ease. where Eaic &
Eximo leg
(n)
(n)
Proof. See Appendix A.
LI
Although this bound is loose and requires approximations based on assumptions, the proposition supports intuition: if we prob the head for too long (i.e. large r), the HP-train-ace can be very high, hence on average Po is close to e, and the features adapt less, and vice versa. Our intuition is further verified in the right panel of Figure where we plot the predicted probability of the correct class, i.e. (p6 (n)) Jy.. for each sample. Furthermore, we find that it is unlikely to obtain zero energy (not as assumed by Kumar et al. (2022)) in many practical applications: even with high accuracy there will always be some gap between Po and e, by definition, and moreover the training accuracy after HP is sometimes far from perfect (see Section 5).
4.2 NON-TRIVIAL TREND OF FEATURE ADAPTATION
Following Proposition 1, we can link the adaptation of features to 7 via AIE. In the proof Propo-
sition 1, we disentangle the dependence of the direction and energy terms using Cauchy-Schwaad
'The case for MSE loss, well as the derivation of this equation, can be found in Appendix A.
II linear model z Bx, n (j.m) (x)) (x(()) Iixc is invariant during FT (i.c., independent of t).
"Note that V.q."M (n) depends on the parameters of the current task head v in the generally, whereas V.q.". (m)
independent of if the model linear.
4

---

Published as a conference paper at ICLR 2023
Thiring accustocy
(o Aqg
Strong
Mild
Midd
xolli
Tiny
ll. xa18
Tiny
Serönng
Daso Gaadd
Claadd
1042
Tiny
Mild
Strong
Figure 3: How Z; changes from Zo. The ellipses in the third panel represent the scattered features. The scatter plots are the PCA projection of the resulting 2, after finetuning. For tiny, mild, and strong adaption,
1024, 64, O, respectively. The heat-maps of the last column are the "head-exchange" experiments (all under the Toy+MLP settings).
Polla. 2. However, recall that the direction term Equation (2) also plays an important role, especially at the beginning of finetuning.
inequality, i.e., (V.q)((e (e, Po) < 1/V2 lle,
To verify our hypothesis about the direction term, we depict the change of this term during finetuning in the right panel of Figure 2. Suppose linear task-head (non-linear heads have similar behavior in the NTK regimes), this quantity can be approximated by the norm of the gradients to Vt, since |V>9++1 V.q. || = +IV, v
7² VV.C As we find ||vel2 changes little during the finetuning stage, the large |/, is more likely from big direction change. As illustrated by Figure 2, when O, V.q, changes lot at the beginning of FT, which make z(1) change in inconsistent directions. When r = 1024, the direction term changes only a little through finetuning.
look deeper at what is the difference between strong adaptation (e.g., r = 0) and a mild adaptation (e.g.. 7 = 4). To get an overall picture of z's change, only ||zr- Zu Zu||2 not enough. Hence we analyze the following four quantities related to the similarity between the features before finetuning Zo and afterwards Zy: ||z zoll2, ||z]||, Zo. and cos cos(27, ZT, Zo). With an overparameterized model, we can analytically calculate the expressions of them and make the following conclusion:
This finding inspires
Proposition 2 (Informal). In an overparameterized two-layer linear model, when t increases (the AIE decreuses), ||Zz zollž monotonically decreases while ||z 1/ and z exhibit a quadratic trend. The trend of cos(zr: 2o) is hard to predict, but there is phase that this value increases fast.
Such trend is illustrated in Figure 3, supported by various experiments in Appendix C. and strictly proved in Appendix D. With this information, we can infer the behavior of zy and sketch it in the second and the third panels. In the "tiny" case, when vi is fully converged (r oo), the features are almost stable during finetuning; this only works well when the pretrained features are perfect for the downstream task. In the "mild" case, when T is reasonably large, the resulting ZT will be stretched ([|zr|l2 increases) in a similar direction (cosine changes little). This kind of z can make the original overlapped features become more separable without changing the manifold of the features too much, which is desirable in most transfer learning scenarios. For the "strong" case, where we only HP for few updates or simply use random head. Zr will change in an unpredictable way, especially for the early updates. Although the fine-tuned model may generalize well, Zr might be quite different from Zo. Thus, we believe the pretrained features are too specific the pretraining dataset, the "strong" case is reasonable choice.
Note that even though the seatter plots of the mild and strong cases look similar, the correspond- ing feature manifold might be quite different. To verify this, we first run HPr-FT (i.e.. load the pretrained model, prob the head for epochs, then finetune to converge) for different т. Then save the converged backbone and task-head separately for each т, and wair-wisely exchange the head and backbone to build 49 new models (without further tuning). The training and validation accuracy of the 49 models are reported in Figure 3. The results match well with our analysis: 1) the
5

---

Published as a conference paper at ICLR 2023
off off-diagonal values are lower, which means the representations learned by experiments with differ- ent T are different; 2) for the strong adapted backbones (the first two columns), the heads from other cases are not compatible, which means the features' manifold are significantly changed; 3) for the mild and tiny cases (latter columns), the aforementioned incompatibility almost disappears, which means the features' manifold of these cases are quite similar.
4.3 BACKBONE DEPTH AND HEAD CAPACITY
Beyond how long train the task head, the structure and capacity of the task head also influence the training prediction after this stage, thus the adaptation of z; We briefly discuss the trend here and verify them Section 5. When using a low-capacity head (e.g. linear head parameterized by h x k matrix), it might be hard to preserve the pretrained Zy even a very large is chosen, as the head cannot achieve a high enough training accuracy decrease AIE. On the other hand, if the capacity of the task head is much bigger than the backbone, the information from the pretrained network might be easily distorted. For example, consider using only the first block of a pretrained ResNet18 as the backbone and concatenating the other 3 blocks with 10-layer wide MLP. If this huge task head is randomly initialized, the information contained in the backbone could be completely rewritten, as the random changing phase of Z; be very long (remember that in Equation (2), Z, will change in random directions before V.q. becomes less noisy).
5 EXAMPLES ON REAL TASKS
In this section, we first provide "user guide" of all the aforementioned methods. Note that given the pretrained and downstream tasks, determining the optimum adaptation energy rather heuristic: the user guide only provides some high-level suggestions for the practitioners. We then provide abundant real examples of how to apply these principles. Generally, we can first try sweeping the optimal r using linear head and consider other advanced tricks if necessary.
5.1
USER GUIDE
The goal of this paper is provide toolbox for preparing the task-head before finetuning. Here, we suggest a "phenomenon + hypothesis - solution" workflow and use the validation performance for verification. Recall how mitigate the overfitting problem using dropout: validation accuracy decrease after convergence - model overfitted -> add dropout. Similarly, we can have: HP train- acc converge 100% no enough energy - use smaller T.
However, in practice, it is unknown that how much energy is beneficial, as the neural network might not encode the input samples as humans expected. Hence, suggest starting from the basic setting, i.c. using a linear head and sweeping 7 to get a high-level understanding of how the energy influence the downstream generalization. Usually, selecting the optimal using validation accuracy can ensure reasonably good test performance, as verified in the next subsection.
The advanced tricks are only applicable to specific scenarios and need more consideration. For ex- ample, if we really want tiny energy but using the linear head only achieves less than 50% training accuracy after head probing, we can consider an MLP head increase the converged training ac- curacy (hence reduce the energy). If we want mild adaptation, but the training accuracy during HP goes to 100% too fast, using label smoothing during HP can considered. If we believe the downstream task only needs low-level features of the backbone, bartial-backbone is good choice. However, these advanced tricks also have side effects that are deleterious to the downstream perfor- mance. We will analyze their advantage and limitations in Section 5.3 with concrete examples.
5.2
BASIC METHOD: EARLIER STOPPING HP
Following our theory, when the learning rate for HP is fixed, the value of is positively corre- lated with the training accuracy, thus negatively correlated with energy. Hence, given a pretrained backbone and downstream task,
always start from sweeping (see Figure 4).
As stated in the motivation part, during finetuning, we usually expect some feature adaptation while keeping some pretrained information. Hence, neither 7 0 nor xo is optimal. We find such

---

Published as a conference paper at ICLR 2023
37.47
06
90.2
60.5
20F
101
96.0
107
30"
100
97.0
307
127
10P
101
204
107
107
+PP vill aoc
300
PP catteess
66
73.0
305
127
122
30"
123
307
327
307
1,27
20"
107
257
10"
10P
mpochhc fing sowel
w0o.oo (log sCile)
ppotee (og srale]
mpothhs [0o SOW
aports (log smile)
aporte Ohe scaee
Figure 4: Sweeping from O to 200 (in 2 fashion). The valid-accuracy of FT-only setting is the left most point in each panel. The first 4 columns are on image classification task, the fifth one is on graph task, and the last one is image segmentation task.
downstream data:
Hlowers
STLTO
CIFARTO0
Tox21
pretraining usk-ddtt::
McCo-IN
Sup-IN
Sup-CI0
McCo-IN
Sup-IN
Sup-C10
MoCo-IN
Sup-IN
Sup-C10
Sup-peba.
HP--rain-acc
59.711
85.826
7,031
91 889
96,291
75.181
56.397
62.709
11.735
79.12]
HPO.PT
76.953
91.295
60.714
96. 136
97.452
86.811
80.432
84.736
65. 65.608
81.274
IIP200-FT
91.653
96.363
97.374
84.778
83.744
84.347
HPPFTPTT
MLX11
92.299
63.71
96.753
97.697
87.739
8X.14
84.971
65.966
81.63
IT
92.746
42,299
96.639
97.959
85.912
83.412
Table 1: Downstream test accuracy across different settings. r" is selected based on validation accu- racy in sweeping. The HPr* always bring some improvement. The IsHP can further improve when the HP-only method achieves a high accuracy (no enough energy for adaptation), but fails when the pretraining features are not suitable (in Sup-C10 case). See Section 5.3 for more discussions.
trend is consistent across various settings: from image input to graph input, from classification task to segmentation task, and from supervisory pre-training to unsupervised pre-training.
Other than this general trend, the nuance of experiments under different settings also supports our analysis well. Specifically, in the image classification experiments shown Figure 4, adapting a ResNet50 to STL10 behaves differently large 7 (small energy) hurts the downstream generalization performance, because the features pre- trained on CIFAR 10 might be far from optimal for the downstream task. In contrast, the features pretrained on ImageNet-IK 1K (INIK for short) all provide good results. Among those INIK pre- trained models, the model from supervised classification task leads the best overall downstream performance, but large is still harmful. In other words, the features pretrained this way might be somewhat too specific to the pretraining dataset, and hence mild adaptation is beneficial. Regarding the unsupervised pretraining cases, BYOL (Grill et al., 2020) is less sensitive to the choice of т, while MoCo (He et al., 2020) behaves more similarly the supervised case.
different pretraining task. In the first column, we see
Another interesting finding is that the HP training accuracy at the optimal +" is usually smaller than the converged value: we cannot select based on the standard early stopping criterion on HP train accuracy. As the task head under 7* usually has not converged on the pretraining dataset, we call this method "earlier stopping HP" (HP7~-FT for short). As shown in Table 1, HPT*-FT can always bring improvements.
5.3 ADVANCED TRICKS: SUPERIORITY AND LIMITATION
Instead of using a linear head, authors of Chen et al. (2020b) claim that using an MLP head can sometimes bring improvement. Following the analysis of this paper, we can consider this trick when we want small energy while the linear head cannot reach high HP training accuracy even after long HP. For example, in the first two columns in Table 2, the HP-train-ace in the linear head cases plateaued after reaching 92% or 78%, while a 2-layer MLP can reach 99% and 95%. As the energy decreases, features adapt less during finetuning (see the decreased distance metrics) and the final models generalize better.
MLP Head:
However, we should be careful when applying this trick we want to use a small T (i.e., large energy). Recall our analysis that the inconsistent direction term (i.e., V2q, defined in Equation (2)) makes the feature adaptation more unpredictable at the beginning of finetuning. Increasing the head capacity in this case would make the head converge slower and hence prolong such chaos phase.

---

Published as a conference paper at ICLR 2023
Sim Real
Sup-Sketch
Sin-STL
Sup-STL
Sim-
Sup-STL
IIP-train--c
92.676
78.213
Baseline Small energy
Small energy
97.607
1 1-cosi2y:22 Xu]
Baseline
0.1269
0.1422
0.0144
0.017
0.0049
0,0256
[ 2oll2
Lincar-ead
5.247
4.752
1.792
6.716
NFT 0.9
1.731
7,391
FT-val-ace
78.075
61.545
9HP
93.914
97.581
Vie 0.9
94.015
97.094
1-coel2m. zol
HP-rrain-ace
|izy zollă
2MLP-head
99.121
0.1098
4.942
94.883
0.1104
4,206
More energy
96.875
0.014
1.841
0.0549
9.278
Opposite Energy
Nee 0.9
0.0109
97.982
3.041
0.0849
13.881
I7T-val-ace
78.453
63.773
94.304
97.92
93,208
97,039
Table 2: How Is-HP and larger head influence the feature adaptation. The models are pretrained using SimCLR (Sim) or supervised (Sup) classification on INIK. Number in blue (red) represent an decrease (increase) compared with its counterpart in the baseline. See more results in Appendix C.
Label Smoothing HP: Recall that the energy is upper bounded by l,e, Po ll2. where Po is the model's prediction after HP for T epochs. Po converges to the labels used in HP when - cc. Hence, instead of changing т, we can also manipulate the labels in HP achieve a similar goal. One simple yet effective way is label smoothing (Müller et al., 2019, e.g.). By setting the labels during HP as ry + (1 7HP) + u, where u is uniform K-class categorical distribution, the HP stage can always reserve at least (1 - 7)HP)* |le, uI/2 energy for the following feature adaptation, even T + oc. Such trick (IsHP for short) is quite helpful when the HP-train-acc converges to 90%+ very fast, yet we still want a mild adaptation, like the example shown in the second two columns in Table 2. With IsH,, we see that the features adapt more even the HiP-train-ace accs are unchanged.
To verify that the aforementioned improvement comes from the reserved energy during IsHP, we further try using smoothed labels during finetuning (e.g., 1FT = 0.9). The results match our anal- ysis: when пHP 1FT 0.9, the reserved energy disappears, as the labels of the two phases are the same again. Hence, all the numbers under this condition are quite similar to the baseline case (NиP 1JF 1). For the "opposite energy" case, we observe larger adaptation but a worse generalization performance. That is because the reserved energy make the features adapt in opposite directions. These results remind us that if we decide to use smooth label in both finetuning and head probing (e.g., we assume most of the samples in the downstream dataset are ambiguous), we need to set MHP S WFT to ensure a correct direction.
In summary, the IsHP trick is suitable for scenarios where the pretrained features are pretty well and the standard HP converges to 90%+ very fast. When the HP-train-ace ace is too low, the assumption used in HP, i.e. Po converges to the labels, no longer holds. Hence, IsHP does not always bring enhancement, an example given the last row in Table 1.
Partial Backbone:
This is a more intricate trick that requires a stronger and more heuristic hypothesis. For example, there is com- mon belief in the deep-learning community that the lower layers extract fundamental features (e.g. edges, curves, and texture in vision tasks) while the higher layers learn more semantic features (c.g. dog heads or wheels) (Bal- dock et al., 2021; Olah et al., 2020). Hence, if we be- lieve the downstream task treats the low-level features beneficial while the high-level features are harmful, re- initializing the higher layers (like the fortuitous forgetting mentioned by Zhou et al. (2022)) and incorporating them as part of the task head can be beneficial.
Real
Sketch
Quick
HP-train-ace
92.676
78.213
64.307
-L4.3
100
100
97.666
-L422 -L411
100
100
100
FT-val-ace
78.075
61.545
59.703
-L4.33
78.528
65.683
67.011
-L4.2
78.427
65.336
67.087
-L4.1 76.689
65.017
68.07
Table 3: ResNet50 pretrained on INIK using SimCLR. -L4.x means reinitializ- ing the blocks in the resnet (merge them to task head) until layer 4.x.
See the results in Table 3, where the Quick dataset is likely to rely more on the low-level features learned dur- ing pretraining. So the optimal setting for the Quick column is removing information from the last three layers (i.e., L4.3, L4.2, and L4.1) in ResNet50, while the optimal setting for the other two cases is to reinitialize only the last layer (i.e., L4.3). Using this trick, the task-head capacity might increase significantly (see the HP-train-ace increase to 100% after reinitializing). Hence, the prin- ciples discussed in the MLP-heal trick also hold here. However, as it is hard to figure what are the beneficial features for downstream tasks, such a trick has the lowest priority in our toolbox.
*Refer to Figure 8 get an intuition about what the samples in theses dataset look like.
8

---

Published as a conference paper at ICLR 2023
6 RELATED WORK AND DISCUSSIONS
HI,, FT and HP-FT. Head probing and fine tuning are two fundamental algorithms in transfer learn- ing, which have also attracted much discussion and debate. Intuitively, by freezing the pretrained backbone, HP will re-combine the features without distorting them, and hence yields better perfor- mance than FT when features are perfect (Peters et al., 2019). However, when pretrain and down- stream tasks are very different, adapting the features is important and FT outperforms HP (Chen al., 2020b; Zhai et al., 2019; He et al., 2022). Combining the strengths of HP and FT, authors of Kumar et al. (2022) demonstrate that HP-FT (i.e., first HP, then FT) yields the best performance on both in-distribution and out-of-distribution cases. Although they provide some theoretical guaran- tees for the superiority of HP-FT, some of their assumptions are dubious. To shed more lights on the pros and cons of these two methods, this paper amalyze in detail how the features change under different task-head settings. Specifically, by controlling the HP epochs or otherwise influencing the energy term, we can design different types of feature adaptations for different downstream tasks. The relationship between the choice of task head and resulting z's adaptation is explained using Equation (2), verified by various experiments, and proved in an overparameterized linear model. Although the paper only analyze these fundamental settings, believe the analysis provided here can also be combined with (or explain the benefits of) other more complex finetuning mechanisms, like those of Guo al. (2019); Zhang et al. (2020a); Aghajanyan et al. (2021); Howard & Ruder (2018). For example, a common practice in transfer learning is to use a small learning rate for the backbone and a large learning rate for task head. From our perspective, this method can weaken the influence of the "noisy direction" term and make the features adapt in low energy condition.
Backbone and head depth. Besides the influence of the initial value of the head, this paper also discusses the influence of the relative capacity between backbone and head. Chen et al. (2020b) show that a simple MLP head sometimes also brings enhancement. When the backbone only copies the early layers of the pretrained network, as the downstream task might need different levels of features (Olah et al., 2020; Baldock et al., 2021; Zhang et al., 2020b), the later layers and the original head can be combined the new complex task-head, like the example in Figure 1. Furthermore, in some encoder-decoder style models, like some the models for some language tasks (Peters et al., 2019; Zhu al., 2019), the head (decoder) might have comparable capacity with the backbone (encoder). In another extreme case, if we use pretrained word2vec features (Mikolov et al., 2013) and plan to FT them in downstream task. the task head is the whole network, which is much bigger than the embedding layer. We believe the discussion in this paper might also help inspire the design of HP-FT strategy for these practical scenarios.
CONCLUSION
This paper studies how the choice of task heads influence the pretrained features z's adaptation and hence influence the downstream performance. By decomposing the learning dynamics of z, we find the keys are the energy and direction terms, which highly correlate with the accuracy at the begin- ning of the FT process (i.e., the accuracy after HP). Hence under common HP-FT framework, we carefully analyze the relationship between the HP epochs (7) and the features' adaptation (discrep- ancy between Zι and Zu). Different from most existing works. which mainly focus on Euclidean or cosine distance, we further analyze 2,z20 and |zoll2 1|2 to depict a more comprehensive relationship between т and Zr, like the examples in Figure 3. This non-trivial trend is strictly proved in an overparameterized linear model and experimentally verified under different practical settings.
Based on these observations, speculate that a suitable adaptation of ZT is beneficial when the pretrained features are not perfect for the downstream tasks. Under different experimental settings, illustrate how achieve better downstream performance by controlling the adaptations of Zr (using early-stopping HP, label smoothing during HP, or manipulating the head's capacity).
Finally, there are still many open questions and phenomena not address by this study. For exam- plc, methods to quantitatively or even adaptively analyze the discrepancy between the pretrain and downstream tasks would be very useful in knowing what kind of head probing to perform in ad- vance. However, we believe the methods proposed in this paper can help provide new perspective on understanding feature adaptation, which we hope will aid future work in this area.

---

Published as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta. Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. International Conference on Learning Represeniations, 2021.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 2021.
L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017a.
Liang-Chicl Chen, George Papandreou, lasonas Kokkinos. Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crís. IEEE transactions on pattern analysis and machine intelligence, pp. 834 848, 2017b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint, 2020b.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statisties. JMLR Workshop and Conference Proceedings. 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 leee, 2009.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in Neural Information Processing Systems, 2018.
M. Everingham, Eslami, L. Gool, C. Williams, J. Winn, and A. Zisserman. Pascal Vis. Obj. Class Challenge: A Retrospective. IJCV, 2015.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 (11):113301, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2123 2020.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEFF/CVE con- ference on computer vision and pattern recognition, pp. 4814, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2023
Tong He, Zhi Zhang. Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition, pp. 558-567, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for classification. ACL, 2018.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint 2.xiv:2005.00687. 2020.
Jacob Devlin Ming-Wa Weig Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https:
//openzeview. wttp://ss//ooommddttotett.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ane-grained categorization. In 4th International IEFE Workshop 3D Representation and Recognition (3dRR- 13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. International Conference on Learning Reprexentations, 2022.
Yann LeCun. The mnist database of handwritten digits. http://yann.. lecun. com/exdb/mnist/, 1998.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 2019.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference Learning Representations, 2020.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang.
Pre-training molecular graph representation with 3d geometry. In International Confer- ence on Learning Representations, 2022. URL https:/ /ttprr////////iwwww//// net wtw.eeewwe//forummtaaem forummidd xxUe wweelpopPast am.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3781, 2013.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances neural information processing systems, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729. IEEE, 2008
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: introduction to circuits. Distill, 2020.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. ACI., 2019.
11

---

Published as a conference paper at ICLR 2023
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):513 2018.
Greg Yang and Edward J Hu. Feature learning in inftnite-width neural networks. arXiv preprint arXiv:201 1. 1..1422,, 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint, 2019.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In European Conference Computer Vision, pp. 698-714. Springer, 2020a.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few- sample bert fine-tuning. In International Conference on Learning Representations, 2020b.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few- sample (bert) fine-tuning. In International Conference Learning Representations, 2021. 2021. URL pettps:/cpeereee
net www.eeooommmdd-ooo111.....
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connec- tionist networks. International Conference Learning Representations, 2022.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. Incorporating bert into neural machine translation. In International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2023
Depends PT/DS tasks
(ey Po)
Energy, direction
Resulting Z
valid accuracy
value of V
1. Theory in overparameterized linear example
2. Decompasition in Eq.2
1. Usually mild adaptation is beneficial Better features need less adaptation Other designs like label-smoothing HP, MLP-head, latter layers forgetting
3. Correlation between (P-train-acc(gap) and dist(zt, z0)
4. Further decompose dist(zt,
5. Trends holds for different settings (MLP, CNN, GNN)
Figure 5: The flow of this paper. Code is available at
hww_too_rapare_tastiie
A DECOMPOSITION OF CHANGE OF Z
Proof of Equation (2):
Recall Z = (x), q = (z) and p = Softmax (q). Use E Rudd1 to represent the vector form of
B E IR hxd The MSE loss on one sample is Lmse (q(1) ey,.) = bll ey 1 The cross-entropy
loss sample is Loe (q(m), eg..) H. (e q (n)), where H(p, q) >n P& log qu is the
cross entropy for categorical distribution. Using 1st Taylor expansion, we have:
+
(0)
Zt
(0)
Voz!
(0)
(bi+1 b.) +0 (llb - by 133
(3)
hx1
hxha
hdx1
We then calculate b,+1
b, assuming the parameters are updated batch-SGD:
VbC ( (n)
bi+1
ey.
(4)
N
hdx1
1xhd
NV
? N
Σ
VgL
qi eyn
V.9
(n)
Vbzz""
(5)
n=1
Exh
Axkd
1xx
N
Y
Σ
(bz((n)
(za((())
VgC
qi (n)
evn
(6)
N n=1
hdxh
hxc
kx1
For different loss functions, have different expressions for the last term:
Vafmse
eyn (m)))
__)))___.)) eg.. ep..):
(9( (m) -ey))
VgLee
(n) Gy..
(p! (n))
egy.
where q is the logits and Softmax (q) is the predicting probability. We see these two kinds of loss have similar expression on this term, so we only explain the cross-entropy version (as is more common in practices) in the main content.
Using the above expressions, we can first bound the high-order term (cross-entropy version):
O(Ib.+I b, 13 O V. (n) 11
b|
(x)
12p
||q (m)
ey,, 11 O
as long as the hyperparameters are appropriately chosen and the loss doesn't 't blow up (or gradient clipping is applied) in FT stage.
13

---

Published as a conference paper at ICLR 2023
Finally, combining all the expressions, Equation (10) can be rewritten
(0)
(0)
Σ
(0)
(vbz! )²
)
T
((=a("))
V VaLco (q. eyn eg.")
+0(77) (9)
2+1
2
N
hx1
hxhd
hdxh
hxk
xx1
7
[0,n)
(zq."") (1))
(n) +0(7²),
Kx
eyn
(10)
hxh
hx
kx1
which is the same with Equation (2) in the main context.
Proof of Proposition 1:
is easy to get Z7
(1) by stacking the LHS of Equation (2) (or Equation (10)) under different
(
(1)
T
(j.)
(Vzql (n) T (e - -1(i)) + +
2T
20
Kit
(11)
T-1
Ex(n)
(1.m)
(v) T (ey.. - p(r)) + +077)
(12)
t=
T-1
=y.
Extn)
k(i.n)
V.q."") (ey. -p!")) + 0077
(13)
t=
7 E Exin)
k(..)
v (ey, -p_h) +0(72)
(14)
0=
where the first equation follows definition. The second equation is assuming a uniform training sample distribution, i.e., p(x) = 3t The third equation follows the slow-change NTK assumption, hence ((ia) no longer depends on t. The last equation follows the linear-head assumption, i.c.,
V.q." (n)
We cannot go further as there are three matrices (or vectors) in the expectation. Hence we instead analyze the F-norm (i.e., 1.2-norm for the vector of the features' change.
14

---

Published as a conference paper at ICLR 2023
T-1
1 - 200l0 -zo
= Ex(n)
h ((in)
Vi
eg.n
Pi
(n))
+0
(15)
1=
$ 7. Ex(n)
V
T
eenn
pi
p((l))
+ 0
(16)
T
89 Extn)
Σ v
(evn
-
P
+0((99
(17)
t=0
1
S7. Ci Ex(n))
Σ v.
(e
p!!m)
+ +0(77)
(18)
1=
T-1
=7. C1 Σ
Exin)
v
cg
Pi (n)
)11 0(73)
(19)
T-1
S9.01
Σ
Ex(m)
2
eyn - p
2
+ 0
(20)
1=
T-1
S7.01 Ci C2
E
Ex(n)
eyo
Pi
+ +0(77)
(21)
2
t=0
S C1 C2 T Ex(n)
eyn
Po (ni
0073)
(22)
Here the first equation is by definition. The second inequality follows triangle inequality. The third inequality follows Cauchy-Schwarz inequality. The forth inequality is assuming the F-norm of NTK bounded by C1. The fifth equation sweeping the summation order. The sixth inequality also follows Schwar- inequality. The seventh inequality assumes the norm of the last layer is bounded by Cz. The eighth inequality assuming a stable learning process where the gap between (n) keeps decreasing during training.
ey., and Po
Finally, by taking expectation of all the input samples, we have:
e ||
2 S c Ex(n) llgg. 1 l|.
(23)
where c = 7 Ci C2 7 is a constant and Excoo) |ley.. Po (n) is the AIE term (Eaie) defined in Proposition 1.
Verification of Slow Kernel Change Assumption:
300
302
150
/07
Train accuracy
Change NTK
Mw
Rnd Backbone
Backbone ep
Norm
Backbone HP10 ep
PT Backbone HP100 ep
103
60 Training (finetuning) time
100
89 Training (finetuning) time
40
80
Training (finetuning) time
Figure In finetuning, the NTK the backbone adapts quite slow the model is pretrained.
We decompose the one-step dynamics of ()) in Equation (2) into three parts, which represent kernel, direction, and energy respectively. In our amalysis, we have mild assumption that the kernel of the
backbone, i.., (j.m) (Vzz.") ()
(Vez!") (" E gg changes slow during finetuning when the
learning rate is small and the training is stable (the loss converges). Although there several
15

---

Published as a conference paper at ICLR 2023
works supporting this assumption (Yang & Hu, 2020; Geiger al., 2020), to make the paper more self-contained, we directly observe how this term adapts during pretraining in Figure
We measure the change of this term by calculating knorm ||e]
|| and kgap - |
(ix)
fit [ F for each & during finetuning. As computing the empirical NTK on the whole dataset re- quires huge memory, we randomly sample 50 our "probing samples", computing Knorm and gapp 2,500 different (x(), (x()) pairs, and then report their mean values each t. To verify our assumption, we compare four different settings:
In Rnd Backbone, we randomly initialize the whole network and train on the downstream dataset;
In PT Backbone HPT ep, we copy the pretrained backbone, attach a randomly initialized head, and then HP for epochs.
In the first panel in Figure 6, we see the training accuracy of all these settings converge to 100% (they also have similar validation accuracy). In the second and third panel, we plot the change of knorm and kgap respectively. It is obvious that compared with using random backbone, the NTK of a pretrained backbone indeed changes slow during the finetuning stage.
B EXPERIMENTAL SETTINGS
B. TOY CASE
The experiments of the toy setting appear in Figure and Figure 3. In the toy setting, first pretrain a 4+1 (4 layers of backbone and layer task head) layers MLP on the full MNIST dataset (LeCun, 1998), with learning rate 10-3 with cosine scheduler until convergence. The hidden width 128 and relu activation is applied for each layer in the backbone. In the downstream task, we only consider the first three classes and randomly select 1000 samples for each class (i.e., 1000 samples in total). We also apply random rotation and center crоp augmentations to simulate the distribution shift in downstream task, like the right panel in Figure 7. The downstream linear task head is then 128*3 matrix (weights) and 1 128*1 vector (bias). To ensure a fine- fine-graine grained observation of feature's s adaptation, we use small constant learning rate (10-4) in all the experiments. The FT epoch is 50, and the HP epoch ranges from 0 to 50,000 (50,000 is simulating the Vipp case, usually train 0 to 1,024 epochs).
2
3
4
5
6
8
9
9
9
(a) Pretrain dataset.
(b) Downstream (HP/FT) datasct.
Figure 7: Dataset for toy experiments. We full MNIST dataset pretrain the network. The downstream dataset is a subset of MNIST (only 0, and 2) applying random rotation and zoom-in.
B.2 PRACTICAL SETTINGS CLASSIFICATION
Besides the simple MNIST and MLP, we also verify our analysis in many real settings. To demon- strate the generality of findings, we conduct experiments image classification, image seg-
16

---

Published as a conference paper at ICLR 2023
real
STAALDONN
sketch
quick
airplane
alarm-clock
angle
apple
banana
basketball
bicycle
Figure 8: DomainNet dataset. Only three domains are used: 'real' is similar INIK, 'sketch' is less sensitive to color and background, "quick" only contains some lines, which is quite different from INIK.
Basic setting:
Partial Backbone:
Bakkia head
B2
Loss
B1
B2
B3
Deal
Loss
ResNet- Backbone
Baakktt-
Image Segmentation:
SimCLR:
Projection
X
ResNet Backbone
Head
Upsample
ASPP
COnv1
Conv2
X
B1
B2
83
B4
Projection
X2
ResNet-Backbone
Head
BYOL:
Projection Head
Prediction Head
X;
Backbone
Backbone
Loss
Loss
Projection
Sample
X2
Backbone
Head
Sg(r()
Memory bank
Figure 9: Introduction of models used in this paper.
mentation, and molecular graph classification tasks. The qualitative trend is quite consistent across different tasks and networks.
Model Structure:
Specifically, we consider ResNet18/34/50 for the image classification task. They all have blocks, each containing layers, as illustrated Figure 9. For the linear task head (or an MLP head), we directly concatenate linear layer (or an MLP layer) to the backbone. For the partial backbone method, several layers in the last block (e.g., L4.3, L4.2, L4.1) might be reinitialized and merged to the task head. We also provide what is the backbone (all the blue squares) when self-supervised- learning is considered, e.g., SimCLR, BYOL, and MoCo. short, after pre-training the backbones under different PT tasks might have the same structure.
Prefraining tasks:
We pretrain the ResNet CIFAR 10 using common recipe (10-2 learning rate with cosine sched- uler (He et al., 2019), 5 * 10 weight decay, simple augmentations like random flipping and crop- ping, etc.). For the model pretrained on ImageNet, open-source implementations. Due to time and computing resource limitations, for vision tasks, we only consider MoCo (He et al., 2020), Byol (Grill al., 2020), SimCLR (Chen et al., 2020a), and
directly download the checkpoints from
17

---

Published as a conference paper at ICLR 2023
supervised classification tasks for pretraining, For the graph dataset case, consider a fundamen- tal 5-layers graph convolutional network (GCN, (Kipf & Welling. 2017)). the rrper-parametess and other detailed settings can be found in our github repo. The datasets applied in this paper are listed in Table 4. Note that the backhone of image segmentation is the same as that in image classification. The segmentation head is more complex, which will be discussed later.
Hyper-parameters for HP and FT:
In this paper, experiments with the same setting in different figures tables share the same set of hyper-parameters. Generally, for all the experiments, the batch size is 128, hidden layer width is 256 (in the MIPP head case). The input image size for CIFAR pretrained model is 32 x 32 while that for the INIK pretrained model is 224 x 224. Hence in the corresponding experiments, we will first resize the input samples and then apply random cropping and random flipping augmentations. For the HP phase, the r² is selected based on the validation performance, but the detailed learning rate and maximum might be different under different settings (as the dataset size are different). For the FT phase, we use standard SGD with momentum (3 0.9). The default learning rate is 10 and cosine scheduler is applied (the maximum FT epoch also varies for different downstream datasets). Note that we will early stop the FT process, hence the maximum FT epoch doesn't 't influence the reported performance a lot. For those dataset-dependent
apper-paramtters, we summarize them as:
For the insufficient pretrained backbone cases, i.e., Res50-C10(Sup), Res18-C10(Sup), in Figure 4, Table and other related experiments in the appendix, we set (00), HP learning rate is 3 * 10-2, maximum FT epochs is 200 (usually converge less than 100 epochs), with usually a larger initial FT learning rate (e.g.. 3 10-3);
For the INIK pretrained backbone (no matter by supervisory training, MoCo, Byol or Sim- CLR), which can be found in Figure 1, 4, Table and many figures in the appendix, we usually set r E [0, 200]. HP learning rate as 10 10-2, maximum FT epochs as 200, and a relative small FT learning rate (e.g.. 3 10 10-4);
For any experiments whose downstream tasks are Flowers, Cars (the dataset is small while number of classes is large), we consider increase the HP learning rate 5 10-2, and the maximum FT epochs to 1000 (usually converge less than 200 epochs);
For any experiments whose downstream task is CIFAR (the dataset is bigger than STL), we consider to increase the HP learning rate to 5 * 10 10-3, and the maximum FT epochs to 100 (usually converge less than 50 epochs);
In summary, a general principle for these hyper-paramett selections is that the maximum r should make the HP training accuracy converge (increase slowly for several consecutive epochs), while the finetuning epochs should make the training accuracy converge when т = O. For example, the downstream task is CIFAR1O, which has 50,000 training samples and only 10 classes, finetune 20 epochs is enough. But CIFAR which also has 50,000 training samples but with more classes, 50 finetuning epochs required (we set the maximum FT epochs 100 for both CIFARI0 and CIFAR 100). For Flowers 102, which only has 1,9 training samples but 102 classes, we set the maximum FT epochs as 1,000 epochs. Anyway, find the proposed trend is quite robust to these hyper-parameters. The detailed settings can be found in our code base.
B.3 PRACTICAL SETTINGS - IMAGE SEGMENTATION
In addition to the image classification tasks, we also conduct experiments on image segmentation tasks to verify the robustness of our analysis on various tasks. An image segmentation task is task where a model is trained to make class predictions at each pixel of an image given an input image. We train and test segmentation models on PASCAL VOC (Everingham et al., 2015) dataset, one of the most popular image segmentation datasets. An example of input image and segmentation label provided in Figure 10.
Model Structure:
For image segmentation tasks, we employ D3eplabV3 (Chen al., 2017a) with ResNet50 backbone. As with the classification tasks, ResNet50 consists of 4 blocks, each of which contains layers. But unlike the classification tasks, kernel strides for convolutional layers adjust to have
18

---

Published as a conference paper at ICLR 2023
train
test
class
Comments
MINST
60,000
10,000
10
Toy downstream use a subset, as Figure 7.
STL10
5,000
8,000
10
Coates et al. (2011)
CIEARIO
50,000
10,000
10
Krizhevsky et al. (2009)
CIFARI00
50,000
10,000
100
Krizhevsky et al. (2009)
Flowers 102
6,149
2,040
102
Nilsback Zisserman (2008)
StanfordCars
8.144
8,041
196
Krause et al. (2013)
Dom-real
<20,000
<4,000
200 (345)
A subset. original 345 classes and
samples.
Dom-sketch
<20,000
<4,000
200 (345)
Less sensitive color and texture.
Dom-quick
<20,000
<4,000
200 (345)
Only contains simple curves. (Peng et al., 2019)
ImageNet-IK
1,281,167
1000
Deng et al. (2009)
ogbg-moltox21
6,272
1,742
12 regression
Wu ct al. (2018)
ogbg-molhiv
32,901
8,226
Hu ct al. (2020)
ogbg-molpeba 437,929
128
Hu ct al. (2020)
Table 4: Datasets (vision and molecular graph) used experiments.
(a) An input image.
(b) A segmentation lahel.
Figure 10: An example of an image and label pair in PASCAL VOC dataset.
the higher spatial resolution for the output features of the backbone; the spatial resolution of the features 7 x 7 for the classification tasks whereas it 17 x 17 for the segmentation tasks.
The features extracted from the ResNer50 backbone passed to segmentation head, which sists of Atrous Spatial Pyramid Pooling (ASPP) (Chen et al., 201 (b) layers followed by a couple of convolutional layers. ASPP is stack of five regular and dilated convolutional layers which provide the features with various size of receptive fields. The spatial resolution of the outputs of the seg- mentation head stays in 17 17, which are then upsampled to the original input image size through bilinear interpolation. Here, we use the whole segmentation head as the task head.
aaper-parameters for HP and FT: The ResNet50 backbone is pretrained on ImageNet dataset. We set т [0, 200), HP learning rate 0.3, maximum FT epochs as 200 (usually converge less than 50 epochs). Also, we use batch size 16, and SGD optimizer with momention (3 0.9) but without weight decay nor learning rate scheduler.
C MORE EXPERIMENTS
C.I VERIFICATION ON THE CHANGE OF
In Equation (2). we decompose the learning dynamics of z, into three parts: relatively stable kernel, direction, and energy. Based on this, we can expect larger adaptation when the initial energy large. Furthermore, to get more precise description of 2, 's adaptation, we analyze a simple overparameterized linear model in Appendix D. We show that as the initial prediction gap (i.e., energy) increases, the resulting Z, will first stretched with a small direction change (compared with the original zo). If the initial energy keeps increasing, the resulting might be more and more dissimilar to the original one. We use four distance related quantity to describe this trend, and verify it under many different settings. In short, the trends are: 1.) more energy leads to larger ||z, zu l and smaller cos 2n); 2.) z Zn and ||z, 11 has a quadratic shape when energy decrease. Figure provides example of how z; changes when different chosen.
19

---

Published as a conference paper at ICLR 2023
Rondom
HP 512 spors
HP 50000 apechs
Classi
Cless
Clesso
Class1
Cless]
15-
Cless1
13
Class2
Cless2
33
Cless2
-12
Figure 11: Change of during FT under toy setting. Using transparent square markers, we first draw the 2-D projections (the first components in PCA) of 100 randomly selected =! Then keeping the eigen-directions of this PCA, we project ((") of different and encode by transparency. The converged is represented by a cross marker with full transparency. clear that FT using
ynd makes Z change a lot, while 50000 doesn't change too much.
hhp
cos za?
Zt
Feature Manifold
8, zolli
Zo
Figure 12: Explanation of why small change of z Zo. cos Zo) can influence lot.
Before diving deep into these examples, we provide some general commands. First, we not expect the experimental results to align perfectly with the examples in Figure 3, because too many designs (like dropout, data augmentation, SGD noise, locking in strange local minima, learning rate schedule, etc.) can influence the shape of the curve, hence we believe observing specific trends in many cases is already a strong support for our analysis. Second, one might be curious about why in some cases, the ranges between z, Zo and cos(z;, Zn) are small. Hence in Figure 12, we provide an illustration of why this happens: as the dot product measures two long vectors (remember they have the same origin), small change of z Zo can make big difference. A large range of z zo and cos(z,, Zo) ) is a sign that z; already moves to another basin, which is the strong-case in Figure 3. Third, in Appendix D, the change experiments, we use T as our x-axis and consider a cross-entropy loss. Combining with the NTK approximation applied in this ideal model, the experimental trends might not be exactly the same with the theoretical ones.
is linked qo ey under an MSE loss, while in these
Then, in Figure 13, Figure 14 and Figure 15, we demonstrate the trend when the model pretrained on different datasets, on different tasks, and when the downstream tasks are different, respectively. Besides the general trends, i.e., decreasing ||z, 2oliž, increasing cos(z,, Za), quadratic z, Zo and ||z we can observe another interesting phenomenon when the model is pretrained using different tasks. See Figure 14, in Byol and SimCLR, sometimes we might get a very big z Zo (together with very small cos(z;, (z. 2). which makes it hard to observe any quadratic trend in some metries. But in the supervised pretraining case (no matter what dataset we use in pretrain and downstream tasks), never observe such phenomenon the ||z: zo 2o always changes in relatively small range. We speculate that in the first several updates during head probing, the features first adapt to the downstream task (i.e., classification), then gradually adapt the downstream dataset distribution (e.g., STL, Flowers, etc.). So a randomly initialized head might influence more when the pretrain task is different from the downstream one. The detailed mechanism of adaptation of task and data distribution might be more complex than we expect, so we left this for the future work.
C.2 INFLUENCE OF THE BACKBONE AND THE TASK-HEAD CAPACITY
In Section 4.3, we mention a case where the task head is more complex. For example, the head can be a two-layers MLP rather than linear layer, as our backbone. However, these methods and tricks are quite complex and might not necessarily enhance the performance, hence put some results and discussions here. Remember that
only copy part of the pretrained model
20

---

Published as a conference paper at ICLR 2023
2t20 norm
2120 dot
21, norm
2120 cos
energy
Clar10
2.75
aCc
1-gap
15
UP
10P
10
121
2P
30
17
10 enlergy
15.0
100
150
6.6
140
26.5
1-gap
130
26.
2tz0 norm
ztz0 dat
zt. horm
21.0 CDS
energy
mol GPN
BCC
Figure 13: Adaptation of z, when model is pretrained on different datasets.
analysis of how energy influences the features adaptation is still valid in these cases, as illustrated by the distance related metrics the tables.
From Table 5, 6, 7, and 8, we can draw the following three conclusions. First, the two-layers MLP design will surely increase the training accuracy after head probing, and hence make the features adapt less (but this might not enhance the downstream performance). Second, we compare the downstream performance when some layers of the backbone are reinitialized as part the task head. The title "+L4.3" means we take the last layer (i.e., the 3-rd layer) of the last block (i.e., the 4-th block) in a ResNet50 out, and treat it as part of the task head. The title "+L4.2" and "+L4.1" means continue treating the 2-nd or the 1-st layer as the backbone. Under such setting, the "+L..11" case will inherit the least amount of information from the pretrained model, and at the same time, have the biggest task head. For the results in these tables, we train the head until convergence in head probing phase, then fine tune the whole network together. Hence we see the "HP-train-acee value all increase roughly 100%. However, the best setting differs when different downstream datasets are considered. When the pretrained features are good (i.e., trained using INI we see the Domain-real dataset needs less adaptation and usually performs the best in the baseline case (i.e., copy all parameters from the pre-trained model). But Domain-quick dataset, which only contains some black-and-white lines, prefers the "+L4.1" setting. In other words, preserving the features captured by the earlier layers of the pretrained model is beneficial. When the pretrained features bad, like the CIFAR- preer--petraining case in Table 7, all these datasets prefer throwing away the later layers.
In summary, although the correlation between features adaptation and downstream task performance under different settings is quite complicated, our analysis of energy still explain some phenom- ena well.
Real
Sketch
Quick
Real
Sketch
Quick
HP-train-ace
98.994
63.369
48.555
100
93.447
63.75
Linear-head Bascline
cos(2 Z; Zo)
0.0315
0.2275
0.3506
2-MLP
0.0149
0.2032
0.3375
||z, - zol
17.868
22.246
27.954
Head
17.694
21.171
27.634
val-ace
54.839
37.789
61.416
35,446
59.199
+L4.3
val-ace
53.629
35,966
61.568
HP-train-ace
99.492
99.072
93.975
+L4,2
val-ace
53.226
36.69
61.391
tiP-train-acc
99.033
99.336
98. 18.193
+L4.1
val-ace
33.856
33.446
39.199
HP-train-ace
99.863
93.447
63.75
Table 5: Results on ResNet50 pertrained on Domain- Real. The pretrained validation accuracy is only 54.612 while the training accuracy is 100. Maybe because the NomainNet-Reas dataset only contains less than 20k samples.
21

---

Published as a conference paper at ICLR 2023
energg
162
6yo
1-g89
32'
14"
2tz0 dot
2tz0
14
BV0
1875
1.25
1-189
2tz0 norm
2120 dot
25, 20_n
2120 cc05
energy
0.7
06
aCc
1-gap
33"
10"
321
10
12'
15P
norm
zt norm
energy
12.5
10.3
acc
1-gap
152
2tz0
10
2t20 dot
25, n0rm
2120 7o6
10
enargy
SimC
322
200
1630
acc
04
1-gap
15"
2tzd horm
10
10P
2t20 do!
10
Aorm
39" 2120 co5
energgy
12'
153
1.0
SimCUR
100
a0c
1-39p
2120 norm
2120 dot
28. nomm
2120 C.0
energy
Stepervieed
acc
738
1-gap
11
109
127
17
32"
321
emergy
1.55
1-gap
Figure 14: Adaptation of z when model is pretrained using different tasks.
Sketch
Quick
Real
Sketch
Quick
HP-train-ace
60.039
39.092
88 96.965
79.58
76.445
Linear-head Baseline
cos(2e, Zo)
0.3492
0.4181
0.5034
2-MLP
0.3296
0.3672
0.4923
|| zoll
9.72
9.287
Head
9,104
9.209
9.874
val-acc
38.346
45.023
61,593
58.77
43.171
60.307
+L4.3
val-ace
$9.929
42,882
39.60
69.602
HIP IIP-train-ae
100
100
98.252
+L4.2
val-ace
60.837
44.734
61.542
tua-train-cce
100
100
100
+14.1
val-ace
61.164
45.775
62.626
HP tra-tran-
100
100
100
Table 6: Results ResNet34 pertrained on CIFAR-100 in classification task.
D ANALYZE LINEAR OVERPAR PROBLEM
D. FORMALIZE THE CHANGE OF REPRESENTATIONS
Section provides some intuitive explanations of how feature extractor f(x; B) changes given dif- ferent g(z;v) v), which are well supported by the experimental results. To provide more insights,
22

---

Published as a conference paper at ICLR 2023
2120 nomm
2t20 dot
2t. norm
2t20 GOS
emergy
33
aCc
190
1-gap
151
energy
Ciaa1100
0.00
0.0
1-00p
2120 foom
3ot
norm
2:20 2os
energy
Dom-raa
650
acc
20"
10
10'
37'
100
10'
18
10'
10'
12'
30'
10"
12'
11'
2t20, norm
2520 2t20_dot 80t
25. norm
2t20 c.05
energy
Dom-quick
20.3
acc
1-gap
102
2120 nomi
10a
ztz0 dot
16
norm
332
2t20 30
23.75
Dom-sbetth
22.90
230
125
acc
21.00
200
675
1-gap
21.23
2t20
2120
2120
Energy
353
700
00
1-0ap
ztz0
2520 dot
morm
ztz0 cos
energy
12.5
acc
7.5
35
187
157
107
157
167
157
10P
10P
3.'
110
31"
30'
2t20 norm
2720 dot
25. morm
2t20 CO5
energy
14.3
a0C
2375
1-gap
130
107
167
175
12
32
100
100
Figure 15: Adaptation of z; when downstream tasks are different.
Real
Sketch
Quick
Real
Sketch
Quick
HP-train-ace
96.875
85,938
64.063
96.671
98.438
84.375
Lincar-head Baseline
cus( Rt; Zo)
0.1384
0.2807
0.4432
2-MLP
0.1544
0.2661
0.4147
||z| zol
16.569
18.005
21.869
Head
16.843
17.672
21.389
val-ace
85.685
72.51
69,531
85.031
69.651
66.683
+14.3
val-ace
84.929
69.734
68.422
trai-ran-
100
100
95.232
+L4.2
val-ace
83.342
68.547
68.7
HP traitrain-cce
100
100
98.087
+14.1
val-ace
81.074
69.155
68.2
68.246
HP HP-ttrin-ace
100
100
98.842
Table 7: Results on ResNet50 pertrained on ImageNet- 1K in classification task.
analyze the change of features (i.e., in simplified overparameterriztion problem, i.e., the one
provided in Section 3 and Kumar et al. (2022). Under some mild assumptions and approxima- tions, provide overview of how the norm and direction of changes under different choice of Nn. We show that the vo satisfying 4, = Y or qo = Y are two critical points in feature adaptation.
We first rewrite Equation (1) in non-matrix form:
23

---

Published as a conference paper at ICLR 2023
Real
Sketch
Quick
Real
Sketch
Quick
HP-train-cce
92.676
78.213
64.307
99.121
94.883
82.354
Linear-head Baseline
cos Zr, zo)
0.1269
0.1422
0.1934
2-MLP
0.1098
0.1104
0.1522
l|z zoll
5.247
4.752
4.361
Head
4,942
4.206
3.868
val-ace
78.075
61,545
59.703
78.453
63.773
63.609
+L4.3
val-ace
78.528
65.683
67.011
HP train-acc
100
97.666
+1.,22
val-ace
78.427
65.336
67.087
HP-train-acc
100
100
100
+L4.1
val-acc
76.689
65.017
68.07
HP-train-acc
100
100
100
Table 8: Results on a ResNet50 pertrained on ImageNet-1K IK in SimCLR task.
N
1
LB. = N
Bx (n) =112.
(24)
12
where E I as we are considering a regression problem (or classification problem using MSE loss). We the subscript represent the time step, e.g., 96 Zo. Vo and Bn output, feature, head parameters and backbone parameters before finetuning. Similarly, 9: Z,, and B, are the corresponding values after finetuned steps. Note that q(X) and z(X) are functions of N input samples X E & but we omit it for simplicity. We use lowercase letters to represent the n- element of the vector. Before discussing specific element, we will clarify whether we are discussing the initialized case the finetuned case. For example, specifying 9,, we use = fxx)) (f())), to represent the prediction of x,, after finetuning.
Remember the goal of this paper is finding suitable way to select task head, i.e., Vo. given the pretrained feature extractor, i.e., Bn- Depending on the discrepancy between the pretraining task and the downstream task, might expect z; change differently after finetuning. In other words, we care about the expected change z, compared to Zg. i.c., ExzD [d(z, zo)], where D is the data distribution of the downstream task. Depending what distance measurement (i.e., d(:,)) choose, there are three different metrics:
Euclidean: deve A EX-D [||z zo 2
Dot product: dior A ExnD
[z zo]
Cosine: dees & Ex-D
z za 2
We start from the Euclidean case:
druc Ex
(z: - 2u) T() (z: zu]]
ExND
|2, zi 2 z
- - z
z zo]
EXND
|z, z, 2z,z, 2o0
EX-D
x B, B,x 2x 2x B B.x B. Bax
Ex
tr ( (B. B. 2B B, B Bo)x)
ExWD
(r. B, 2B B, + B B xx
EXWD
(r( B. 2B B, + B B xx
< ExDD
tr(B. B 2B B, + B Bb) M
(B. B,) 21r B B,) + hr B Bp)
[X-D[M]
(25)
where M A tr(xxT] T) or M A ||xx llap).
Similarly, get the expressions of dant and ||z, which are building blocks of dens:
24

---

Published as a conference paper at ICLR 2023
ExwD
[2] zo] = M MM (rr(B ( B.)
(26)
ExnD [|z]|]]] M tr B,)
(27)
D.2 CRITICAL POINTS
Our goal is to find a good initialized task head, i.e., Vo. which can lead to better downstream per- formance. Instead of directly linking Wo the expected risk, which is common practice for gen- eralization analysis 5, we consider this problem from another indirect way. Specifically, we assume the SGD algorithm with appropriate regularization in FT stage can find good optimum for the learning task. What we care more about is whether the features learned from the pretraining stage adapts well to the downstream task. We believe that if the features are properly adapted to the new environment, the model has more potential to generalize better. Hence in this part, considering the aforementioned three distance metrics as the target functions, we formalize how Vo influence them.
From Equation (25), (26) and (27), we find that the behavior of tr(B, B,) and tr(B, B,) are the keys. In these two terms, Bo is given and cannot change, while B, is determined by the choice of Vo- One way to link these two quantities is using Lemma A.4 in Kumar et al. (2022) or theorem 2.2 in Du et al. (2018):
(28) however, the expression of v. is still hard to obtain (but collecting and visualizing is much more cheaper than B,). We left this direction for our future work. In this paper, we analyze the problem in the NTK regime (this is the main assumption of our analysis, which can cause discrepancies between the theory and experiments).
Vo B, v.v. B,B, Vt,
Closed-form of parameters under NTK approximation:
To get more insights, we approximate the behavior of this model in the NTK regime, which the converged parameters can be analytically calculated. Specifically, be applying Equation (8) in Lee et al. (2019) and assuming t - oc, we can have:
0, = On - (Voq,) (4
(29)
where 9, E R-Nx1 is the model's prediction on training samples, i.e., X, and O,, 0o E R(d+1)-Ax1
the stacked parameters. Without loss of generality, the first d * h parameters in & come from B and the last h parameters come from v.
The Ko- Vequ (Voq.)T E NxN here is the empirical NTK on X. Specifically, by stacking the paramters, we can calculate each elements in this kernel as:
<(x, x)) = T
B Bx +
ΣΣ (uxz))ua)))
==
=x BB B Bx + Σ 0
x B MM2 Iaxa)x',
(30)
where v and 2; the i-th element in and x respectively. Then, the matrix form of emperical NTK
AC, = XB Bu + ||voll2 XT.
With the help of Equation (29), we can get the closed-form expression of b, vec(B,) (i.e., the vectorization of matrix B,):
b. = bo - (Vb4)) Ko (4. Y),,
(31)
In Equation (31), we know qu = XB, Vg- The term Vb9n also depends on As controlling qo> i.e., the model's prediction on training samples, easier practice (we can directly observe the training loss or training accuracy), we will consider qo as the optimizing variable in the rest of the paper. The following lemma can link Vpg to 4
But is hard get tight and informative bounds in deep learning, especially in such practical scenario.
25

---

Published as a conference paper at ICLR 2023
Lemma 1. For bu R and Vb9 E R Nxhed, have Vb9o bu 40-
Proof. We check that equation elementwise. For the и-th element in the RHS, we have:
Qn = x B vo
d
uı B )is
1= j=1
Tyyy (Bo)a.:
(32)
==
where 9n is the nith element 9o Fy is the i-th element of x,, and Uy is the j-th element of Vo-
For the LHS, the n-th value e, Vb4. by. where e a one-hot row vector selecting the n-th row
of Vbqu- From the definition, know e,, Voq veo [21v0 22 ... TaVo ], which a long row vector. Then, the n-th element in the LHS should be:
e, Vb40 bo vec(vox, bo
had
Σ (vec(vox)) ))(bb))
Σ 1/ (Ba)s.
On
(33)
where the last equation holds the rule of stacking elements in matrix. As each elements of the two sides are equal, the lemma holds.
Critical points tr(B, B:) and tr(B, B:): Lemma 2. q Y is maximum of tr (B, B,).
Proof. By definition, have:
Ir (B, B,) bo b
by (bo - (Vbq,) T (qo Y))
by bo (Voq, bo) Kr - Y)
const. qo Ci
(34)
where we call C KCT1 for convenience. By taking first derivative to qo and let is equal zero, we know qi LY a critical point. As the second derivative is -2C1 and C1 positive definite, this critical point is maximum.
Lemma 3. q = aY,a (0,0.5) is critical point (usually maximum)) of (r, B,). Here (INxN + (C2 -CI) Y, where C2 Kr "okKo and Ro XB BoX (V..))(Vo4))) is the NTK when fixing the parameters of the backhone.
26

---

Published as a conference paper at ICLR 2023
1|z - zol12
T Zi Zo
1|ze11
1l 1 92
Pali 0
Ib. 15
[BbilB
Y/2
Figure 16: Illustrations of different metrics if they are 1-D quadratic functions. Here = aY and d2 YT YTY. The x-axis represents the choice of qn where aY an example the critical point of 1. The black dots are the critical points.
Proof. Similar Lemma we can write tr tr(B, B,) as b, b, and substitute Equation (31):
tr(B, B,) = b, b,
(bo- (bbgg) NT Y)) (bo - (Vbqu) ( (a - -
by bo 2(Vb9n bo) _o
Ko (q, Y) (4n - r) (oo)))
Y)
bo bo KG (go Y) + (4o K (q0
by bo (i - Y) + (4. - C2(40 YY)
(bo bo CCY) + q. (2C) - 2C2) + 40 (C2 2C1)4.
const. q. (2C) - 2C2) Y + q. (C2 201)a.
(35)
By taking first derivative and letting it equal zero, assuming (C2 - 2C1) is invertible, we know q = (INxN + (C2- 201)-1Ca is critical point. The second derivative is 2C² - 4C1, which is usually positive definite in settings (explain later).
To get more insights, we can look deeper into the critical point mentioned in Lemma 3. Following the definition of C1 and C22 we can have:
C2 - 201 No - RoKo 2K
(KG Ko N
(36)
where No- Ko is the key of understanding this term.
The exact form of this expression is hard to obtain, but as all the NTK or covariance matrices mentioned here are sysmetric, we can compare the trace of them get some insights. By def- inition, (r(Ko) tr(BB, TT
lr(B, B,XT X). each dimensions the data sam- ples are independent, then x x Laxas and tr(Ko) (b b,) = |bbll.. Then similarly, tr (rC,)) ||bol|2 + | 22
Thus the behavior of term Ki Kn can be described by b -NNNN:
||b ?
if |boll f >> | which might happen the backbone contains more parameters than
the head, Ko Ko INXN.
if | Kg Ko R INNN.
< 1/v 12. which might happen if we split the network in the earlier layer,
But in either case, the negative of the second derivative of tr Bt), i..., Equation (36), is likely to be positive definite, which means the critical point in Lemma is usually a maximum. Another interesting fact of Lemma is q under the aforementioned two extreme conditions. After some
calculation, can verify that q 0 if | > vull3 and q !Y if | < ||v.1..
Critical points of denes dast and deasi
27

---

Published as a conference paper at ICLR 2023
Recall the definitions of different distance metrics care about. First, ddot is proportional to z Zu, hence its shape would like the second panel in Figure 16. For deuc. we know it is proportional to
tr(B, B₃) - 2tr(B, B₂) + tr(B,, Bo). Substituting results in Equation (34) and Equation (35), we
have:
denc Ir(B, B,) 2/r(B B B,) + const.
4. (2C1 2C2) 40 (C2 201)q. + Ci YY const.
24h C2V + +4 C240 + const.
(37)
Obviously. 4 = Y is the minimum of denes as depicted in the first panel in Figure 16. The shape of deas hard to obtain. However in the last panel of Figure 16, we demonstrate ||z 12.12, its denominator, to assist our further analysis.
D.3 LEARNING DYNAMICS OF THE REPRESENTATION
Enic
dene
ddot
12
dens
Y Y Y Y + aY
1
1
(L)
1
aY 0
Table 9: Question marks in the last column means we cannot accurately predict its change. But it usually decreases in experiments.
In this subsection, we will put everything together to provide an overview of how changes com- pared with Zn. Remember when 9, Y. the gradient of Bn is zero, hence z, Zo (Kumar al., 2022). When q moves linearly from Y to O, we have the following three phases (see Table 9):
Y y: z, lengthen its norm with slightly change in the direction, hence dast also
increase;
Y + aY: the norm z; keeps increasing, but its direction drastically changes in this phase, which makes daos decrease;
aY O: the norm of z begins decrease and the angle between z, and Zy keeps increas- ing, which makes changes a lot.
28


Reviews:
[{"review_id": "t1gBQnMY_OX", "reviewer": "Reviewer_xD14", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus on the choice of task head in fine tuning, and how the task head controls feature adaptation and downstream model quality. \n\nThe gradient was decomposed as the product between a direction and an energy, and analysis was done on the effect of the energy vs feature adaptation after FT step.\n\nThe paper's extensive experiments would have guidance to how to use a few techniques to get better model, such as early stopping at HP, and label smoothing at HP.", "summary_of_the_review": "Recommend as accept but marginally above threshold. If the authors could help me to appreciate better the paper's empirical value, then it would be awesome", "strength_and_weaknesses": "Strength\n1) Extensive study for numerical experiments.\n2) The energy decomposition seems novel.\n3) The problem of how to make pretraining / fine tuning work better is very core, and thus the paper may seem of great interest to people in the area.\n\nQuestion\nThe overall writeup seems challenge for a regular non-DL-theory practitioner to follow, even with the Section 4 on real world problems. Can you please make the implication / take away message a little more explicit for model training of real-world problem?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hG21_-_6Zt", "reviewer": "Reviewer_3J3Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the two-stage transfer learning approach of head tuning first (a generalization of linear probing where the head can be nonlinear), followed by finetuning both the backbone and continuing to finetune the head. Their work builds upon the setup of Kumar et al, but they claim to relax the assumption made in that work that the pretrained model is optimal for the downstream task. \n\nThey decompose the training update to the features during finetuning and show theoretically that when the adaptation ‘energy’ increases (meaning that the predictions on the downstream task differ from the correct labels), the final representations (obtained after finetuning) change more. So, intuitively, controlling that energy is one way of controlling how much adaptation is performed on the features. One way to increase the energy is by doing less head probing before the fine-tuning phase (this way the head will be less suited to the downstream task at the start of the finetuning phase, so more adaptation occurs). Since the number of head probing epochs is difficult to tune, they propose label smoothing as an additional knob that can affect energy increase and also explore the effect of the capacity of the head. They show in several empirical scenarios results consistent with their intuitions and theoretical results. \n", "summary_of_the_review": "This paper analyzes the effect of different head designs on the adaptation that occurs in the learned features after the finetuning phase. Their analysis is novel to the best of my knowledge and some findings are interesting. However, I found that the writing and presentation can be improved (see clarity section above) and I’m also unsure about the usefulness of practical takeaways from the paper (see above detailed comments). For this reason, I’m doubtful if the size of the contribution meets the bar for acceptance.\n\n\n===============================\n\nAfter rebuttal: The new version of the paper is clearer and better structured. I particularly liked the User Guide section. I also like the discussion around difficulty in making these design choices in practice, the risks associated with some decisions (e.g. label smoothing, etc) and the inclusion of recommendations for making them based on increasingly-risky interventions. Based on this, I increased my score from a 5 to a 6.", "strength_and_weaknesses": "Strengths\n========\n- To the best of my knowledge, this analysis of the influence of ‘energy’ in the head on the changes made to the features during FT is novel and it is interesting.\n\n- The finding that adding label smoothing can make the HP-FT procedure more robust to the number of epochs for HP is an interesting one.\n\nWeaknesses\n==========\n- I found that the paper has clarity issues (see below)\n\n- Does label smoothing always help? Are there cases where this addition can degrade performance? (I see at least one such entry in Table 1, for Flowers dataset, Sup-C10). The authors don’t discuss the potential dangers of using it / weaknesses of this approach.\n\n- I’m not sure what the practical take-aways are (see below)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e4z8KCk7Wf", "reviewer": "Reviewer_kiZg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how the choice of task heads influence the pretrained features z’s adaptation and hence influence the downstream performance.  The authors explain feature adaptation by decomposing the learning dynamics and identify the energy and direction terms matter most. They find that the training accuracy and loss at the beginning of finetuning determines the energy for the feature’s adaptation. They also identify a trend in the effect of changes in this initial energy on the resulting features after fine-tuning, which helps to design an appropriate fine-tuning procedure. Finally they provide examples of how to enhance downstream performance in practice. They suggest paying more attention to how much and in what ways we want the features to adapt.\n", "summary_of_the_review": "The paper presents an analysis of the fine-tuning behavior with head probing in different settings. However, it is not clear whether the analysis can lead to a concrete approach or recommendation for controlling feature adaption.", "strength_and_weaknesses": "Strength\n- The paper is well structured and written.\n- The paper characterizes the change of the backbone and head during finetuning, which provides a better understanding about the evolving process of feature adaptation during fine-tuning.\n\nWeakness\n- The authors made observations on the quadratic trend of $z_t^Tz_0$ and $\\|z_0\\|_2^2$, however, it is not clear how this trend can help adapt features on the downstream task.\n- The authors claimed that the change of the NTK term is slow during FT without further examination. Is it possible that the NTK term also matters when the source and target data are different and the change could be large?\n- How much to adapt may also depends on the dataset size, which were not considered.\n- There are no recommendations for how to select the backbone depth or manipulate the head’s capacity. The selection of $\\tau$ is also not clear.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WwvzstztBaT", "reviewer": "Reviewer_6K2Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the role of choice of task head, i.e. head tuning (HT), in the downstream performance of fine-tuning (FT) from a pre-trained model.\nRecent work has suggested that it is a good idea to first perform head tune on a downstream task and then use that head for fine-tuning (so that features do not move too much), rather than fine-tuning from a randomly initialized head (where features move a lot).\nThis paper also considers an intermediate case of stopping early during head tuning (to mildly adapt features) and finds empirically that in many cases the optimal stopping criterion is somewhere in the middle.\n\nThe paper delves deeper into the amount of feature adaptation w.r.t. number of steps $\\tau$ of HT performed initially, and find some interesting trends like norm of change in features $\\|z\\_t - z\\_0\\|$ is monotonically decreasing w.r.t. $\\tau$ while the norm of the features $\\|z\\_t\\|$ has a quadratic behavior. Theory on an overparametrized linear setting (NTK regime) that tries to explain these trends.\nThe analysis of 1 step of feature dynamics highlights an “energy” term that strongly determines the amount of feature adaptation: energy is low when initial HT on top of pre-trained features already has good accuracy and vice versa.\n\n\nUsing these findings and analysis, the paper identifies label smoothing during HT as a way to control the “energy” and allow feature adaptation. This often (but not always) performs comparably in practice to the optimal $\\tau$ (number of HT steps) which is not easy to find.", "summary_of_the_review": "Overall I find the contributions of this paper in understanding the role of task head in fine-tuning to be positive. However due to the presentation concerns I have only assigned a score of weak accept.", "strength_and_weaknesses": "**Strengths**\n\n- Overall the paper presents an interesting analysis of the role of task head in the fine-tuning performance. The following observations are interesting: the optimal stopping criterion is somewhere in the middle and that label-smoothing during head-tuning can help get closer to the performance of this optimal stopping criterion.\n\n- The paper also performs deeper analysis into some of these phenomena, e.g. analyzing the level of feature adaptation as a function of number of initial HT steps $\\tau$ or \"energy\". Many of the hypotheses are verified empirically even if not theoretically proved.\n\n\n**Weaknesses**\n\n- A lot of the analysis is hand wavy. For e.g. the effect of the task head on the FT features is studied by just looking at just 1 step of gradient descent. The notion of \"energy\" derived from this is also vague since it is technically a vector. Although empirically heuristics like the training accuracy or probability gap seem to be a good proxy for \"energy\". The theoretical result for the linear model uses $\\alpha$ in $\\alpha Y$ as a proxy for \"energy\". Despite all of these, the analysis seems somewhat meaningful.\n\n- I think the presentation of the paper can be significantly improved. I found the paper hard to follow and it took me at least 2 passes to understand how the different points in the paper connect to each other. There is a constant switching from number of HT steps being important to the \"energy\" being crucial to feature adaptation being the key. Rather than presenting the analysis first, I think it might help to start with the empirical observation that the ideal choice of task head is to initially HT for some $\\tau^*$ number of steps before FT. Then the goal of the entire study could be to understand this phenomenon and try to achieve this optimal performance in a simple way (e.g. label-smoothing is a solution). Without such a concrete goal, I was lost many times about the purpose of different sections.\n\nOther comments/questions:\n- If my interpretation is correct, it is a bit misleading to suggest in Proposition 1 (informal) that the result is about the effect of $\\tau$, because the X-axis seems to be something different, $\\alpha$ in $\\alpha Y$\n- \"This kind of z can make the original overlapped features become separable after adapting to a similar manifold\" on page 5. Could you please elaborate what this means?\n- How was the label-smoothing coefficient of 0.9 picked? How much of an effect does this have?\n- The discussion in Section 3.3 about head capacity seems to be hand-wavy and a distraction, since most of the results seem to use a linear head", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NEmtWFR-s9h", "reviewer": "Reviewer_5UFU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As clearly stated in the title, the paper presents an extensive analysis of how to prepare (or train) a task head in a backbone + task head neural network configuration. The authors also make a lot of reference to the cited paper by Kumar et al., frequently pointing out the differences in their approach (2022). The authors analyze the behavior of features after model pre-training and during the head probing (HP) phase, in which only the parameters of the head are updated, and the fine-tuning (FT) of the entire network. The analysis is carried out both with a toy example and also with more general downstream real tasks. By decomposing the learning dynamics of the representation\nextracted by the backbone, they find the keys are the energy and direction terms, which highly correlate with the accuracy at the beginning of the finetuning process, then they show how to manipulate the energy (e.g., stop the Head Probing stage earlier, label smoothing only in Head Probing, non-linear head, etc.) to enhance the downstream performance.", "summary_of_the_review": "The paper could have been definitely better with a stronger experimental section (see my comments above.)\n\nPOST REBUTTAL:\nThe paper is now satisfying, the User Guide section is useful, many explanations have been provided.", "strength_and_weaknesses": "strenghts:\n\nThe paper is well written and the problem is clearly stated multiple times\n● Every consideration is presented both in a discussion and in a formal fashion\n● Several comparisons are made with other similar analyses, outlining the differences in the proposed approach\n● When realized (see the weaknesses), the experiments are extensive and well commented\n● The topic of the paper is very relevant, and such an analysis, if done comprehensively, can be very useful, theoretically and practically, to many researchers in building better models and learning procedures\n\nweaknesses\n\nSometimes the claims and observations of the authors seem a bit too excessive in comparison with the results actually obtained from the experiments. Some example are:\n○ “Although we do not analyze the relationship between the change of zt and out-of-distribution performance – this adds too many “moving parts” to the current setting – our analysis is also relevant to finding features which will work in new, unseen domains” in Section 4. I think that out of distribution analysis is necessary to get a more thorough idea about the interaction between HP and FT, as stated in the highly mentioned Kumar et al., 2022\n○ The paper lacks experiments regarding the relationship between backbone depth and head capacity. There are some examples in the appendix but they are too limited, considering that the use of non-linear and more complex heads (instead of the more basic linear probing) is one of the 3 practical suggestions given in the introduction\n● The experimental section lacks some information (that can often be found in the appendix, but that I consider essential for the fruition of the paper):\n○ A description, even short, of the used architectures (and how they are decomposed in backbone and head)\n○ A small description of the considered configurations, for example of the Byol and MoCo unsupervised cases\n○ In table 1 it is not clear to me what HP𝛕*-FT means. If it means the configuration with the best 𝛕, how 𝛕 was chosen?\n● Some images (Figure 8 and 11) in the appendix are cited in the paper. Since these are quite useful perhaps it would be appropriate to include them somehow in the main paper\n● I think the difference between the dataset used for pre-training and the dataset used for fine-tuning is a key element of such an analysis. This and what is meant by distance between tasks are elements that should be explored further in the main paper in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster