Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Authors: Akiyoshi Tomihari
Issei Sato
Venue: NeurIPS 2024 poster (2024)
Keywords: fine-tuning
transfer learning
neural tangent kernel
Primary area: other
TL;DR: We analyze fine-tuning strategy of LP-FT from NTK perspective and demonstrates its effectiveness for language models.

Abstract: The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (LP-FT), outperforms linear probing and FT alone. This holds true for both in-distribution (ID) and out-of-distribution (OOD) data. One key reason for its success is the preservation of pre-trained features, achieved by obtaining a near-optimal linear head during LP. However, despite the widespread use of large language models, there has been limited exploration of more complex architectures such as Transformers. In this paper, we analyze the training dynamics of LP-FT for classification tasks on the basis of the neural tangent kernel (NTK) theory. Our analysis decomposes the NTK matrix into two components. This decomposition highlights the importance of the linear head norm alongside the prediction accuracy at the start of the FT stage. We also observe a significant increase in the linear head norm during LP, which stems from training with the cross-entropy (CE) loss. This increase in the linear head norm effectively reduces changes in learned features. Furthermore, we find that this increased norm can adversely affect model calibration, which can be corrected using temperature scaling. Additionally, we extend our analysis with the NTK to the low-rank adaptation (LoRA) method and validate its effectiveness. Our experiments using a Transformer-based model on multiple natural language processing datasets confirm our theoretical analysis. Our study demonstrates the effectiveness of LP-FT for fine-tuning language models. Code is available at https://github.com/tom4649/lp-ft_ntk.

Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Akiyoshi Tomihari The University of Tokyo tomihari0g. ecc. u-tokyo. jp.
Issei Sato The University of Tokyo sato0g ecc. u-tokyo.. ac jP
Abstract
The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (L.--FT), outperforms linear probing and FT alone. This holds true for both in- distribution (ID) and out-of out-of-distribution (OOD) data. One key reason for its success is the preservation of pre-trained features, achieved by obtaining near-optimal linear head during LP. However, despite the widespread use of large language models, there has been limited exploration of more complex architectures such Transformers. In this paper, we analyze the training dynamics of LP-FT for classification tasks on the basis the neural tangent kernel (NTK) theory. Our analysis decomposes the NTK matrix into two components. This decomposition highlights the importance of the linear head norm alongside the prediction accuracy at the start of the FT stage. We also observe significant increase in the linear head norm during L.,, which stems from training with the cross-entropy (CE) loss. This increase in the linear head norm effectively reduces changes in learned features. Furthermore, we find that this increased norm can adversely affect model calibra- tion, which can be corrected using temperature scaling. Additionally, our analysis with the NTK to the low-rank adaptation (LoRA) method and validate its effectiveness. Our experiments using Transformer-basee model multiple natural language processing datasets confirm our theoretical analysis. Our study demonstrates the effectiveness of LP-FT for fine tuentnning language models. Code available at htttt:://// //gi gighht....... com/ www..ooommm/mom--o/eeoo
extend
Introduction
Fine-tuning pre-trained models for new tasks is common practice across various fields. However, simply fine-tuning the entire model can lead to overfitting on training data, which may negatively
impact generalization and cattoo-ddisrrbuution (OOD) performance (Li al. 2020 Lee et 2023
To address this, the two-stage approach known as linear probing then fine tuning (LP-FT) Kumar
al., 2022] has demonstrated high performance on both in-distribution (ID) and OOD data. Initially, linear probing (LP) optimizes only the linear head of the model, after which fine-tuning (FT) updates
the entire model, including the feature extractor and the linear head. This method has been extensively
analyzed and enhanced [Trivedi et al. 2023 Ren et al. 2023 Ha et al. 2024 Kirichenko et al.) 2023
The feature distortion theory, introduced by Kumar
[2022] explains the effectiveness of LP-FT on the basis of theoretical analysis with a two-layer linear model. This theory suggests that LP-FT minimizes changes to pre-trained features by starting FT with an already optimized linear head from LP. However, our understanding of LP-FT, particularly when applied to complex architectures such as Transformers (Vaswani al.) 2017]. remains incomplete, Thus, it is crucial further explore the training dynamics of LP-FT in more complex models than the two-layer linear model.
In this paper, apply the neural tangent kernel (NTK) theory [Jacot al. 2018 clarify the mechanisms underlying I.--T, focusing on the training dynamics of classification models. The NTK is a theoretical tool that analyzes training dynamics by applying first-order approximation
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to changes in the model outputs with respect to its parameters. Therefore, the NTK is suited for analyzing feature changes during FT dynamics (Wei al. 2022 Malladi et al. 2023]. Our analysis reveals that after I.,, both more accurate predictions and increased norms of the linear head compared to their initial values contribute to minimizing feature changes. We then identify a significant increase in the linear head norm during LP from the analysis of training with cross-entropy (CE) loss, which contributes to small feature changes in the FT stage. On the other hand, we found that this increase in the linear head norm can worsen calibration, causing predicted probabilities to deviate from actual probabilities, which can be corrected with temperature scaling [Guo et al. (0177] Furthermore, we extend our analysis based on the NTK to the low-rank adaptation (LoRA) method [Hu et al. 2022 parameter-efficien tne-ttning strategy, and validate its effectiveness.
Our contributions are summarized as follows:
We show that both accurate predictions and increased norms of the linear head during LP reduce feature changes in LP-FT within the NTK regime (Section 4 which is consistent with the feature distortion theory. (Corollary 4.3).
We find that norms of the linear head significantly affect the balance of the NTK matrix components and influence the training dynamics of FT (Proposition 4.1).
We also highlight that increased linear head norms can negatively affect model calibration, and this can be fixed with temperature scaling.
We extend our analysis based on the NTK to the LoRA method and provide theoretical validation of its efficacy (Proposition 4.4).
2 Related work
LP-FT FT and LP are well-established transfer learning techniques with extensive empirical and theoretical studies [Zhuang et al. 20200] Kornblith et al. 2019 Tripuraneni al.) 2020]. Kumar et al. [2022] analyzed the effectiveness of these techniques using a two-layer linear model. Then, they proposed LP-FT that is a combined approach of LP then FT. Building this study, subsequent
studies have explored LP-FT in more detail. Trivedi et al. [2023] investigated L.-FT through the lens of safety objectives. proposing modifications mitigate simplicity bias. |enn al. [2023) analyzed LP-FT from the perspective of the initial discrepancy between predicted and actual probabilities, emphasizing the importance of the number of probing epochs during LP. Ha al. [2024) further
improved LP-FT by aligning batch normalization layers with the target domain. Kirichenko et al.
2023) highlighted the challenge that models depend on spurious features and proposed last-layer retraining as a cost-effective strategy to improve model robustness.
Other FT methods Various FT strategies other than LP-FT have been proposed, including two-
stage approaches (Zhang et al.] 2020). regularization-basee techniques (Jiang et al., 2019). and
parameter-efficient fine-tuning methods [Houlsby al., 2019 He al.
2022
One prominent
example of a parameter-efficient method is LoRA, proposed by Hu et al.
[2022] This approach
draws inspiration from the concept of intrinsic dimensions (Aghajanyan al. [202 suggesting that
data can be effectively represented in a lower-dimensional space. Zeng and Lee [2024) explored the
expressive power of LoRA, and Jang et al. [2024) provided a theoretical analysis of its convergence
properties. However, challenges remain in parameter-effice FT methods, including potential
instability issues identified by Chen et al. (2022.
Neural tangent kernel (NTK) The NTK, which was first introduced by Jacot et al. [2018). has become a valuable tool for analyzing the training dynamics of neural networks. Studies by Lee et al. (2019) and Arora et al. [2019] used the NTK to gain insights into how networks learn. Building on this foundation, Wei et al. [202]] introduced the concept of the empirical NTK, which extends the application of NTK to FT scenarios. This approach replaces the randomly initialized parameters in the standard NTK with the parameters of the pre-trained models. Further expanding on the empirical NTK, Malladi et al. (2033) conducted a theoretical and experimental investigation and found that prompt-based fine-tuning exhibits behavior consistent with the predictions of the kernel framework. Jang et al. (2024) extended this perspective to analyze LoRA.

---

3 Preliminary
In this section, provide overview of the FT methods used in this paper, followed by brief explanation of the NTK.
LP-FT In standard FT, the parameters of the linear head, weight V and bias b, are initialized with random values. In contrast, in LP-FT, J LP is conducted before the FT stage, and the FT stage is started with the obtained parameters. The performance of LP-FT is higher than that of LP and FT on both ID and OOD data [Kumar et al. 2022]. The original LP-FT paper [Kumar et al., 2022] explains the reason behind it as the feature distortion theory: the success of LP-FT stems from the minimal feature changes because of starting the FT stage with the linear head parameters which are close to the optimal solution. We analyze the training process of LP-FT throughout this paper.
LoRA LoRA (Hu et al. 2022 introduces trainable rank decomposition matrices into each layer of the Transformer architecture. This approach, inspired by the concept of "intrinsic dimensions" from Aghajanyan et al. [202]]. constrains updates to pre- pre-trained weight matrices via low-rank decomposition. The update of a pre-trained weight matrix Wo E R1x* is approximated by W AW Wo + B BLORA A LaRA where BLORA E g and A L E R*** are the only matrices optimized during fine-tuning. Here, r min(q, s) represents the small intrinsic rank of the weight matrix, reflecting the low-rank approximation. The standard initialization of BLORA and A LORA is BLORA O and A LORA is drawn from normal distribution with mean 0.
Neural tangent kernel (NTK) Jacot et al. 2018 introduced the NTK, which captures the training dynamics over time. They demonstrated that in the infinite width limit, the NTK remains constant. In this limit, training dynamics are governed by a linear model derived from a first-order Taylor expansion around the initial parameters of the network, known as the linearized or NTK regime (Lee eet al. 2019]. For networks with finite width, this limiting kernel depends on the initialization parameters and is known as the empirical NTK (Wei et al. 2022]. J. Although the empirical NTK differs from the infinite width limit, it is valuable for analyzing the local training dynamics of
models [Ren et al. 2022 Fort et al. 2020 Mohamadi and Sutherland 2023 Wei al. 2022 Jang
et al. 2024]. and has been used in FT RRen et al. 2023
Malladi et al. 2023
Analysis of LP-FT from NTK perspective
The original analysis of LP-FT by Kumar et al. [022] is based two-layer linear model and proposes the feature distortion theory, which suggests that minimal changes in pre-trained features are the reason behind the robust performance of LP-FT. In this section. use the NTK theory to analyze LP-FT to better understand the training dynamics LP-FT in complex models like Transformers. After introducing the notation, discuss the increase in the classifier weight norm during training, followed by the training dynamics in the NTK regime. We then extend our analysis to the LoR method. These analyses suggest the LP-FT reduces feature distortion with the increased norm of the classifier weight and the near-optimal prediction after LP.
4.1 Notation
Let X (æI,.... NN C Rd represent the training samples, paired with labels from the set J = [y1,.... yNm C 21, C), where d, C, and N denote the dimensions of the input space, the number of classes, and the number of training samples, respectively. This forms training dataset {(01, (x).....,(N.....) (xN.yN) x, E X. K, E V), and we use x E Rd to denote both a training and a test sample. We denote the k-th clement of vector as [a]s. We use the Euclidean norm || for vectors and the Frobenius norm || ||e for matrices. (..-) denotes the inner product of two vectors. ex represents the one-hot vector for class k and Ic is the identity matrix of size C.
The model function, denoted as f(::)) : > RCC, is parameterized by set of parameters and sometimes abbreviated as f-). The model includes a linear head, also referred to as the classifier, which consists of a weight matrix V and bias vector The feature extractor denoted by ((.) R. + C where / represents the hidden dimension. The output of the model is given by f(z) V (ax) + Parameters for a function g(-) and matrix A are sometimes denoted as 0 and 0d respectivelyy Subscripts represent iteration epoch, fi(-) denotes the model at time t.
3

---

With the loss function & RC x y + R, the training objective is to minimize the empiri- Di {ff [ v.). We use the CE loss, {f) R + RC is the softmax function with its k-th element
cal risk L(f) L(f(:0)) N
log (f (y)))) where OSM exp(ff(m))e) given by [s ((())) Exep(((())e
4.2 Training dynamics the NTK regime
We use the NTK (Jacot et al. 2018 more specifically the empirical NTK (Wei et al. 2022 Malladi eet al., 2023), to analyze the training dynamics of both FT and LP-FT. The empirical NTK, defined as the NTK with the parameters at the start of training, is valuable tool for understanding the neural
network training process, particularly in the context of FT [Wei al. 2022 Malladi et al. 2023 Ren ct al. 2023). The empirical NTK applies first-order approximation to changes in model outputs with respect to its parameters, so this is expected to capture changes in features.
To investigate the feature distortion theory in FT and LP-FT, we decomposed the updates into the following two parts. The part influenced by feature updates, unique to FT and absent LP, is termed the FT-effective component of the NTK matrix, represented as Fax, xz)) In contrast, the part not influenced by feature updates, common to both FT and LP, determined by the pre-trained model, termed the pre-truin-effective component, represented as P(x, v.,. This decomposition highlights distinct training dynamics of LP-FT the NTK regime in the following proposition.
Proposition 4.1 (FT in the NTK regime). The NTK of a model f(æ) = Vá(x) + b, denoted by @ can be decomposed as:
of (x, x..) P(x, 22x,,;)
where the pr-trrain-fffective component P(x,, and the F'T-effective component F(x, (()) are defined using the classifier weight matrix Vo and the feature extractor Φu at starting point of training
P(x, xx.) = ((bo(x), 20 1)
((x, = Vo 0 dzp(x:;
00% 00 VoT.
Consequently, assuming that one-epoch training within the NTK regime approximates FT, the logits and feature vectors for a sample æ after FT, denoted as FT) and & (x)(x, to the starting point of training, fa(x) and Φo(x), can be expressed as:
(x) fo(x) n
Σ
(P(x, F(x, 6,,
(1)
N
TT - )) = Σ
(2)
where @ is the NTK matrix of the feature extractor o. d, == ey. sw (o((()()))) represents the difference between the one-hot label for the class yi and the predicted probability, and nj is the learning rate.
The proof of this proposition is included in the Appendix (Appendix A.2. of the NTK matrix, the pre-train-effective component P(x,, æ;, is a diagonal matrix and remains unchanged after I.P, while the FT-effective component F(x, ax) is not diagonal matrix and does change after LP, resulting in distinct characteristics for these components. The Frobenius norm of the classifier weight matrix, Voll F. influences the balance between the pre-train-effectio and FT-effective components because it affects only the FT-effective component. This indicates that the classifier weight norm || Volle e has significant impact the training dynamics of FT.
In our decomposition
Hypothesis on reduced feature changes in LP-FT The above proposition provides insights into why LP-FT causes fewer feature changes compared to FT:
1. The impact of the classifier weight norm || differs the equations: it affects feature changes linearly (2) and affects logits quadratically I]. This implies that higher norm can result in significant logit updates with relatively minor changes to the feature extractor, reducing feature changes in LP-FT compared with FT due to the increased classifier weight norm after LP.

---

Tel: acouracy
Panarcuneyy
1001
040
3792
Clasa weggt
Canater
15
131
23
25
40
35
(a) LP
(b) FT
(c) After training
Figure I: Increase in classifier weight norms during training on the RTE dataset. (a) and (b) show the increase of the both accuracy and classifier weight norms with training. (c) shows classifier weights norms after training.
2. The magnitude of changes in both features and logits ( and is proportional to d,. the difference between the predicted probability and the one-hot label. This suggests that feature changes are less pronounced in LP-FT than in FT since the difference 6; is smaller after LP.
3. The learning rate 7/, typically smaller in LP-FT than in FT [Kumar et al. [0022 Ren et al. 2023 Ha et al. 2024), helps moderate the direct influence large classifier weight norms.
Prior studies [umarr et al. 2022 Ren et al. 2023] have suggested that reduced feature changes in LP-FT stem from the near-optimal linear head obtained during LP.. However, our analysis reveals that feature changes LP-FT are also influenced by the classifier weight norm Vo after I.P. Our analysis focusing on classifier weight norms provides a new perspective on the training dynamics of LP-FT, highlighting the importance the classifier weight norm in reducing feature distortion.
4.3 Derivation Lemma A.3 from Kumar et al. in the NTK regime
The analysis presented in the original LP-FT paper by Kumar et al. [Kumar et al. 2022 operates within a framework where the feature extractor a linear function. We define this framework in our context as follows:
Definition 4.2 (Linear model [Kumar et al., 2022 ). linear model is defined as finear (x) Bx where V Cx is the classifier weight matrix and B Rhxd is the weight matrix of the feature extractor.
The linear model is model whose feature extractor a linear transformation. In this setting, derive a corollary from Proposition 4.1 in our context, which is the pivotal lemma in the original LP-FT analysis [Kumar et al.) 2022
Corollary 4.3 (I (Lemma A.3 from Kumar et al. in the NTK regime). Within the context of the linear model Defimitionn4.2 f for any sample Span(X)+, the orthogonal complement of the subspace spanned by the training sample set X, the features after FT remain unchanged, expressed as:
(TT(x) (
Φo(œ),
where (x)( and Φb(x)) denote the feature vectors after and before FT, respectively.
This corollary shows that feature vectors for the samples in the orthogonal complement of training sample subspace not updated. Therefore, given that pre-trained features have characteristics beneficial to downstream tasks, significant feature changes in FT, dependent on small training samples in LP, lead to poor generalization and OOD performance. The proof of this lemma be found in the Appendix (Appendix A.2.2
4.4 Increase in the classifier weight norm
The analysis in the previous section suggests that the classifier weight norm affects both feature changes and logits. On the basis of this insight, we examine classifier weight norms during training. Figure [1 shows that classifier weight norms consistently increase over time for LP, standard FT, and LoRA. As the training proceeds, norms of classifier bias and logits increases, while training loss decreases. Notably, LP shows significantly larger increase in the norm compared to FT and LoRA.
5

---

Consider the transpose of the k-th row of matrix V denoted as 2 E Rh for 1 S C, where C' is the number of classes. Let The represent the angle between Φ (z)) and vk. which expands (Ug, ((x))) || Φ (z)] cos Tks The probability that class is chosen for sample æ; is given by the softmax (xp( .(())))) function [aSM (x:)))] = Ev exp( ((23))) Consequently, with the CE loss for an input æ;;
classified into class y, defined (ff())) partial derivatives:
log ((rsM (1)))))) we have the following
all (xs))))) 3 cos The
(asM (f(4 ) ] I [+)))))]]
if # V4. if = yi.
- [as (f Mll (lllo((())))))
where the derivative with respect to cos Ty is negative and positive for k * yt- As training progresses, cos Ty.s tends to increase towards positivity, while cos Thi for & o y tends to become negative for each i. The derivative with respect to || is given by:
OL(F) a|| mk
N
Eloss (f(x))]))] cos TRI Ko
(1 loss (f(xs)))s: ((bb(())) cas Tui k=
Therefore, with adequate training and cos Tke < and cos Tyux > 0, the derivative with respect to ||x is likely to become negative for each class k. The training of the model proceeds so that the empirical risk decreases, so the norm | tends to increase. This finding aligns with prior studies (Soudry et al. 2018 Kim and Kim 2020
Remark: increase in classifier weight norms is more pronounced in LP than in FT In FT, particularly within an overparameterized setting, the model f may achieve perfect classification on the training dataset. That is, sM (f (*:))] becomes close to O for k + y/ and for k: yı- In this scenario, the derivative in Eq. (3) becomes close to zero, or the training itself is finished. Conversely, perfect classification is typically unattainable in LP unless the training dataset is linearly separable, so the derivative continues to be negative. In addition, while all parameters are updated FT, only the classifier is optimized in LP, so the change in the classifier weight needs to be larger in LP than in FT to achieve the same classification performance. Consequently, the classifier weight norm tends to increase more significantly in LP than in FT, as shown in Figure[[(c)
4.5 Training process of LoRA
We extend our analysis based on the NTK the training process of LoRA. We follow the linear model setting as in Definition and analyze the training dynamics of LoRA in the NTK regime.
Proposition 4.4 (LoRA approximates FT). Consider the linear model setting (Definition 4.2 and let f floRA and fP be the models obtained via one-epoch training with LoRA and standard FT the NTK regime. Let r denote the rank of the LoRA hyperparameter, and o² represent the variance of the low-rank weight matrix initialization. Assume the input sumples x satisfy ||a||| S c. Then, for each sample pair ai, Fy E X, the pre-truin-effective components of the NTK matrix for LoRA and FT, (ax
PLaRA (Ii, 2.;) and PFT (1 I,), are identical:
PLORA (x,, (x,) = (.),
Moreover, with at least 4cxp(-(( 2)/44) probability, their FT-effectiive components, FLORA az) and (f R.)), satisfy:
|FLORA (z; x;)) - - FF xI)) < <ce| VOV. I.
This proposition suggests that with high probability, the only difference of the NTK matrix between LoRA and standard FT is a scalar factor of the FT-effective component in the NTK matrix, and the scalar factor depends on the hyperparameters of LoRA. This implies that when the hyperparameters of LoRA are set appropriately, LoRA training is similar to standard FT training. This consistent with the analysis by Malladi et al. 22023), where the NTK matrix of LoRA and standard FT are close to each other. It is important to note that the proposition is also valid for L.-FT and L.-LoRA (I.P then LoRA). The proof of this proposition is included in the Appendix (Appendix A.2.3)

---

4.6 Discussion
An increased norm of the classifier weight reduces feature distortion and enhances the contribution of the FT-effective component of the NTK matrix during training. As result, a higher classifier weight norm in LP-FT can be advantageous. However, since the increased norm is dependent on LP training, its optimality is not guaranteed. Specifically, during test time, although the increased classifier weight norm does not influence accuracy, affects the calibration of the model. Calibration is defined as the alignment between the predicted probabilities and the actual probabilities [Guo al., 2017]. An excessively high classifier weight norm can lead to overconfident predictions, which might be detrimental in practical applications. Consequently, there is potential for refining LP-FT by adjusting the classifier weight norm to enhance calibration.
Tuning the norm of the classifier after training can be effectively equated to applying temperature scaling [Guo al. 2017] at test time. Temperature scaling adjusts the output logits with a temperature parameter T, thereby improving model calibration. Specifically, temperature scaling with parameter V
T, expressed as [x:/ = T (xx)) b and bias b by the temperature parameter
T can be viewed as scaling the norm of classifier weight V
5 Numerical evaluation with transformer models
In this section, we numerically justify the following aspects obtained from our analysis:
The changes in features during training are smaller in LP-FT than in FT, and the norms of the classifier significantly increase during LP (Section 5.2
The FT-effective component of the NTK matrix more effectively captures the input data than the pre-train-effective component (Section 5 and is more pronounced in LP-FT than FT.
A large classifier weight norm reduces the feature change during training, and its negative effects on calibration can be improved by temperature scaling (Section 5.4).
Details the datasets, setup, and additional results, including performance evaluations for the experimental and practical application, are available the Appendix (Appendices A.3 and A.4
5.1 Setup
Datasets and models We used total of 13 classification datasets from various benchmarks: SuperGLUE [Wang al. 2019], GLUE [Wang et al. 2018), BOSS (Yuan et al. 2023], and PubMed 20k RCT [Derooncourt and Lee 2017). The breakdown of the datasets is follows: five datasets from SuperGLUE (BoolQ. CB, RTE, WiC, and WSC), three datasets from GLUE (CoLA, MRPC, and SST-2), four datasets from BOSS (Amazon, Dynasent, SemEval, and SST-5), and PubMed 20k RCT. Following experimental settings in studies that analyze FT dynamics from NTK perspectives [Malladi al. 2023 Jang et al. 2024) and the study with similar settings Chen et al. 2022. we employed the KoBERTa base model
Liu et al. 2020 as our Transformer- based model.
Table 1: Changes in features (F) and classifier (C) norms the CB and RTE datasets. CS, Diff, FDR, and Norm represent the cosine similarity between features, the difference in norms from the pre-trained model, Fisher' S discriminant ratio, and the norm, respectively. After LP-FT, Diff(F) smaller compared to FT, while preserving the high CS(F) and low FDR(F) of the pre-trained features. In contrast, Norm(C) is significantly larger after LP and LP-FT than both the pre-trained model and after FT. This trend is also observed when training with LoRA.
Method
CB
RTE
CS(F)
Dif((F)
FDR(F)
Norm(C)
CS(F)
Diff(F)
FDR(F)
Norm(C)
Pre-trained
0.997
8.14
104
9.51 x 10-1
0.996
8.59
10
7.76
10-1
LP
0.997
8.14
x
104
2.48 x 101
0.996
8.59
x 10a
3.10 x 10a
FT
0.336
2.21 x
7.39
10a
9.60 x 10-1
0.260
2.16 x
1.42
104
7.84 x 10
LoRA
0.499
1.92
10a
8.91
106
1.43 x 10a
0.759
1.06
10a
2.97
103
1.21
100
LP-FT
0.804
1.20 x 10
6.47
106
2.48
x 10'
0.942
4.70 x 10
1.57
x 10a
3.10 x 10
LP-LoRA
0,837
9.08 x 100
2.10
x
108
2.49 x 10
0.924
4.63 x 10
2.06
x 10
3.10 x 10
7

---

Table 2: Kernel statistics on the CB dataset. FN, Acc, and FT Ratio denote the Frobenius norm, kernel regression accuracy, and contribution of the FT-effective component, respectively. Pre-
10-
Singular value
train E and FT E refer to the pre-train-effectivee and FT-effective components of the NTK matrix.
28"*
Pre-traint E
LP-FT
Method
Kernel
Rank 1N(x109)
Ace (train/tstt FT Ratio
LORA
12n0
LP-LORA
Pre-mmin E
18
51.0
87. 87.11/79.17
FT
FTB NTK
608
13.9
84.74/79.76
0.1987
Index
210
64.9
84.74/79.76
LORA
FTE
500
0.0226
08.22/79.11
0.0004
Figure Singular value distribution normalized by the maximum value on the CB dataset, show- ing the common pre-train-effective component (Pre-train E) and the FT-effective components for each training option.
NTK
51.0
02.15/84.52
LP-FT
NTK
FT
344
344
7250
7280
100.00/
100.00/56.31
00/00.33
/6.31
1.0000
LP-LORA
FTE
307
15.1
94. 96 /85.7
NTK
188
62.6
11/8511
1.0137
Implementation and training We used the Transformers library [Wolf et al., 2020 and Adapter- Hub [PPeiiffer et al. [020] for our implementation. Our training protocol followed the experimental setup described by Chen al. [2022). Hyperparameter tuning, especially for learning rates during the FT stage of LP-FT, was conducted through grid search based on the validation set performance. For LP, we used logistic regression with L2 regularization on pre- pre-trained features.
5.2 Small feature changes during LP-FT and significant norm increase during LP
LP-FT achieves notable performance with Transformer-based language models, outperforming standard FT in both ID and OOD settings, as detailed in Appendix (Appendices and To understand the underlying reasons for these results and validate small feature changes suggested by analysis (Section 4.2) we analyzed changes in both the classifier and the features.
According to statistics presented in Table T the features after LP-FT demonstrate smaller changes from those of the pre-trained model than FT. Consequently, LP-FT preserves high cosine similarity among its features and exhibits low Fisher's discriminant ratio (FDR) [Fisher] 1936). which assesses linear separability. Conversely, the classifier norms after LP and LP-FT are substantially larger than those of the pre-trained model and after FT, suggesting significant increase classifier weights during L.. A similar trend is observed in training with LoRA.
5.3 Kernel analysis
We examined the overall NTK matrix and its pre-train-effectivee and FT-effective components to understand their properties. Kernel regression was performed on the train and test sets to evaluate the performance of each kernel matrix.
Analysis of NTK matrix components and effectiveness of LP-FT In Table the FT-effective component of the NTK matrix for LP-FT shows higher rank and greater kernel regression accu- racy compared to the pre-train-effeetive component, and the overall NTK matrix has intermediate properties. Additionally, the FT-effective component contributes more significantly to the overall kernel LP-FT than in FT, as indicated by a higher FT Ratio. This ratio, calculated as the average of IE F(x, 1,.)),11/ l|||| E enhanced influence of the FT-effective component in LP-FT than in FT. These results suggest that the NTK matrix LP-FT better captures input data through the increased influence of the FT-effective component.
(P(x, x.)) + F(x, x. xx)) ) 8 for the train set samples, reflects the
Similarities between LoRA and FT The ranks of the FT-effective components in LoRA and FT (or LP-LoRA and LP-FT) are similar, as indicated in Table Their distributions of singular values normalized by the maximum singular value, also closely align, as shown Figure These results suggest that the FT-effective components of the NTK matrix in FT and A differ only by scalar factor. This consistency demonstrates that our analysis (Section 4.2), originally based on a two-layer linear model, is applicable to more complex Transformer-base. models.

---

Table 3: ECE and MCE with temperature scal- ing on the test set of the RTE dataset. w/o TS and w/ TS denote without and with temperature scaling, respectively, and Imp. represents the im- provement because of temperature scaling. We bold the best improvements. This table shows that poor calibration of LP-FT can be effectively mitigated through temperature scaling.
Norm the feature difffrencc
FT
LP-FT
Metric
Method
w/o TS
w/ TS
Imp.
10a
10t Norm classifier weight
FT
21.16
5.13
16.03
LP-FT
21.72
5.48
16.24
ECE (%)
LoRA
11.92
6.17
5.76
Figure 3: Feature differences SST-5 (OOD). Solid lines show mean values; shaded areas rep- resent standard errors. Dashed vertical lines indi- cate the classifier weight norm after training. This figure validates our analysis that larger classifier weight norms reduce feature changes.
LP-LoRA
18.14
5.72
12.42
53.11
25.87
27.24
LP-FT
63.95
13.94
50.01
MCE (%)
LoRA
25.04
13.75
11.29
LP-LoRA
40.46
18.82
21.63
5.4 Analysis of classifier weight norms and temperature scaling
We experimentally verified significant effects of classifier weight norms in training (Section 4.2 and at test time (Section 4.6) in the following.
Effects of classifier weight norms in training scaled the classifier weight norms at the start of the FT stage of LP-FT. The results, shown in Figure 3 indicate that larger classifier weight norms almost monotonically lead to smaller feature differences both FT and LP-FT, Notably, LP-FT consistently shows smaller feature differences than FT, particularly when the classifier weight norms are large, validating our analysis that larger classifier weight norms reduce feature changes.
Temperature scaling at test time We implemented temperature scaling at test time, which is equivalent to adjusting the classifier weight norms, as discussed Section [46] We optimized the temperature parameters the validation sets based on CE loss, following the methodology suggested by Guo et al. [2017]. Table 3 presents the results on the RTE datasets. We assessed the expected calibration error (ECE) and maximum calibration error (MCE) [Naeini al. 2015), which quantify the absolute differences between predicted and actual probabilities, with lower values indicating better calibration. These results show that the improvements in calibration with temperature scaling are the largest in LP-FT for both ECE and MCE, with notably substantial improvements in MCE. This suggests that large classifier weight norms contribute to poor calibration of LP-FT, which can be effectively mitigated through temperature scaling. These results highlight the effectiveness of refining I.P IP-FT by temperature scaling.
6 Connclusion
In this paper, we explored the LP-FT training dynamics in complex classification models using the NTK to analyze feature changes. Our analysis identified classifier weight norms at the start of the FT stage as key factor influencing FT dynamies. These norms balance the NTK matrix components and help reduce feature changes. Our findings support the existing feature distortion theory from an NTK perspective and emphasize the role of classifier weight norms alongside prediction accuracy. We also found that increases in classifier weight norms, characteristic of training with CE loss, may negatively impact model calibration, and this can be mitigated by temperature scaling. Additionally, the approximation effectiveness of LoRA is theoretically validated in terms of the similarity of the NTK matrix components. Empirical experiments with Transformer-based language models supported our theoretical insights, validating our understanding of the NTK, feature changes, and the benefits of temperature scaling. Overall, our study substantiates the efficacy of LP-FT robust method for adapting pre- trained complex models while preserving their well-trained features.

---

Limitations The main limitation of our study is that it is based on the NTK regime, which might not fully capture the training dynamics. Additionally, we consider just one epoch of gradient descent in FT, which may not effectively represent the overall training. In our experiments, we specifically focused on validating the effectiveness of LP-FT on language models. Therefore, areas other than natural language processing are outside the scope of our experiments.
Acknowledgments and Disclosure of Funding
This work was supported by JSPS KAKENHI Grant Number 24H00709 Japan.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10. N..18653/11/202111 acl- 1on.1ong. URL https: https://aclaatthllogg. org/ 2021 acl-long 568
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Jill Burstein, Christy Doran, and Thamar Solorio. Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume (long and short papers). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang, Revisiting parameter-efficien tuning: Are we really there yet? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2612-2626, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 111665//11222222mmmmmmm// 168. 168. URL htttp:::/acllamtooooyy.ooo emn1p-main.168
Christopher Clark, Kenton Lee, Ming-Wai Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Papers), pages 2924 2936, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1853//1//N19-13000 URL httpss:/aclaathhollos
1300
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177-190. Springer, 2005.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. volume 23, pages 107-124, 2019.
Franck Dernoncourt and Ji Young Lee. Pubmed 200k ret: a dataset for sequential sentence classifica- tion in medical abstracts, 2017.
10

---

Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005), 2005.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, (2):179-188, 1936.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution the neural tangent kernel, 2020.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entuilment and paraphrasing, pages 1-9, 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pages 1321-1332 PMLR, 06-11 Aug 2017. URL httttees//proceeednggg mlr pwwps/ww0/ //o0/// http:i/tgmmm.m...m//////tm.a/
html
Scokhyeon Ha, Sunbeom Jeong, and Jungwoo Lee. Domain-aware fine-tuning: Enhancing neural network adaptability. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12261-12269 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficien transfer learning. In International Conference on Learning Representations, 2022. URL.btttss::/opperrr/e/
Richard Zou Horace He. functorch: Jax-like composable function transforms for pytorch. https //github. com http://yooooffffttttocoe
2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799, PMLR, 09-15 Jun 2019. URL houlsby19a. html
prwss/ /
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International
Conference on Learning Representations, 2022. URL
net/ forumorrmmo/
I2ZeVeeFYf99
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Uijeong Jang. Jason D. Lee, and Ernest K. Ryu. Lora training in the ntk regime has no spurious local minima, 2024.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. arXiv preprint arXiv: 1911.03437, 2019.
Byungju Kim and Jummo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access, 8:81674-81665 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. The Eleventh International Conference on Learning Representations, 2023. httpp htt.a///w......eeroorrrr///////.
Simon Korbblith, Jonathon Shiens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2661-2671, 2019.
11

---

Ananya Kumar, Aditi Raghumathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform distriibiittiiiiiionn In International Conference on Learning Representations, 2022. URL. http.ttttts://peeeevviii
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations, 2023. URL APuPRxjHvZ
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Hao Li, Pratik Chaudhari, Hao Yang. Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning, In Internatiiona Conference on Learning Representations, 2020. URL httpp:://openreviiew.aeee/////// httpppppww.aaaaaeeeeeooeoaooaaan
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro(bert) A robustly optimized (bert) pretraining approach, 2020. URL htk.htttss///ppeneriieee http:///ppppppeeeeeeeeeeeeeeerea nett:/oorummdddSyyyS0ttttt
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610-23641. PMLR, 2023.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36, 2024.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of ACM Conference on Recommender Systems, 2013.
Mohamad Amin Mohamadi and Danica J. Sutherland. A fast, well-founded approximation to the em- pirical neural tangent kemne,, 2023. /ttltttts////onrrooot wwt.:wwwwwweeeoowmm/eee Hit.ooo ov5Q
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference artificial intelligence, volume 29, 2015.
Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sehastiani, and Veselin Stoyanov. SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of International Workshop on Semantic Evaluation (SemEval), 2016.
Roman Novak, Jascha Schl-Dickstein. and Samuel S Schoenholz. Fast finite width neural tangent kernel. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17018-17044, PMLR, 17-23 Jul 2022. URL https:://proceddiie
mlr.
html
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-28300 2011.
12

---

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. AdapterHub: A framework for adapting transformers. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference Empirical Methods Natural Langagge Processing: System Demonstrations, pages 46-54, Online, October 2020. Association for Computational Linguistics. doi: 00.188653/1120220emmmp/pp htttps://aclanthologg org org /w00g/ emmlp-demos.7
URL
Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. DynaSent: A dynamic benchmark for sentiment analysis. In Proceedings of ACL-IJCNLP, 2021.
Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing learning paths, 2022.
Yi Ren, Shangmin Guo, Wonho Bae, and Danica J. Sutherland. How to prepare your task head for finetuning. In The Eleventh International Conference on Learning Representations, 2023. URL
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, 2013.
Daniel Soudry, Elad Hoffer, and Nathan Srebro. The implicit bias of gradient descent on separable data. In International Conference on Learning Representations, 2018. URL httpss://opeereeiiis net./forru??d-rr1//nggglll
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances neural information processing systems, 33:7852-7862, 2020.
Puja Trivedi, Danai Koutra, and Jayaraman J. Thiagarajan. A closer look at model adaptation using feature distortion and simplicity bias. In The Eleventh International Conference on Learning Representations, 2023. URL pettps://openreviee net/forrmmid-wwkeeee....
IwTZ
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL hhtps: //procee//rr . neurips. 7166e44547aae11100ccccccaco
Paper pdf
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purposs language understanding systems. Advances in neural information processing systems, 32, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 10.1
tacl_ 00290. URL httptstt////aaaattoooogg/ hwaplloooyoorgggg/ ptgg/19---0 1040
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize, In Proceedings of the 39th International Conference on Machine Learning, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Compu- tational Linguistics. URL hothhtts::/ / /www wclweb. org/ arggannhologyy//n0 2020 2020. emnlp-demos: demos. 6
13

---

Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting disttr-ddittibuuioon robustness in nlp: Benchmark, analysis, and llms evaluations. arXiv preprini
2023.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In The Twelfth net/
International Conference on Learning Representations, 2024. URL Forum/id=LikkV
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 698-714. Springer, 2020.
Fuzhen Zhuang. Zhiyuan Qi, Keyu Duan, Dongho Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong. and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76. 2020.
A Appendix / supplemental material
A.1 Abbreviation and notation
Table 4 and Table show our abbreviations and notations, respectively.
Table 4: Table of abbreviations.
Abbreviation
Definition
FT
fine-tuning
LP
linear probing
LP-FT
linear probing then fine-tuning
NTK
neural tangent kemel
LoRA
low rank adaptation Hu et al. 2022
ECE
expected calibration error ([Naeini et al.
2015
MCE
maximum calibration error (Naeini et 2015
ID / OOD
in-distribution / out-of-distribution
FDR
Fisher's discriminant ratio [Fisher
1936
A.2 Proof of theoretical results
Additional notation The parameters for a function g. a weight matrix A, and vector a is denoted as 08, 04, and 09, Given a function g(: 09): Rd - R trained on N training samples
X' ( .2, NNN } C IRd, we denote the NTK matrix of g at time f as Og. which is defined as
Og 0g: (0g((
E Nsx.Ns where (x') = vec
and (g.(X)) 300
E Naxp
De0
with p parameters. The sub-matrix (x; ..y defined as O (2,, 2x) = (ws)
(8g. (x,)
T
00w
Raxa describing the relationship between training samples æ; and æ, in X. the infinite width
limit with NTK parameterization and general assumptions, the NTK matrix converges to O ==
0gal2) 869
(Qgn(X) 86
as shown by [Jacot et al. 2018). Subscripts represent iteration or epoch. so g.).)
denotes the model time l. @ denotes the kronecker product of two matrices defined as
an "nn B
B12
Oin
a21 B
a2
B
a2
B
A A B =
amm B
am
B
Omn
B
where A [ayy] is an n matrix and B is any matrix.
14

---

Table 5: Table of notations.
Variable
Definition
C
N
number of classes training samples input dimension hidden dimension rank of LoRA trainig samples labels
d
h
f/r
X
R/Y
a.
sample / label
a
k-th element of vector
/6,0
Euclidean norm Frobenius norm inner product
By
one-hot encoding of label
Ic
x C' identity matrix loss function empirical risk softmax function model output feature extractor classifier weight / bias classifier weight feature extractor the start of training feature extractor weight matrix in two-layer linear model low-rank weight matrices in LoRA
{(f(x))))
ss
f(x)
Φ(x)
V
Vo Φp
B
A ALORA BLORA
09 0A) 0a
parameter of function g matrix A vector
0f/ /
NTK matrix of model feature extractor pre-train-effective FT-effective component of NTK matrix difference between one-hot label and predicted probability learning rate kronecker product of two matrices
P(x, x.) F( a.)
d,
7
A.2.1 Proof of Proposition 4.1
Proposition 4.1. decomposed as:
The NTK matrix of model f(x) Vdd(x) b, denoted by Or, can be
(x = P(x, ax)) + F F(x, a;;;
where the pre-train-effective component P(x 2;)) and the Fl-effective component F(x, o;) are defined using the classifier weight matrix Vo and the feature extractor Φo starting point of training
as:
P(x, xx.) = ((bo(x), (b((xx))))))))))))(())) + 1)
0(p(()) aœ)( VOT.
F(ar ar)) Vo
O0 Og
Consequently, assuming that one-epoch training within the NTK regime approximates FT, the logits and feature vectors for a sample æ after FT, denoted as (FT(()) and & (æx), to the starting point of training, fo()) and (ar),, can be expressed
N
( -f(()) = (P(æ, + F(x, 8,,
Σ
i=1
N
$ PT (x) - Φ0(x) Σ
0
x; Vo 8,,
where 8, := ev. SM (ffo
(o(f)))))) (z;)) represents the difference between the one-hot label and the predicted probability, and v is the learning rate.
Proof of Proposition 4.1
15

---

Proof. The parameters of f. denoted as of, consist of 0V, 0", and 0 The derivative of the model with respect to each parameter given by:
0f(x)
d0V Φ(x) T @ Ic.
(4)
0f())
00h Ic.
(5)
0f(z) v
@Φ(+)
(6)
000
000
Therefore, the NTK matrix of f. defined as O (xx x.,)
0fo(m)
(0fula.))
can be expressed as:
30t
0fo(z)
(0fo(z))
+ 0(1 xx)
00s
00s
(f((x)
0f(
fo(œ)
0fo(x;)
Qf((z)
ff((x,)) T
30V
80V
006
30b
000
80%
= (<b((x) @ Ic) (x))
T 2 Ic) Ic
V
( Eqs.
44
5
6
000
000
(db(a)) (o Ic Vo
Vo
00%
000
@Φv())
((do(w)), do (a:)) 1)1e Vo
Vo
000
00%
P(x,,,) F...
For gradient descent, the update to the parameters 0f at time t is given by:
[T(Ff))
0 - 0f =-1
001
a @log([ (f (z, af((m:)
001
(ew, - ofM(f(x:))) (x:)
(f(
001
6,,
(8)
00
where d, is defined as 6; = Ey: o fo ())). Assuming that one-epoch training approximates FT, the model is expressed as foF f1. Therefore, the update to the model f in the linearized regime given by:
f (T) - folx) fi()) fo(x) (:. onoe-epochh approximation of fine-tuning)
(f) 10f of)
(t. linearized regime)
001
N
fu(())
(Of((z.)
1 00f
(a0e 005
8, (:: Eq. (8)
(P(,.;)) + a(x, 8, (: Eq.
16

---

Finally, replacing of with 8 in Eq. (8. the update to the parameters 0 at time is given by
N
f(.))))) T
0
0
8,.
000
Therefore, the update to the feature extractor after FT, given by t = Φi for the same assumption,
is:
2
Φ0(x))
Φb (20)
iΦ0 (z))
0 (0
(. linearized regime)
000
(()(()))
N
ff((a))
T
6,
(: Eq. (
00Φ
00%
(x))
axpo(ass) ag
Vo
ō; (:: Eq. (6)
00
N (p)((a)
(Oppo(as))
Vola. 8:
00Φ
000
Σ
g* (xr, x., Vo 8
i=1
This completes the proof.
A.2.2 Proof of Corollary 4.3
Corollary Within the context of the linear model (Definition 4.2p. for any sample Span(X) the orthogonal complement of the subspace spanned by the training sample set X, the features after FT remain unchanged, expressed as:
st (z) (x)).
where (xx) and Φo(z) denote the feature vectors after and before FT, respectively.
Proof of Corollary 4.3
Proof. The feature extractor is given by ((x) Bæ, where B is the weight matrix. The derivative of the feature extractor with respect to the parameters 8 B is:
0Φ(x)
dBx
B0 x & I
20+
so the empirical NTK matrix of the feature extractor becomes:
( m.)) = a(p((a;) 0<po(a;)) T
000 O0%
= -xx, xz) 8 Ip
where & denotes the kronecker product.
From the Proposition 4.1 the feature update is given by:
(T) Φo( = 11
0 (a., 2; Vo 5;
d=1
Σ ar, x.) Vo 8;
"
where 5; Cy. ssm(fo((;)) Vo is the classifier weight matrix at the start of training, and r is the learning rate. For any sample E Span(X') (z, x.) = 0 for all æ; E X, so the feature update is 0 for OOD samples, namely:
(x) - (x =
This completes the proof.
17

---

A.2.3 Proof of Proposition 4.4
Proposition 4.4 Consider the linear model setting (Definition 4.2 and let floRA and ffT be the models obtained via one-epoch training with LoRA and standard FT in the NTK regime. Let denote the rank of the LoRA hyperparameter, and o² represent the variance of the low-rank weight matrix initialization. Assume the input samples a satisfy ||||| S c. Then, for each sample pair a; jjj X, the pre-train-ffeecivv components of the NTK matrix for LoRA and FT, PLoRA æF) and pit (æ;, 22), are identical:
PLORA PLoRA 23)
pri (z,, zy).
Moreover, with at least - 4 pexp(_(e"" (2)r/4) probability, their FT-effective components, FLORA (T,; 2z)) and F (æi, (x; Tt), satisfy:
|| | 23,) r² F (I,, x,)ي S ce Vo
||.
Proof Approach prove this theorem, we use a lemma from distributional properties:
Lemma A.1 (Corollary of the distributional
Lemma). Given vectors Rd with [lall ||||| S c, and a random matrix A E Rexd with i..d. entries from a distribution with
mean 0 and variance for any > 0:
Pr [r(A (Av) u u 2 ce] S 4exp (-)62-c))//44)
Proof of Proposition 4.4
Proof. The feature vector of LoRA is given by GLoRA (x)) = Box + BLORA ALoRA where pre- trained feature weight matrix B, is fixed during training, and A LORA E Rrxd and BlaRA e Raxr low-rank weight matrices in LoRA. ALORA is initialized from normal distribution with mean and variance a², while BLORA is initialized with zeros. The LoRA feature updates are represented as @LooR (x) Ba BLoRA A L with Bg fixed during training.
The pre-train-efffctive components of LoRA and FT, denoted as PLoRA x;) respectively, are defined as:
and PFT (x. x.))
PLORA (r, ax;) = ((0boRA())) A + +1).C,
(F x.) ((057 (x),
(x)) o5 o (x,)) +1Jcc
where Ic is the identity matrix size C. These pre-train-effiee effecttve components are identical since:
@
LoRA
(x) Boæ BRA LLRA x Byæ = -5 (x),
for all æ E because BLORA is initialized as a zero matrix i.e. BBORA O.
For the FT-effective component of the NTK matrix, consider the derivatives concerning LoRA parameters BLoRA and A AlaRA,
acBLORA(2))
Ogmuar = Ax & V,
AeSLORA (x: @ V BLLORA BerAABaann BLoRAT V
Agaia
Here, B and 4A denote the parameters of BLORA and LORA respectively.
The FT-effective component the NTK matrix for LoRA, denoted as (L is derived by combining these partial derivatives:
( @oLORA (z) (OLORA(Z,))
doLoRA (æ) aeboRAA
FLORA (x. x;)) =Vo
Vo
BB Bi
A0amnn
Vo ((ABORA x, Ab LORA x.,) + (x. xx Bl
LORA BL
BLORAT) Vo-
= =AALURA x, Al A xx) (.RAA,
V.V.
18

---

where the last equality holds because aa is zero matrix. Similarly, the FT-effective component of the NTK matrix for standard FT, FFT is given by:
@ T (x)
(xx)
)
F (x, =Vo
Vo
00
80g
Using the Johnson- Somnson-Lindenstrauss lemma, with probability of at least 1 4 11-eep(-(2=-e)))
KA LORA
(AALRA a, A LARA A1=R^x,) - oar(x, xiهil S co²re,
which implies:
|| |FLLRA (x) - o2r.FI((r,,) a2)l =11 (ARRA A ALORA v;) VoVo 02r(2, VoVo
5| LORRA 3;; A ALORA mi) - o² r2 ..) VoV
Seo VoV.
This completes the proof.
A.3
Experimental details
A.3.1
Datasets
From the SuperGLUE benchmark [Wang et al. 2019 we used the five datasets: BoolQ C lark
et al.
2019]. CB (CommitmentBank) [De Marnetle et al. 2019]. RTE (Recognizing Textual Entail-
ment) [Dagan et al. 2005 Bar-Haim et al. |2006] Giampiccolo et al. 2007 Bentivogli et al. 2009]
WiC (Words in Context) [Burstein et 2019], and WSC (Winograd Schema Challenge) [evessue
et al.
2012 From the GLUE benchmark [Wang et al.) 2018). we used the three datasets: CoLA
(Corpus of Linguistic Acceptability) [Warstadt et al. 2019]. MRPC (Microsoft Research Paraphrase
Corpus) Dolan and Brockett] 2005]. and SST-2 (Stanford Sentiment Treebank, version 2) [Socher
al. 20133. Four datasets from BOSS Yuaan al.. 2023 were used in OOD evaluation: Amazon
Reviews McAuley and Leskovec 2013 Dynasent [Potts et al. 2021], SemEval [Nakov et 20,,)
and SST-5 [Socher et al.) 2013]. Finally, we used the PubMed 20k RCT dataset [Dernoncourt and
Lee 2017
for validation in practical settings. The dataset statistics are detailed in Table
For the datasets from the GLUE, SuperGLUE, and BOSS benchmarks, we divided the original training set using a 9:1 raaiing-to-vvlidaat ratio, using the original validation set as the test set, in accordance with Chen et al. [2022]. For PubMed 20k RCT, we used the original training, validation, and test sets for their respective purposes.
A.3.2 Implementation and training details
When applying LoRA, LoRA was applied only to the query and value projection matrices of the attention mechanism in the Transformer architecture, following the approach described in the original paper by Hu et al. [2022). The LoRA settings were fixed at a and 8 for all experiments.
The model was trained for 10 epochs without early stopping, and the one showing the best performance on the validation set was chosen for further evaluation. We used the Adam optimizer [Kingma and Ba 2017). Our code is built on PoTorch [Paszke et al. 20199. using the HuggingFace Transformers library Wolff et al., 2020 and AdapterHub [Pieiffer al. 2020). All experiments were run on single NVIDIA A100 GPU. The results reported are averages from 3 tuning seeds and 5 evaluation
seeds.
For LP, cross- validation and automatic hyperparameter adjustment were used to find the optimal L2 regularization strength, using seikit-learn (Pedregosa al. 2011] with its standard training parameters.
Details on the hyperparameters for our experiments can be found in Table 6
19

---

A.3.3 Details of each experiment
Experiments the GLUE and SuperGLUE benchmarks For the FT and LoRA methods, the learning rate and batch size were adopted from Chen et al. [222], where these hyperparameters were optimized using grid search on the validation set. For LP and LP-LoRA, batch size is fixed at and we tuned the learning rate.
Experiments on BOSS benchmark and the PubMed 20k RCT dataset For the experiments on BOSS benchmark and the PubMed 20k RCT dataset, we tuned the learning rate and batch size using grid search based on the validation set performance.
Calculation of the NTK matrix We computed the NTK matrix for FT, LoRA, LP-FT, and LP-
LoRA as specified in Eq. ( We separately calculated the pre-train-effectivv and FT-effective components of the NTK matrix. Following the methodology by Malladi et al. [024]] we used functorch [Horace He 2021] and forward-mode auto-differenniation [Novak et al. 2022 for these calculations. To reduce computational costs, we randomly selected 10% of the parameters from the word embedding matrix for derivative calculations. For datasets with more than 250 samples, we used subset of 250 randomly selected samples to compute the NTK matrix.
Solving the kernel regression Following the methodology described by Malladi al. [0024), treated each output logit independently in our kernel regression model. This method is based on the representer theorem, where the empirical risk minimizer is expressed a linear combination of kernel features from the training data: f(x) = matrix or its component for a training set of size NC NC. We solved this optimization using logistic regression with L2 regularization and used the resulting coefficients a; compute logits on the test set via its corresponding NTK matrix.
Ei=l a; (xx 2z).. with K representing the NTK
Effects of classifier weight norms in training We scaled the norms of the classifiers within the range of 0.5, 1, 2, 5, 10, 50, 100) before proceeding to the FT stage of training, specifically after random initialization in FT and after LP training in LP-FT. We conducted this experiment using the CB and RTE datasets and Boss benchmark. We apply the LoRA method on the CB and RTE datasets. We averaged the results over 5 seeds for the CB and RTE datasets and seeds for the Boss benchmark, plotting these with their standard deviations.
Temperature scaling We applied temperature scaling Guo et al. 2017 to the logits of the
model at test time. Following the methodology of the original paper Guo et al. 2017 tuned the temperature parameter using the validation set to minimize the negative log- implementation, we employed the Adam optimizer [Kingma and Ba 2017 with learning rate of 1 x 10-3, optimizing the temperature for 1 x 10 steps. We incorporated early stopping based on the negative ho,liik,,,,od, with a patience of 10 iterations starting from an initial temperature value of 1.0. The number of the bins to calculate ECE and MCE is set to 15.
For
A.4 Additional experimental results
A.4.1 Results on the SuperGLUE and GLUE benchmarks
Table shows the test results for the SuperGLUE and GLUE benchmarks. We report accuracy and its standard deviation on the test sets, except for the CoLA dataset, which uses the Matthew's correlation coefficient for the performance metric.
Figure 4 shows the increase in the classifier weight norm during training on the CB dataset. With more iterations epochs, there is noticeable increase in both accuracy and the classifier weight norm.
Figure 5land Figure F t-SNE visualizations of the feature vectors from the CB dataset. After FT, the features are distinctly separated by class. contrast, the classifier row vectors remain nearly identical to those of the pre-trained model. After LP-FT, the features retain the structure of the pre-trained model, but the classifier row vectors deviate from their initial state. similar pattern observed with the LoRA method.
20

---

Table 6: Hyperparameter configurations. The settings include batch size (bs), learning rate (Ir), alpha
(a), and rank (r).
Method
Name
CB
RTE
BoolQ
WiC
WSC
CoLA
SST-2
MRPC
Amazon PubMed
16
16
32
32
16
FT
5e - le 5 le 5
le 5
le 3 5e 5
le
5 le 5 le - 5e
16
16
32
16
16
16
LoRA
Ir
le -
3
le - 3 5e
4
1e
-3
3 le
4
le 3
5e
4
5c 4 1e - 3
5e -
bs
LP-FT
le
le
le
le
5
le
-e-5
le
5e
bs
le
le 1 le 3 1e le 5e
le
LP-LoRA
Table 7: Dataset statistics. This table provides detailed counts of the classes, training, validation, and test samples for different datasets across various tasks including natural language inference (NLI), word sense disambiguation (WSD), question answering (QA), coreference resolution (coref.), sentiment analysis (sentiment), and sequential sentence classification (sequential).
Dataset
Benchmark
Classes
Train
Val
Test
Task
CB
225
25
RTE
2
2,241
249
277
NLI
BoolQ
SuperGLUE
2
8,484
943
3,270
QA
WiC
2
5,400
600
638
WSD
WSC
498
56
104
coref.
CoLA
7,695
855
1,040
acceptability
SST-2
GLUE
2 60,614
6,735
872
sentiment
MRPC
2
3,301
367
408
sentiment
Amazon
3
27,000
3,000
38,905
sentiment
Dynasent SemEval
3
4,320
sentiment
BOSS
3
20,622
sentiment
SST-5
1,067
sentiment
PubMed 20k RCT
PubMed
5
15,000
2,500
2,500
sequential
Table 8: Test results on the SuperGLUE and GLUE benchmarks. We report the accuracy and its standard deviation, other than the CoLA dataset, which is evaluated by the Matthew's correlation coefficient. We take the average of five seeds.
Dataset
LP
FT
LP-FT
LoRA
LP-LoRA
CB
77.86 + 4.24
81.43
3.91
84.64
77.50 = 5.30
75.71 + 2.04
RTE
57.69
1.10
74.73 3.044
7.87
2.85=1.44
74.08 2.57
SST-2
86.31 + 0.10
92.41
0.32
94.52+0.22
50.92 + 0.00
94.22 + 0.45
WIC
61.32 +.28
65.89
66.14
62.70 7.37
64.29 1.82
CoLA
16.27+0.33
58.75 + 1.70
57.95 1.95
57.29 + 2.98
58.21 1.55
MRPC
0.88
0.73
87.60
68.38 0.00
$7.79
WSC
63.46
63.46 = 0.00
03.46
63.46 0.68
63.46
BoolQ
64.66
78.69 0.27
19.00
77.59 = 0.39
77.67 1.57
21

---

10-
Tain accuracy
2200
100
accut acy
1400
2.8
1500
Accuracy
Nom Cless OT
10.80
0.7
1000
0.50
Classifer weight
Clasoffter weight
0. 47970
Number Reralons
Number epachs
(a) LP (CB)
(b) FT (CB)
Train accuracy
Train accuracy
CI0S werihht
69
Norm coossifee Weei ght
-1.00
2.8
Accurecy
0.0
0.50
1.00
0.7
9.00
0.6
0.5
Classifier weight
3.80
Clacsifier
0.00
10
40
20
50
Number apachs
Number epochs
(c) LoRA (CB)
(d) LoRA (RTE)
LN
01
3nnd
Norm classifeer weight
(c) After training (CB)
Figure 4: The increase in the norm of the classifier weight during training.
feauue
Fadtuce
Pretaut Inaire
(a) FT
(b) LP-FT
Figure 5: Small changes in feature and large changes in classifier weight during LP-FT. We visualize the t-SNE plot of the penultimate layer features and the classifier row vector of the model trained on the CB dataset. (a) The features after FT are clearly separated by class, while the classifier row vectors are plotted nearly the same place as the pre-trained model. (b) The features after LP-FT keep the structure of the pre-trained model, while the classifier row vectors are changed from the initialization.
22

---

Curuaier
Dirastoc
Prezui Hestee
(a)
(h) LP-LORA
Figure 6: The t-SNE plot of the penultimate layer features and the classifier row vector of the model trained with LoRA on the CB dataset.
A.4.2 Results of NTK analysis
Table 9 displays the kernel statistics, while Figure 7 Sssows shows the distribution singular values. Figure and Figure visually depict the trace norms of sub- matrices within the NTK matrix. For the kernel matrix NCXNC, we calculated the trace norms of the sub-matrix K (x,,, xx) E RCxC for each sample pair (x,, xT) in the training sets.
Figure X reveals consistent pattern in the FT-effective component of the NTK matrix across all datasets: pairs of identical samples in diagonal positions typically exhibit higher trace norms. This suggests that the FT-effective component more effective capturing relationships among samples compared to the pre-train-effective component, Additionally, in the CB dataset, certain sample pairs, particularly in classes I and 3, show notably high trace norms, indicating that the pre-trained model effectively differentiates between these class samples.
30P
10
Singular valuee
Singular
10'*
Pre-train
Pre-train E
FT
10113
10
LoRA
LORA
LP-LOR
LP-LORA
100
200
303
600
439
100
200
400
419
Index
Index
(a) RTE
(b) BoolQ
10
Singular value
10
Pre-train
10-0
FT
LP-FT
14
LP-LOAA
100
X56
300
400
439
Index
WIC
Figure 7: Singular value distribution normalized by the maximum singular value the RTE, BoolQ, and WiC datasets. Pre-train E denotes the pre-train-effective component, and other plots denote the FT-effective component of NTK matrix with each training option.
23

---

115
1005
4985
1005
34003
(a) Pre-train E (CB)
(b) FT E (FT, CB)
(c) FT E (LP-FT, CB)
3375
26
3475
200
(d) Pre-train E (RTE)
(e) FT (FT, RTE)
(I) FT (LP-FT, RTE)
72
2000
1900
25.0
3800
2500
44
20.0
(g) Pre-train E (BoolQ)
(h) FT (FT, BoolQ)
(i) FT E (LP-FT, BoolQ)
33000
33530
25
110000
11206
39000
20000
3000
20000
(i) Pre-train E (WiC)
(k) FT E (FT, WiC)
(1) FT E (LP-FT, WiC)
Figure 8: Heat map of NTK matrix the CB, RTE, BoolQ, and WiC dataset. We calculate the trace norm of the sub-matrix of the NTK matrix for each sample pair and visualize them grouped by class. Pre-train E and FT E refer to the pre-train-effective and FT-effective components of the NTK matrix.
24

---

Table 9: Kernel statistics on the RTE, BoolQ. and WiC datasets. FN, Acc, and FT Ratio denote the Frobenius norm, kernel regression accuracy, and contribution of the FT-effective component, respectively. Pre-train E and FT E refer to the pre-train-effective and FT-effective components of the NTK matrix.
Dataset
Method
Kernel
Rank
FN
Ace (train/test)
FT Ratio
Pre-train E
28
4.70 x 104
66.40/5 /51.20
FTE
488
1.29 x 104
96.60/ 96.60/53.40 /53.40
FT
0.2148
RTE
NTK
191
5.98 x 104
97.60/ 53..00
FT
432
2.51 x 10'
70.80/ /54.60
LoRA
0.0005
NTK
30
4.70 x 104
59.60/54 25.60/54.80
FTE
250
3.80 X 106
100.00/5) 000/5..20 51.20
LP-FT
0.9918
NTK
251
3.84 x 106
100.00 /51.20
LP-LoRA
FTE
243
7.60 x 10a
84.80/ 00/./..22 1..0
0.1942
NTK
103
5.26 x 104
88.00/51 /5.220
Pre-train E
32
4.48 x 104
53.60/51 0500/5..20 .0
FTE
495
1.24 x 104
100.00 00..00/56.00 /56.440 40
FT
0.2139
BoolQ
NTK
215
5.67 x 104
53.80/ /57 .020
FT
448
2.48 x 10"
53.60/ 000/5.125 .22
LoRA
0.0005
NTK
34
4.48 x 104
53.60/ 0000/7..5 .0
FTE
247
4.46 x 106
100.00/61 96006..60 160
LP-FT
0.9921
NTK
248
4.49 x 106
100.00/61 61.20
LP-LoRA
FT NTK
237
8.56 x 10a
68.80/ 660/63.60
0.2118
99
5.07 x 104
86.00/59.29
Pre-train E
16
4.81 x 104
66.00/54
FT
488
1.45 x 104
89.00 0900559.0
89.00/59.00
FT
0.2216
WiC
NTK
235
6.17 x 104
90.60/59 00.60/59.00
FT
438
2.58
10a
72.00/52 52.00
LoRA
0.0005
NTK
19
4.81 x 104
15.80/56.445
FTE
218
7.77 x 107
100.00/56 556.80
LP-FT
0.9996
NTK
219
7.77 x 107
100.00/56.44
LP-LoRA
FT E
218
1.09
103
72.00/59.66
0.7454
NTK
195
1.47 x 105
80.80/ /50.609
A.4.3 Experiments on BOSS benchmark
Table shows indicate that LP-FT surpasses FT OOD robustness and achieves higher accuracy in ID settings than LoRA. This suggests that LP-FT is effective in enhancing model robustness to OOD samples with reduced feature changes.
Table TT displays the statistics of feature and classifier changes on the Amazon, Dynasent, SemEval, and SST-5 datasets. The FDR within the is lower for L.P-FT than for FT, whereas the FDR for OOD is higher for LP-FT than for FT. This indicates that LP-FT is learning robust features that are less sensitive to OOD data.
A.4.4 Change of feature and classifier norms
Table 12 shows the changes in features during the FT stage, indicating that the changes are smaller during ..F.T compared to FT. Table shows the classifier norms, which increase during training, with noticeable increase observed during LP than during FT.
25

---

Table 10: Evaluation results BOSS benchmark. We report the average accuracy and standard deviation over five seeds. The best results are highlighted in bold.
ID
OOD
Method
Amazon
Dynasent
SemEval
SST-5
LP
83.04 0.01
42.69 + 0.05
50.04 = 0.01
56.81 + 0.11
FT
88.66 + 1.62
44.33 1.11
52.20
72.52 + 1.28
LoRA
86.05 2.16
46.70
55.29 + 2.93
72.88 + 1.84
LP-FT
88.89 + 1.02
45.41
51.96 = 2.72
73.78 + 1.05
LP-LoRA
88.17
43.37 + 1.50
48.84 = 3.20
72.31 + 1.30
Table 11: Comparison of feature and classifier changes the Amazon (ID), Dynasent, SemEval, and SST-5 (OOD) datasets. CS, Diff, FDR, and Norm denote cosine similarity, difference norm, Fisher's discriminant ratio, and norm, respectively. (F) and (C) indicate feature and classifier statistics. Averages were calculated over five seeds.
Amazon
Dynasent
Method
CS(F)
Difi(F)
FDR(F)
Norm(C)
CS(F)
Diff((F)
FDR(F)
Pre-trained
0.996
1.30 x 10
9.51 x 10-1 01.996
1.94 x 10"
LP
0.996
1.30
10
1.20
10
0.996
1.94
10"
FT
0.691
1.94 x 10
3.74 x 10
9.50 x 10
0.652
1.80 x 10¹
2.03 x 10"
LoRA
0.848
1.16 x 10
3.38
10
1.81 x 10"
0.855
7.53
10"
2.06
10"
LP-FT
0.999
2.27 x 10 3.00 x 10
1.20 x 10
0.998
2.54
x 10
2.20 x 10
LP-LoRA
0.999
2.24 x 10
3.01 x 109
1.18 x 10
0.999
2.56
10
2.04
10
SemEval
SST5
Method
CS(F)
Dif((F)
FDR(F)
CS(F)
Difi(T)
FDR(F)
Pre-trained
0.996
1.24 10" 0.998
1.69 x 10a
LP
0.996
1.24 x 10
0.998
1.69 x 10
FT
0.727
1.68
10
1.49 x 10
0.604
1.84 x 10
2.26 x 10
LoRA
0.885
6.74
10
1.44
10
0.837
x 10"
2.01 x 10a
LP-FT
0.997
2.06
10
1.45 x 10
0.998
1.86 x 10
2.02
x 10a
LP-LoRA
0.999
2.08
10
1.19
10
0.998
1.85
10
1.95
10a
Table 12: Feature change in FT stage. The change during LP-FT is smaller than during FT.
Dataset
FT
LoRA
LP-FT
LP-LoRA
CB
2.11 X 10
2.07 X 10
1.15 X 10'
7.85 x 10"
RTE
2.12 X 10
1.51 X 10
3.33 x 100
3.87 x 10º
COLA
1.91 101
1.10 101
3.05 x 10a
2.75 10"
SST-2
2.31 x 10a
3.78 x 10
6.95 x 10
2.17 x 10
MRPC
2.11 x 10a
1.80 x 10"
1.84 10
1.94 x 10"
BoolQ
2.23 10a
1.55 x 10a
2.31 10
1.95 10"
WiC
2.08 X 10a
1.04 10a
2.28 x 10
2.16 X 10
WSC
9.14 X 10"
x x 1
7.33 x 10
2.02 x 10
Amazon
1.98 10a
1.35 x 10a
2.21 x 10a
2.28 10"
Dynasent
1.94 X 10a
8.02 x 10"
2.47 x 10
2.59 X 109
SemEval
1.83 x 10
6.97 x 10
1.99 x 10
2.14 x 109
SST-5
2.03 10
9.08 10"
1.79 x 10
1.89 10
26

---

5.00
3305
2000
2340
3338
3300
-20
A150
(a) LoRA, CB
(b) LoRA, RTE
(c) LoRA, BoolQ
(d) LoRA, WIC
400
10
430
(c) LP-LORA, CB
(0) LP-LoRA, RTE
(g) LP-LoRA, BoolQ
(h) LP-LoRA, WIC
Figure 9: Heat map of NTK matrix of FT-effective component with LoRA on the CB, RTE, BoolQ, and WiC dataset with LoRA. We calculate the trace norm of the sub-matrix of the NTK matrix for each sample pair and visualize them grouped by class.
Table 13: The classifier weight norms. The classifier weight norms increase during training, and the increase is more pronounced LP.
Dataset
Pretrain
FT
LoRA
LP
LP-FT
LP-LORA
CB
9.47
10-1 9.51 x 10
1.56
10
3.35
10
3.35
10
3.35 x 10
RTE
7.95 x 10-1 8.05 x 10-1
1.45
x 10"
2.86
10a
2.86 x 10a
2.85 x 10a
COLA
7.95
10-1 7.88 x 10-1
1.06
10
3.46
10a
3.46 x 101
3.51 x 10a
SST2
7.95 10 7.20 x 10
1.96
10"
1.32
102
1.09
10
1.03 x 10a
MRPC
7.95 x 10-1 7.98 x 10-1
1.35 x 10"
1.12
x 10a
1.12 x 10a
1.12 X 10a
BoolQ
7.95 x 10-1 7.98 x 10-1
1.15 x 10
1.27
10a
1.27 x 10a
1.25 x 10a
WiC
7.95 10-1 7.98 x 10-1
1.14 10
3.21
10
3.25 10
3.27 x 10
WSC
7.95 x 10-1 6.87 x 10
7.88 x 10
2.26
10
1.08 x 10
2.16 x 10
Amazon 9.51 x 10 9.47 x 10
1.67 x 10"
1.21
102
1.21 X 102
1.20 x 102
A.4.5
Effects of classifier weight norms in training
(the CB and RTE datasets) illustrate the changes in features from the pre-trained models. Except for the CB dataset, the change in features in LP-FT is generally smaller than in FT when using large classifier norms. The CB dataset has smaller sample size, which could be an exception.
Figure 10 (Boss benchmark) and Figure I
A.4.6 Temperature scaling
The result of temperature scaling SuperGLUE and GLUE is presented in Tables 14 and 15
A.4.7 PubMed 20k
In addition to the natural language understanding benchmarks. we also evaluated LP-FT on the PubMed 20k RCT dataset to evaluate its effectiveness practical applications. The PubMed 20k RCT dataset, subset of PubMed 200k [Dernoncourt and Lee, 2017]. comprises 20,000 medical abstracts from randomized controlled trials, categorized into five classes. Efficient tools for navigating extensive medical literature essential for the medical community.
27

---

Norm the feature difference
Norm of the feature difference
20
10-
10-
FT
FT
LP-FT
LP-FT
10a1
10a
1o
162
10a
16a
of classifier weight
Norm classifier weight
Amazon (ID)
(b) Dynasent (OOD)
Norm the feature difference
15
10-
FT LP-FT
10r
10?
10.
Norm classifier weight
(c) SemEval (OOD)
Figure 10: Difference of features of the samples with scaling the classifier weight norms on BOSS benchmark. Solid lines show mean values; shaded areas represent standard errors. The dashed vertical lines indicate the original norms the classifier weight.
25
LP-FT
the Teature difference
Norm of the feature difference
LoRA
20
LP-LORA
15
LP-FT
LORA
LP-LORA
Norm classifier weight
Norm of classifier weight
(a)
(b) RTE
Figure 11: Difference of features of the samples with scaling the classifier weight norms the CB and RTE datasets. Solid lines show mean values; shaded areas represent standard errors. The dashed vertical lines indicate the original norms of the classifier weight.
28

---

Table 14: ECE and MCE with temperature scaling on SuperGLUE. wo TS and w/ TS denote without and with temperature scaling, respectively, and Imp. represents the improvement because of temperature scaling. We bold the best improvements. We take seeds and report the mean and the standard deviation.
Dataset
Metric
Method
w/o TS
w/ TS
Imp.
FT
15.60 1 0.96
14.64
1.75
0.95
LP-FT
13.93
1 0.45
13.13
0.56
0.80
ECE (%)
LoRA
12.89
0.41
16.22
0.55
33.4
LP-LoRA
14.78 + 0.98
13.51 1.67
1.27
CB
FT
75.99 6.12
69.99
5.83
6.01
LP-FT LoRA
76.78 + 3.66
70.28 + 3.27
6.50
MCE (%)
52.58 + 4.72
66.75 7.96
14.16
LP-LoRA
68. + 4.95
60.80 2.30
7.36
FT
21.16 + 1.36
5.13 0.63
16.03
LP-FT
21.72 + 0.28
5.48 + 0.77
16.24
ECE (%)
LoRA
11.92 + 2.23
6.17 0.20
5.76
LP-LoRA
18.14 0.99
5.72 + 0.48
12.42
RTE
FT
53.11 + 8.51
25.87 + 6.30
27.24
LP-FT
63.95 1 7.70
13.94 1.80
50.01
MCE (%)
LoRA
25.04 1 3.33
13.75 1 0.91
11.29
LP-LoRA
40.46 1 7.22
18.82 1: 2.00
21.63
FT
13.63 + 0.61
1.83 0.09
11.81
LP-FT
18.93 + 0.15
2.41
16.51
ECE (%)
LoRA
0.38
1.45 0.18
7.43
LP-LoRA
14.09
+ 0.92
2.07 0.19
12.02
BoolQ
FT
23.26 + 1.48
5.79 0.90
17.47
LP-FT
40.82 + 1.94
5.21
5.21+0.0. 0.53
35.60
MCE (%)
LoRA
13.96 + 0.72
3.85 0.56
10.11
LP-LORA
24.60 + 2.52
5.51 0.72
19.09
FT
25.88 2.39
8.85 0.53
17.03
LP-FT
29.47 + 1.57
7.68 + 0.55
21.78
ECE (%)
LoRA
18.66 + 4.39
5.93 1.42
12.73
LP-LoRA
22.22 1: 1.98
8.06 1 0.60
14.15
WIC
FT
41.59 + 5.39
17.01 2.87
24.58
LP-FT
39.20 + 2.74
17.04 1.50
22.16
MCE (%)
LoRA
27.95
4 7.38
11.40
16.54
LP-LoRA
30.99
3.64
14.45 1.01
16.54
FT
6.26
7.97
0.06
-171
LP-FT
6.38
1.78
8.01 0.06
-163
ECE (%)
LORA
10.53 + 1.35
9.19 0.60
1.34
IP-LORA
11.40 + 0.23
+.2400.00
3.15
WSC
FT
6.26 + 2.37
7.97 + 0.06
-171
LP-FT
6.38 + 1.78
8.01 1 0.06
1.63
MCE (%)
LoRA
13.27
+ 1.12
11.12 1.51
2.15
LP-LoRA
11.40 + 0.23
8.24 0.01
3.15
The results are presented in Table 16 The LoRA model outperforms other models, although the performance of FT. LP-FT, and LoRA models relatively similar.
29

---

Table 15: ECE and MCE with temperature scaling on GLUE. w/o TS and w/ TS denote without and with temperature scaling, respectively, and Imp. represents the improvement because of temperature scaling. We bold the best improvements. We take 5 seeds and report the mean and the standard deviation.
Dataset
Metric
Method
w/o TS
w/ TS
Imp.
FT LP-FT LoRA
15.08 0.55
4.46 = 0.83
10.61
0.40
9.53 = 1.23
6.21
ECE (%)
11.25 1.32
4.18
7.07
LP-LoRA
13.82 0.48
4.30
9.52
CoLA
FT
47.19
24.35
= 3.33
22.84
LP-FT LoRA
54.59
2.94
20.31
= 1.37
34.28
MCE (%)
31.01 5.83
15.23
+ 2.74 15.78
LP-LoRA
38.36 + 7.85
15.36
1.83
23.00
FT
4.61
2.26
2.35
LP-FT
5.67 0.12
2.00
3.66
ECE (%)
LoRA
4.14
2.71
2.12
LP-LoRA
6.22
2.53
3.69
SST-2
FT
49,22
2
6.50
LP-FT
74.91 1.72
42.77 5.75
32.13
MCE (%)
LoRA
$4.20
36.58 = 5.82
17.63
LP-LoRA
7.917
32.47 = 3.74
38.65
FT
10.71
4.61 4.61 1612 0.24
6.10
LP-FT LoRA LP-LORA
10.35
3.68
6.68
ECE (%)
4.04 0.68
2.54
0.85
3.89
5.14
MRPC
FT
7.93
32.72 = 1.69
29.12
LP-FT
74.43
22.73 = 17.57 = 2.00
51.70
MCE (%)
LoRA
28.80 5.05
11.23
LP-LORA
52.20 + 6.64
22.76 + 7.60
29.44
Table 16: Test accuracy on PubMed 20k.
LP
FT
LP-FT
LoRA
LP-LORA
82.64 + 0.02
87.09 + 0.17
87.05 0.11
87.13 + 0.09
86.85 0.07
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that our paper analyzes a fine-tuning method, specifically linear probing then fine-tuming from neural tangent kernel perspective. The abstract succinctly summarizes the main contributions, and the introduction provides a thorough overview of the paper's scope with motivation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explicitly discuss the limitations of our theoretical analysis in the limitations section of our paper, highlighting the need for further investigations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
31

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We clearly state our assumptions alongside the propositions and provide complete proofs in the appendix. This ensures that our theoretical results are well-supported and verifiable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all essential details needed to replicate our main experimental results within the paper. This includes hyperparameters and data splits to ensure that our findings are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
32

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will release the code and associated data before the review process begins. This release will include comprehensive instructions to ensure faithful reproduction of our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
public/gute Souiides/CodeSubissiiioo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" " an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /nipsscccpublic/uuiddee htw.bl//ccguidde//ddeebbbbbiiiia www.oodeeumiisiiooooo.ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail all necessary training parameters, including data splits and hyperpa- rameters, to ensure that our experimental results can be faithfully reproduced.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars and standard deviations in our results where applicable, ensuring that the statistical significance of our findings is clear and well-documented.
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed descriptions of the computational resources used, including hardware
specifics and implementation details, provided in the Appendix to aid in reproducing experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttp:://nenueii htt..////b....h.cccciieeeeeeeeee
Answer: [Yes]
Justification: After thoroughly reviewing the NeurIPS Code of Ethics, we confirm that our research adheres all the specified guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
34

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given the theoretical nature of our work, we assess that it does not directly engage with societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not involve the release of data or models that pose high risks for misuse, hence specific safeguards are not required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: Our study does not use any external assets, thus no licensing or attribution issues applicable.
35

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are introduced in our paper, there are associated documen- tation requirements.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
36

---

Answer:
[NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "lQ08ILAnZa", "reviewer": "Reviewer_zDRZ", "summary": "The paper presents a novel application of neural tangent kernel (NTK) theory to analyze the training dynamics of the linear probing then fine-tuning (LP-FT) method for large language models, demonstrating its effectiveness and extending the analysis to include the low-rank adaptation (LoRA) method.", "questions": "Can you expand on the practical applications of your findings? More specific examples of how practitioners can apply your recommendations in real-world scenarios would enhance the practical value of your work.", "limitations": "he paper does not discuss the computational resources required for implementing LP-FT and NTK analysis. Providing information on the computational cost and efficiency of the proposed methods would help readers understand the practical feasibility of adopting these techniques.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper stands out for its originality by applying NTK theory to the LP-FT method and extending it to include an analysis of LoRA. It's strong in terms of quality because it combines solid theoretical work with thorough experiments to back up the claims. The writing is clear and well-organized, making it easy to follow the arguments and findings. The significance of the research is evident in its potential impact on fine-tuning practices, not just in NLP but across various fields that use large pre-trained models. Overall, this paper makes a valuable contribution to advancing our understanding and methods in transfer learning.", "weaknesses": "One weakness of the paper is that while it presents a solid theoretical foundation and empirical evidence, it could benefit from more detailed explanations in certain sections, particularly the experimental setup and hyperparameter tuning. This would enhance transparency and reproducibility. Additionally, the paper could further strengthen its claims by including a broader range of experiments across different domains beyond NLP. Another point for improvement is the discussion of the impact and practical applications of the findings, which could be expanded to provide more actionable insights for practitioners. Lastly, while the paper briefly mentions temperature scaling to address calibration issues, a deeper exploration of this aspect could provide more comprehensive recommendations for improving model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time and effort to review our paper. We appreciate your valuable advice and suggestions for improvements.\n> W1: One weakness of the paper is that while it presents a solid theoretical foundation and empirical evidence, it could benefit from more detailed explanations in certain sections, particularly the experimental setup and hyperparameter tuning. This would enhance transparency and reproducibility.\n\nA: We have included detailed descriptions of the experimental setup and hyperparameter tuning in Section 7.3 of the Appendix. Additionally, we have made our implementation available on GitHub, though we cannot provide the specific URL at this time due to the anonymity of the review process. Our implementation uses the HuggingFace Transformers library and AdapterHub. For hyperparameter tuning, we conducted a grid search on the validation set. These details are intended to enhance transparency and reproducibility. We agree that transparency and reproducibility are crucial for academic papers.\n> W2: Additionally, the paper could further strengthen its claims by including a broader range of experiments across different domains beyond NLP. \n\nA: The effectiveness of LP-FT in the vision domain has already been validated by Kumar et al. [1], the original authors of LP-FT. Our focus is on applying LP-FT to fine-tuning language models, addressing the current demand in the NLP domain. This is why our experiments are limited to the NLP domain.\n> W4: Lastly, while the paper briefly mentions temperature scaling to address calibration issues, a deeper exploration of this aspect could provide more comprehensive recommendations for improving model performance.\n\nA: We conducted additional experiments of temperature scaling using 8 datasets from the SuperGLUE and GLUE benchmarks, as shown in Tables 14 and 15 in the Appendix. These experiments demonstrate that temperature scaling significantly improves ECE and MCE, particularly with LP-FT and LP-LoRA. We agree that a deeper exploration of temperature scaling to enhance calibration is crucial for practical applications.\n> W3: Another point for improvement is the discussion of the impact and practical applications of the findings, which could be expanded to provide more actionable insights for practitioners.\n\n> Q1: Can you expand on the practical applications of your findings? More specific examples of how practitioners can apply your recommendations in real-world scenarios would enhance the practical value of your work.\n\nA: Our findings have two key practical applications:\n1. LP-FT for language models: LP-FT is an effective method for fine-tuning language models, as it minimizes changes to valuable pre-trained features.\n1. Enhancing calibration: We recommend combining LP-FT with temperature scaling to improve calibration.\n\nTo demonstrate practical effectiveness, we conducted experiments using the PubMed 20k medical dataset [2], as detailed in Section 7.4.7 and Table 16 of the Appendix. This example provides concrete scenarios for practitioners to apply LP-FT in real-world settings.\n\n[1] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022.\n\n[2] Franck Dernoncourt and Ji Young Lee. Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Md8v6jXjy", "reviewer": "Reviewer_R1ei", "summary": "This paper studies the dynamics of Linear probing and fine tuning by means of NTK theory. The authors provide a connection between the  Frobenius norm of  linear probing weights and FT-effective component of the NTK matrix.", "questions": "In Fig.3, What does “number of feature difference”? please provide the definition.\n\n-How did you calculate the NTK matrix for numerical simulations?  There is a body of literature for only  approximation of NTK for transformers.\n\n-Did you use empirical NTK for simulations? if yes, then can you say if your results hold in the finite width regime too?", "limitations": "NTK theory in general holds in the infinite width limit. Further discussions are needed to adapt the methods in this paper to finite regime.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The NTK  sees the model as an  Gaussian process, making it a  powerful tool for analysis of neural networks’ convergence and generalization. This paper  derives an interesting connection between NTK  theory and  fine tuning language models. The NTK is decomposed into  pre-trained and fine tuned terms.  Feature distortion theory is employed to interpret performance of FT.", "weaknesses": "-The organization of  material   can be improved. for instance, you  used the linear model $f(x)$ in Proposition 1, however it is introduced later in Definition 1. \n\n-Table 2 is very uninformative and the numbers are not in line, making it hard to read. please provide  precise description of each table and  figure in the text. \n\n-Same for Fig.3, some numbers are floating on it and the message of the figure is not clear.\n\nplease  further enhance the presentation of  results,  I found it hard to  infer  your  achievements from the numerical results section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time and effort to review our paper. We appreciate your valuable advice and suggestions for improvements.\n\n> W1: The organization of material can be improved. for instance, you used the linear model  in Proposition 1, however it is introduced later in Definition 1.\n\nA: The model in Proposition 1 is defined on line 136 in Section 4.1. That is, the linear model in Defition 1 is different from one in Proposition 1. However, since this is confusing, we changed the definition of the linear model in Definition 1. We now define the linear model as $f_{\\text{linear}}(x) = VBx + b$, i.e., $\\phi(x)=Bx$, which is a special case of the general model form $f(x) = V\\phi(x) + b$ defined on line 136 and used in Proposition 1.\n> W2: Table 2 is very uninformative and the numbers are not in line, making it hard to read. please provide precise description of each table and figure in the text.\n\n> W3: Same for Fig.3, some numbers are floating on it and the message of the figure is not clear.\n\nA: Thank you for pointing out the formatting issues and the need for clearer descriptions. We recognize the importance of clear and precise presentation and will make appropriate revisions.\n\n### Formatting Issues in Table 2 and Figure 3\n\nI understand that \"293\" in Table 2 and \"328\" in Figure 3, which reference line numbers in the preprint version, have caused confusion. We will ensure these numbers are removed in the camera-ready version of the manuscript to prevent any misinterpretation.\n\n### Interpretation of Table 2\n\n- The table shows that the FT-effective component outperforms the Pre-train effective component in terms of rank and kernel regression accuracy. This suggests that the FT-effective component has greater expressiveness.\n- The superior performance of LP-FT over the standard method is indicated by higher kernel regression accuracy. The Frobenius norm and the FT ratio suggest that the significant contribution from the FT-component is the reason for this.\n\nWe have identified a typo in the Frobenius norm of the LP-LoRA, which should be 15.1, not 1.51. We will correct this in the camera-ready version.\n\n### Interpretation of Figure 3\n- Figure 3 shows the inverse relationship between the norm of the classifier weight and the norm of the feature difference. This suggests that the large classifier weight norm in LP-FT reduces the feature changes. \n- The lines in the figure represent the mean values, while the shaded areas indicate the standard errors. We will clarify this point in the caption in the camera-ready version of the paper.\n\n### Conclusion from the numerical results section\nOverall, we validate the following three points in the numerical results section:\n1. The changes in features during training are smaller in LP-FT than in FT, and the norms of the classifier significantly increase during LP.\n1. The FT-effective component of the NTK matrix more effectively captures the input data than the pre-train-effective component and is more pronounced in LP-FT than FT.\n1. A large classifier weight norm reduces the feature change during training, and its negative effects on calibration can be improved by temperature scaling.\n\n> Q1: In Fig.3, What does “number of feature difference”? please provide the definition.\n\nA: That is not **\"number\"**, is **\"norm\"**. We understand this might be confusing, and we will change the font size of the figure caption to make it a little more understandable.\n\nIn Figure 3, we measure the L2 norm of the difference between features extracted from the trained model and those extracted from the pretrained model. Specifically, for a pre-trained feature extractor $\\phi_0$, a trained feature extractor $\\phi$, and training examples $x_i (i=1,\\cdots N)$, we measure $\\frac{1}{N}\\sum_{i=1}^N \\| \\phi(x_i) - \\phi_0(x_i)\\|_2$.    \n\n> Q2: How did you calculate the NTK matrix for numerical simulations? There is a body of literature for only approximation of NTK for transformers.\n\nA: We describe these points in section 7.3.3 in the Appendix. \n\nWe separately calculated the pre-train-effective and FT-effective components of the NTK matrix. Following the methodology by Malladi et al. [1], we used functorch and forward-mode auto-differentiation for these calculations. To reduce computational costs, we randomly selected 10% of the parameters from the word embedding matrix for derivative calculations. For datasets with more than 250 samples, we used a subset of 250 randomly selected samples to compute the NTK matrix.\n\n> Q3: Did you use empirical NTK for simulations? if yes, then can you say if your results hold in the finite width regime too?\n\n> L1: NTK theory in general holds in the infinite width limit. Further discussions are needed to adapt the methods in this paper to finite regime.\n\nA: Yes, we used empirical NTK in our experiments. \n\nWe assume that the fine-tuning dynamics of large Transformer models is explained with NTK. Although Transformer models have a finite number of parameters, the number is extremely large, and changes in parameters during fine-tuning are smaller compared to standard training.\n\nThe empirical NTK has been successfully used to analyze fine-tuning in the finite width regime in previous studies [1, 2]. Therefore, we applied the NTK to analyze the LP-FT method with Transformer models.\n\n[1] Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610–23641. PMLR, 2023.\n\n[2] Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learning, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdI0vudBy0", "reviewer": "Reviewer_PtuD", "summary": "The authors present a theoretical analysis of the linear probing and fine-tuning framework based on neural tangent theory, supported by experiments with transformer-based models on natural language processing benchmarks. Their analysis decomposes the NTK matrix into the FT-effective component and the pre-train-effective component, demonstrating an approximation between the fine-tuning matrix and the LoRA matrix.", "questions": "Minors: In Figures 3 and 11, it would be helpful to explicitly indicate the purpose of the shaded area in the caption to enhance readability.", "limitations": "The authors have made a clear statement about the limitations of the proposed analysis.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is very well written, with detailed technical content. The authors have provided enough mathematical details to support their theoretical analysis. The proofs for theorems and corollaries are detailed delineated in the appendix.   \n\n2. A timely topic that bridges established theory and recent advancements in natural language processing.\n\n3. The authors have provided comprehensive benchmarks to support the claims of the paper.", "weaknesses": "Although the experiments included an exhaustive list of benchmarks, I noticed that you only implemented one transformer model in your setup. I believe it would be more effective to include multiple types of transformer-based models to better illustrate the practical advantages of your approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time and effort to review our paper. We appreciate your valuable suggestions for improvements.\n> W1: Although the experiments included an exhaustive list of benchmarks, I noticed that you only implemented one transformer model in your setup. I believe it would be more effective to include multiple types of transformer-based models to better illustrate the practical advantages of your approach.\n\nA: We show the results of the T5 model [1] below. \n\nIn the main text, we conducted experiments using the RoBERTa model, which is an encoder model. To provide a broader perspective, we included experiments with the T5 model, an encoder-decoder model.\n\nThe results from the T5 experiments demonstrate the following:\n1. Smaller feature changes in LP-FT.\n1. A significant increase in classifier weight norms during LP.\n1. Effectiveness of LP-FT.\n\nThese findings further support the validity of our analysis presented in the main text.\n\n### Norm of the feature difference from the pre-trained T5 model\n| Dataset | FT           | LoRA         | LP-FT        | LP-LoRA      |\n|---------|--------------|--------------|--------------|--------------|\n| CB      | 2.77×10¹     | 1.72×10¹     | 9.51×10⁰     | 6.36×10⁰     |\n| RTE     | 1.46×10¹     | 1.43×10¹     | 1.18×10¹     | 7.46×10⁰     |\n\n### Classifier weight norms of T5 model\n| Dataset | Pretrain       | FT             | LoRA           | LP             | LP-FT          | LP-LoRA        |\n|---------|----------------|----------------|----------------|----------------|----------------|----------------|\n| CB      | 9.97×10⁻¹      | 1.36×10⁰       | 1.51×10⁰       | 1.21×10¹       | 1.21×10¹       | 1.23×10¹       |\n| RTE     | 8.24×10⁻¹      | 1.01×10⁰       | 1.83×10⁰       | 3.69×10⁰       | 1.08×10¹       | 1.14×10¹       |\n\n\n### Test accuracy of T5 model\n| Dataset | LP             | FT             | LoRA           | LP-FT          | LP-LoRA        |\n|---------|----------------|----------------|----------------|----------------|----------------|\n| CB      | 74.40 ± 2.73   | 82.14 ± 3.09   | **84.52 ± 7.22**   | **84.52 ± 2.06**   | 81.55 ± 1.03   |\n| RTE     | 58.00 ± 1.46   | 73.89 ± 3.28   | **76.17 ± 0.96**   | 75.09 ± 1.30   | 74.97 ± 1.85   |\n> Q1: Minors: In Figures 3 and 11, it would be helpful to explicitly indicate the purpose of the shaded area in the caption to enhance readability.\n\nA: Thank you for pointing this out. \nIn Fig.3 and 11, we added the following sentence in the current version:\n\n**\"Shaded areas represent standard errors.\"**\n\n[1] Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via\nlayer-peeled model: Minority collapse in imbalanced training. Proceedings of the National\nAcademy of Sciences, 118(43), October 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAaBQNPnpi", "reviewer": "Reviewer_qfrt", "summary": "The authors analyze the training dynamics of LP-FT for classification models based on NTK theory. They decomposed NTK matrix and highlight the importance of linear head norm. Additionally, they highlight the increased linear head norms can negatively affect the model calibration and can be fixed by temperature scaling. Finally, they extend their analysis to the LoRA method.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The application of NTK theory to analyze LP-FT in complex models like Transformers is interesting.\n2. The theoretical analysis is thorough, well-supported by mathematical proofs, and aligns with empirical observations.", "weaknesses": "1. The experiments primarily focus on the classification tasks. Since the authors also extend their analysis to LoRA, exploring additional domains and harder tasks (reasoning, math, code generation) combined with LLMs could siginificantly strengthen the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time and effort to review our paper. We appreciate your insightful question.\n> Q1: The experiments primarily focus on the classification tasks. Since the authors also extend their analysis to LoRA, exploring additional domains and harder tasks (reasoning, math, code generation) combined with LLMs could significantly strengthen the generalizability of the findings.\n\nA: We conducted experiments on question-answering tasks with the RoBERTa model on the SQuAD1.1 [1] and SQuAD2.0 [2] datasets. The following results show the following:\n1. Smaller feature changes in LP-FT.\n1. A significant increase in classifier weight norms during LP.\n1. Effectiveness of temperature scaling on LP-FT.\n\nThese points validate our analysis. Although the F1 score of LP-FT on SQuAD1.1 is lower than that of standard fine-tuning (FT), this result is influenced by the choice of hyperparameters and could improve with further optimization.\n\n### F1 score on test set\n| Dataset | LP             | FT             | LoRA           | LP-FT          | LP-LoRA        |\n|---------|----------------|----------------|----------------|----------------|----------------|\n| SQuAD1.1   | 24.14 ± 0.15   | 91.72 ± 0.05   | 91.57 ± 0.07   | **91.80 ± 0.09**   | 91.64 ± 0.06   |\n| SQuAD2.0   | 22.69 ± 0.21   | **82.50 ± 0.44**   | 80.74 ± 1.56   | 81.31 ± 0.50   | 80.72 ± 0.22   |\n\n### Classifier weight norms\n| Dataset | Pretrain     | FT          | LoRA        | LP          | LP-FT       | LP-LoRA     |\n|---------|--------------|-------------|-------------|-------------|-------------|-------------|\n| SQuAD1.1   | 7.82e-01     | 8.50e-01    | 8.72e-01    | 1.21e+01    | 1.21e+01    | 1.21e+01    |\n| SQuAD2.0   | 7.82e-01     | 8.41e-01    | 8.71e-01    | 1.26e+01    | 1.26e+01    | 1.26e+01    |\n\n### Norm of the feature difference from the pre-trained model\n| Dataset | FT          | LoRA        | LP-FT       | LP-LoRA     |\n|---------|-------------|-------------|-------------|-------------|\n| SQuAD1.1   | 7.50e+00    | 7.49e+00    | 2.92e+00    | 2.94e+00    |\n| SQuAD2.0   | 7.59e+00    | 7.49e+00    | 3.17e+00    | 3.19e+00    |\n\n### Effect of temperature scaling on the SQuAD1.1 dataset\n| Metric | Method   | w/o TS | w/ TS | Imp(%) |\n|--------|----------|--------|-------|--------|\n| ECE(%) | FT       | 15.94  | 8.09  | 49.25  |\n|        | LoRA     | 14.85  | 8.82  | 40.63  |\n|        | LP-FT    | 14.77  | 7.21  | **51.17**  |\n|        | LP-LoRA  | 14.97  | 7.33  | 51.05  |\n| MCE(%) | FT       | 25.64  | 14.06 | 45.17  |\n|        | LoRA     | 21.76  | 15.01 | 31.03  |\n|        | LP-FT    | 22.14  | 12.20 | 44.88  |\n|        | LP-LoRA  | 23.40  | 12.59 | **46.20**  |\n\n### Effect of temperature scaling on the SQuAD2.0 dataset\n\n| Metric | Method   | w/o TS | w/ TS | Imp(%) |\n|--------|----------|--------|-------|--------|\n| ECE(%) | FT       | 7.42   | 1.67  | 77.50  |\n|        | LoRA     | 5.96   | 1.66  | 72.15  |\n|        | LP-FT    | 5.56   | 1.10  | **80.12**  |\n|        | LP-LoRA  | 4.68   | 1.24  | 73.46  |\n| MCE(%) | FT       | 13.66  | 5.23  | 61.73  |\n|        | LoRA     | 10.43  | 5.89  | 43.51  |\n|        | LP-FT    | 11.73  | 4.04  | **65.55**  |\n|        | LP-LoRA  | 9.22   | 5.68  | 38.43  |\n\n[1] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\n[2] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses NTK theory to analyze the training dynamics of the two-stage “linear probing then fine-tuning” (LP-FT) method, and tries to explain why LP-FT consistently outperforms LP or FT alone for both ID and OOD data. This analysis decomposes the NTK matrix into two components, highlighting the importance of the linear head norm alongside the prediction accuracy at the start of the FT stage.

The reviewers agree that this paper presents a solid theoretical contribution. The empirical evidence adds to the contribution of this paper. 

The concerns mentioned by the reviewers are mostly about the presentations. A reviewer asked for experiments on additional models, and the results on T5 model were added during the discussion period. The same applies to SQuAD, which also strengthened the generalizability of the finding.
Decision: Accept (poster)