Title: Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: fine-tuning

transfer learning

neural tangent kernel
Abstract: The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (LP-FT), outperforms linear probing and FT alone. This holds true for both in-distribution (ID) and out-of-distribution (OOD) data. One key reason for its success is the preservation of pre-trained features, achieved by obtaining a near-optimal linear head during LP. However, despite the widespread use of large language models, there has been limited exploration of more complex architectures such as Transformers. In this paper, we analyze the training dynamics of LP-FT for classification tasks on the basis of the neural tangent kernel (NTK) theory. Our analysis decomposes the NTK matrix into two components. This decomposition highlights the importance of the linear head norm alongside the prediction accuracy at the start of the FT stage. We also observe a significant increase in the linear head norm during LP, which stems from training with the cross-entropy (CE) loss. This increase in the linear head norm effectively reduces changes in learned features. Furthermore, we find that this increased norm can adversely affect model calibration, which can be corrected using temperature scaling. Additionally, we extend our analysis with the NTK to the low-rank adaptation (LoRA) method and validate its effectiveness. Our experiments using a Transformer-based model on multiple natural language processing datasets confirm our theoretical analysis. Our study demonstrates the effectiveness of LP-FT for fine-tuning language models. Code is available at https://github.com/tom4649/lp-ft_ntk.

Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective

Abstract

The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (LP-FT), outperforms linear probing and FT alone. This holds true for both indistribution (ID) and out-of-distribution (OOD) data. One key reason for its success is the preservation of pre-trained features, achieved by obtaining a near-optimal linear head during LP. However, despite the widespread use of large language models, there has been limited exploration of more complex architectures such as Transformers. In this paper, we analyze the training dynamics of LP-FT for classification tasks on the basis of the neural tangent kernel (NTK) theory. Our analysis decomposes the NTK matrix into two components. This decomposition highlights the importance of the linear head norm alongside the prediction accuracy at the start of the FT stage. We also observe a significant increase in the linear head norm during LP, which stems from training with the cross-entropy (CE) loss. This increase in the linear head norm effectively reduces changes in learned features. Furthermore, we find that this increased norm can adversely affect model calibration, which can be corrected using temperature scaling. Additionally, we extend our analysis with the NTK to the low-rank adaptation (LoRA) method and validate its effectiveness. Our experiments using a Transformer-based model on multiple natural language processing datasets confirm our theoretical analysis. Our study demonstrates the effectiveness of LP-FT for fine-tuning language models. Code is available at https://github.com/tom4649/lp-ft_ntk .

1 Introduction

Fine-tuning pre-trained models for new tasks is a common practice across various fields. However, simply fine-tuning the entire model can lead to overfitting on training data, which may negatively impact generalization and out-of-distribution (OOD) performance [Li et al., 2020, Lee et al., 2023]. To address this, the two-stage approach known as linear probing then fine-tuning (LP-FT) [Kumar et al., 2022] has demonstrated high performance on both in-distribution (ID) and OOD data. Initially, linear probing (LP) optimizes only the linear head of the model, after which fine-tuning (FT) updates the entire model, including the feature extractor and the linear head. This method has been extensively analyzed and enhanced [Trivedi et al., 2023, Ren et al., 2023, Ha et al., 2024, Kirichenko et al., 2023].

The feature distortion theory, introduced by Kumar et al. [2022], explains the effectiveness of LP-FT on the basis of a theoretical analysis with a two-layer linear model. This theory suggests that LP-FT minimizes changes to pre-trained features by starting FT with an already optimized linear head from LP. However, our understanding of LP-FT, particularly when applied to complex architectures such as Transformers [Vaswani et al., 2017], remains incomplete. Thus, it is crucial to further explore the training dynamics of LP-FT in more complex models than the two-layer linear model.

In this paper, we apply the neural tangent kernel (NTK) theory [Jacot et al., 2018] to clarify the mechanisms underlying LP-FT, focusing on the training dynamics of classification models. The NTK is a theoretical tool that analyzes training dynamics by applying a first-order approximation

Issei Sato

to changes in the model outputs with respect to its parameters. Therefore, the NTK is suited for analyzing feature changes during FT dynamics [Wei et al., 2022, Malladi et al., 2023]. Our analysis reveals that after LP, both more accurate predictions and increased norms of the linear head compared to their initial values contribute to minimizing feature changes. We then identify a significant increase in the linear head norm during LP from the analysis of training with cross-entropy (CE) loss, which contributes to small feature changes in the FT stage. On the other hand, we found that this increase in the linear head norm can worsen calibration, causing predicted probabilities to deviate from actual probabilities, which can be corrected with temperature scaling [Guo et al., 2017]. Furthermore, we extend our analysis based on the NTK to the low-rank adaptation (LoRA) method [Hu et al., 2022], a parameter-efficient fine-tuning strategy, and validate its effectiveness.

Our contributions are summarized as follows:

We show that both accurate predictions and increased norms of the linear head during LP reduce feature changes in LP-FT within the NTK regime (Section 4), which is consistent with the feature distortion theory. (Corollary 4.3).

Wefind that norms of the linear head significantly affect the balance of the NTK matrix components and influence the training dynamics of FT (Proposition 4.1).

We also highlight that increased linear head norms can negatively affect model calibration, and this can be fixed with temperature scaling.

We extend our analysis based on the NTK to the LoRA method and provide a theoretical validation of its efficacy (Proposition 4.4).

2 Related work

LP-FT FT and LP are well-established transfer learning techniques with extensive empirical and theoretical studies [Zhuang et al., 2020, Kornblith et al., 2019, Tripuraneni et al., 2020]. Kumar et al. [2022] analyzed the effectiveness of these techniques using a two-layer linear model. Then, they proposed LP-FT that is a combined approach of LP then FT. Building on this study, subsequent studies have explored LP-FT in more detail. Trivedi et al. [2023] investigated LP-FT through the lens of safety objectives, proposing modifications to mitigate simplicity bias. Ren et al. [2023] analyzed LP-FT from the perspective of the initial discrepancy between predicted and actual probabilities, emphasizing the importance of the number of probing epochs during LP. Ha et al. [2024] further improved LP-FT by aligning batch normalization layers with the target domain. Kirichenko et al. [2023] highlighted the challenge that models depend on spurious features and proposed last-layer retraining as a cost-effective strategy to improve model robustness.

Other FT methods Various FT strategies other than LP-FT have been proposed, including twostage approaches [Zhang et al., 2020], regularization-based techniques [Jiang et al., 2019], and parameter-efficient fine-tuning methods [Houlsby et al., 2019, He et al., 2022]. One prominent example of a parameter-efficient method is LoRA, proposed by Hu et al. [2022]. This approach draws inspiration from the concept of intrinsic dimensions [Aghajanyan et al., 2021], suggesting that data can be effectively represented in a lower-dimensional space. Zeng and Lee [2024] explored the expressive power of LoRA, and Jang et al. [2024] provided a theoretical analysis of its convergence properties. However, challenges remain in parameter-efficient FT methods, including potential instability issues identified by Chen et al. [2022].

Neural tangent kernel (NTK) The NTK, which was first introduced by Jacot et al. [2018], has become a valuable tool for analyzing the training dynamics of neural networks. Studies by Lee et al. [2019] and Arora et al. [2019] used the NTK to gain insights into how networks learn. Building on this foundation, Wei et al. [2022] introduced the concept of the empirical NTK, which extends the application of NTK to FT scenarios. This approach replaces the randomly initialized parameters in the standard NTK with the parameters of the pre-trained models. Further expanding on the empirical NTK, Malladi et al. [2023] conducted a theoretical and experimental investigation and found that prompt-based fine-tuning exhibits behavior consistent with the predictions of the kernel framework. Jang et al. [2024] extended this perspective to analyze LoRA.

3 Preliminary

In this section, we provide an overview of the FT methods used in this paper, followed by a brief explanation of the NTK.

LP-FT In standard FT, the parameters of the linear head, weight V and bias b , are initialized with random values. In contrast, in LP-FT, LP is conducted before the FT stage, and the FT stage is started with the obtained parameters. The performance of LP-FT is higher than that of LP and FT on both ID and OOD data [Kumar et al., 2022]. The original LP-FT paper [Kumar et al., 2022] explains the reason behind it as the feature distortion theory: the success of LP-FT stems from the minimal feature changes because of starting the FT stage with the linear head parameters which are close to the optimal solution. We analyze the training process of LP-FT throughout this paper.

LoRA LoRA [Hu et al., 2022] introduces trainable rank decomposition matrices into each layer of the Transformer architecture. This approach, inspired by the concept of 'intrinsic dimensions" from Aghajanyan et al. [2021], constrains updates to pre-trained weight matrices via low-rank decomposition. The update of a pre-trained weight matrix W 0 ∈ R q × s is approximated by W + ∆ W = W 0 + B LoRA A LoRA , where B LoRA ∈ R q × r and A LoRA ∈ R r × s are the only matrices optimized during fine-tuning. Here, r ≪ min( q, s ) represents the small intrinsic rank of the weight matrix, reflecting the low-rank approximation. The standard initialization of B LoRA and A LoRA is B LoRA = O and A LoRA is drawn from a normal distribution with mean 0 .

Neural tangent kernel (NTK) Jacot et al. [2018] introduced the NTK, which captures the training dynamics over time. They demonstrated that in the infinite width limit, the NTK remains constant. In this limit, training dynamics are governed by a linear model derived from a first-order Taylor expansion around the initial parameters of the network, known as the linearized or NTK regime [Lee et al., 2019]. For networks with finite width, this limiting kernel depends on the initialization parameters and is known as the empirical NTK [Wei et al., 2022]. Although the empirical NTK differs from the infinite width limit, it is valuable for analyzing the local training dynamics of models [Ren et al., 2022, Fort et al., 2020, Mohamadi and Sutherland, 2023, Wei et al., 2022, Jang et al., 2024], and has been used in FT [Ren et al., 2023, Malladi et al., 2023].

4 Analysis of LP-FT from NTK perspective

The original analysis of LP-FT by Kumar et al. [2022] is based on a two-layer linear model and proposes the feature distortion theory, which suggests that minimal changes in pre-trained features are the reason behind the robust performance of LP-FT. In this section, we use the NTK theory to analyze LP-FT to better understand the training dynamics of LP-FT in complex models like Transformers. After introducing the notation, we discuss the increase in the classifier weight norm during training, followed by the training dynamics in the NTK regime. We then extend our analysis to the LoRA method. These analyses suggest the LP-FT reduces feature distortion with the increased norm of the classifier weight and the near-optimal prediction after LP.

4.1 Notation

Let X = { x 1 , . . . , x N } ⊆ R d represent the training samples, paired with labels from the set Y = { y 1 , . . . , y N } ⊆ { 1 , 2 , . . . , C } , where d , C , and N denote the dimensions of the input space, the number of classes, and the number of training samples, respectively. This forms a training dataset { ( x 1 , y 1 ) , . . . , ( x N , y N ) | x i ∈ X , y i ∈ Y} , and we use x ∈ R d to denote both a training and a test sample. We denote the k -th element of vector a as [ a ] k . We use the Euclidean norm ∥ · ∥ for vectors and the Frobenius norm ∥ · ∥ F for matrices. ⟨· , ·⟩ denotes the inner product of two vectors. e k represents the one-hot vector for class k , and I C is the identity matrix of size C .

The model function, denoted as f ( · ; θ ) : X → R C , is parameterized by a set of parameters θ , and sometimes abbreviated as f ( · ) . The model includes a linear head, also referred to as the classifier, which consists of a weight matrix V and a bias vector b . The feature extractor is denoted by ϕ ( · ) : R h → R C , where h represents the hidden dimension. The output of the model is given by f ( x ) = V ϕ ( x ) + b . Parameters for a function g ( · ) and matrix A are sometimes denoted as θ g and θ A , respectively. Subscripts represent iteration or epoch, so f t ( · ) denotes the model at time t .

With the loss function ℓ : R C × Y → R , the training objective is to minimize the empirical risk L ( f ) := L ( f ( · ; θ )) = 1 N ∑ N i =1 ℓ ( f ( x i ; θ ) , y i ) . We use the CE loss, ℓ ( f ( x ) , y ) = -log ([ σ SM ( f ( x ))] y ) , where σ SM : R C → R C is the softmax function with its k -th element given by [ σ SM ( f ( x ))] k = exp([ f ( x )] k ) ) ∑ k ′ exp([ f ( x )] k ′ ) .

4.2 Training dynamics in the NTK regime

We use the NTK [Jacot et al., 2018], more specifically the empirical NTK [Wei et al., 2022, Malladi et al., 2023], to analyze the training dynamics of both FT and LP-FT. The empirical NTK, defined as the NTK with the parameters at the start of training, is a valuable tool for understanding the neural network training process, particularly in the context of FT [Wei et al., 2022, Malladi et al., 2023, Ren et al., 2023]. The empirical NTK applies a first-order approximation to changes in model outputs with respect to its parameters, so this is expected to capture changes in features.

To investigate the feature distortion theory in FT and LP-FT, we decomposed the updates into the following two parts. The part influenced by feature updates, unique to FT and absent in LP, is termed the FT-effective component of the NTK matrix, represented as F ( x , x i ) . In contrast, the part not influenced by feature updates, common to both FT and LP, determined by the pre-trained model, is termed the pre-train-effective component, represented as P ( x , x i ) . This decomposition highlights the distinct training dynamics of LP-FT in the NTK regime in the following proposition.

Proposition 4.1 (FT in the NTK regime) . The NTK of a model f ( x ) = V ϕ ( x ) + b , denoted by Θ f , can be decomposed as:

where the pre-train-effective component P ( x , x i ) and the FT-effective component F ( x , x i ) are defined using the classifier weight matrix V 0 and the feature extractor ϕ 0 at starting point of training as:

Consequently, assuming that one-epoch training within the NTK regime approximates FT, the logits and feature vectors for a sample x after FT, denoted as f FT ( x ) and ϕ FT ( x ) , to the starting point of training, f 0 ( x ) and ϕ 0 ( x ) , can be expressed as:

where Θ ϕ is the NTK matrix of the feature extractor ϕ , δ i := e y i -σ SM ( f 0 ( x i )) represents the difference between the one-hot label for the class y i and the predicted probability, and η is the learning rate.

The proof of this proposition is included in the Appendix (Appendix A.2.1). In our decomposition of the NTK matrix, the pre-train-effective component P ( x , x i ) is a diagonal matrix and remains unchanged after LP, while the FT-effective component F ( x , x i ) is not a diagonal matrix and does change after LP, resulting in distinct characteristics for these components. The Frobenius norm of the classifier weight matrix, ∥ V 0 ∥ F , influences the balance between the pre-train-effective and FT-effective components because it affects only the FT-effective component. This indicates that the classifier weight norm ∥ V 0 ∥ F has a significant impact on the training dynamics of FT.

Hypothesis on reduced feature changes in LP-FT The above proposition provides insights into why LP-FT causes fewer feature changes compared to FT:

The impact of the classifier weight norm ∥ V 0 ∥ F differs in the equations: it affects feature changes linearly (2) and affects logits quadratically (1). This implies that a higher norm can result in significant logit updates with relatively minor changes to the feature extractor, reducing feature changes in LP-FT compared with FT due to the increased classifier weight norm after LP.

(a) LP

(b) FT

(c) After training

0.85

Train accuracy

500

1.00

Train accuracy

0.804

LP-LoRA

0.75

0.80

400

fclassifierweight

0.95

0.802

0.800

classifierweight

LP-FT

0.70

300

0.90

0.798

LP

0.85

0.65

0.60

200

0.80

0.794

0.796

Norm

LoRA

100

FT

0.55

0.75

0.792

0.50

Classifier weight

0

0.70

Classifier weight

0.790

Rand.

100

101

102

103

104

0

10

20

30

40

50

0

5

10

15

20

25

30

Number of iteraions

Number of epochs

Norm of classifier weight

The magnitude of changes in both features and logits ((1) and (2)), is proportional to δ i , the difference between the predicted probability and the one-hot label. This suggests that feature changes are less pronounced in LP-FT than in FT since the difference δ i is smaller after LP.

The learning rate η , typically smaller in LP-FT than in FT [Kumar et al., 2022, Ren et al., 2023, Ha et al., 2024], helps moderate the direct influence of large classifier weight norms.

Prior studies [Kumar et al., 2022, Ren et al., 2023] have suggested that reduced feature changes in LP-FT stem from the near-optimal linear head obtained during LP. However, our analysis reveals that feature changes in LP-FT are also influenced by the classifier weight norm V 0 after LP. Our analysis focusing on classifier weight norms provides a new perspective on the training dynamics of LP-FT, highlighting the importance of the classifier weight norm in reducing feature distortion.

4.3 Derivation of Lemma A.3 from Kumar et al. in the NTK regime

The analysis presented in the original LP-FT paper by Kumar et al. [Kumar et al., 2022] operates within a framework where the feature extractor is a linear function. We define this framework in our context as follows:

Definition 4.2 (Linear model [Kumar et al., 2022]) . A linear model is defined as f linear ( x ) := V Bx + b , where V ∈ R C × h is the classifier weight matrix and B ∈ R h × d is the weight matrix of the feature extractor.

The linear model is a model whose feature extractor ϕ is a linear transformation. In this setting, we derive a corollary from Proposition 4.1 in our context, which is the pivotal lemma in the original LP-FT analysis [Kumar et al., 2022]:

Corollary 4.3 (Lemma A.3 from Kumar et al. in the NTK regime) . Within the context of the linear model (Definition 4.2), for any sample x ∈ Span( X ) ⊥ , the orthogonal complement of the subspace spanned by the training sample set X , the features after FT remain unchanged, expressed as:

where ϕ FT ( x ) and ϕ 0 ( x ) denote the feature vectors after and before FT, respectively.

This corollary shows that feature vectors for the samples in the orthogonal complement of training sample subspace are not updated. Therefore, given that pre-trained features have characteristics beneficial to downstream tasks, significant feature changes in FT, dependent on small training samples in LP, lead to poor generalization and OOD performance. The proof of this lemma can be found in the Appendix (Appendix A.2.2).

4.4 Increase in the classifier weight norm

The analysis in the previous section suggests that the classifier weight norm affects both feature changes and logits. On the basis of this insight, we examine classifier weight norms during training. Figure 1 shows that classifier weight norms consistently increase over time for LP, standard FT, and LoRA. As the training proceeds, norms of classifier bias and logits increases, while training loss decreases. Notably, LP shows a significantly larger increase in the norm compared to FT and LoRA.

Consider the transpose of the k -th row of matrix V denoted as v k ∈ R h for 1 ≤ k ≤ C , where C is the number of classes. Let τ ki represent the angle between ϕ ( x i ) and v k , which expands ⟨ v k , ϕ ( x i ) ⟩ to ∥ v k ∥∥ ϕ ( x i ) ∥ cos τ ki . The probability that class k is chosen for sample x i is given by the softmax function [ σ SM ( f ( x i ))] k = exp( ⟨ v k , ϕ ( x i ) ⟩ ) ∑ k ′ exp( ⟨ v k ′ , ϕ ( x i ) ⟩ ) . Consequently, with the CE loss for an input x i classified into class y i defined as ℓ ( f ( x i ) , y i ) = -log ([ σ SM ( f ( x i ))] y i ) , we have the following partial derivatives:

̸

̸

where the derivative with respect to cos τ y i i is negative and positive for k = y i . As training progresses, cos τ y i i tends to increase towards positivity, while cos τ ki for k = y i tends to become negative for each i . The derivative with respect to ∥ v k ∥ is given by:

̸

̸

Therefore, with adequate training and cos τ ki < 0 and cos τ y i i > 0 , the derivative with respect to ∥ v k ∥ is likely to become negative for each class k . The training of the model proceeds so that the empirical risk L decreases, so the norm ∥ v k ∥ tends to increase. This finding aligns with prior studies [Soudry et al., 2018, Kim and Kim, 2020].

̸

Remark: increase in classifier weight norms is more pronounced in LP than in FT In FT, particularly within an overparameterized setting, the model f may achieve perfect classification on the training dataset. That is, [ σ SM ( f ( x i ))] k becomes close to 0 for k = y i and 1 for k = y i . In this scenario, the derivative in Eq. (3) becomes close to zero, or the training itself is finished. Conversely, perfect classification is typically unattainable in LP unless the training dataset is linearly separable, so the derivative continues to be negative. In addition, while all parameters are updated in FT, only the classifier is optimized in LP, so the change in the classifier weight needs to be larger in LP than in FT to achieve the same classification performance. Consequently, the classifier weight norm tends to increase more significantly in LP than in FT, as shown in Figure 1 (c).

4.5 Training process of LoRA

We extend our analysis based on the NTK to the training process of LoRA. We follow the linear model setting as in Definition 4.2 and analyze the training dynamics of LoRA in the NTK regime.

Proposition 4.4 (LoRA approximates FT) . Consider the linear model setting (Definition 4.2) and let f LoRA and f FT be the models obtained via one-epoch training with LoRA and standard FT in the NTK regime. Let r denote the rank of the LoRA hyperparameter, and σ 2 represent the variance of the low-rank weight matrix initialization. Assume the input samples x satisfy ∥ x ∥ ≤ c . Then, for each sample pair x i , x j ∈ X , the pre-train-effective components of the NTK matrix for LoRA and FT, P LoRA ( x i , x j ) and P FT ( x i , x j ) , are identical:

Moreover, with at least 1 -4 exp( -( ϵ 2 -ϵ 3 ) r/ 4) probability, their FT-effective components, F LoRA ( x i , x j ) and F FT ( x i , x j ) , satisfy:

This proposition suggests that with high probability, the only difference of the NTK matrix between LoRA and standard FT is a scalar factor of the FT-effective component in the NTK matrix, and the scalar factor depends on the hyperparameters of LoRA. This implies that when the hyperparameters of LoRA are set appropriately, LoRA training is similar to standard FT training. This is consistent with the analysis by Malladi et al. [2023], where the NTK matrix of LoRA and standard FT are close to each other. It is important to note that the proposition is also valid for LP-FT and LP-LoRA (LP then LoRA). The proof of this proposition is included in the Appendix (Appendix A.2.3).

4.6 Discussion

An increased norm of the classifier weight reduces feature distortion and enhances the contribution of the FT-effective component of the NTK matrix during training. As a result, a higher classifier weight norm in LP-FT can be advantageous. However, since the increased norm is dependent on LP training, its optimality is not guaranteed. Specifically, during test time, although the increased classifier weight norm does not influence accuracy, it affects the calibration of the model. Calibration is defined as the alignment between the predicted probabilities and the actual probabilities [Guo et al., 2017]. An excessively high classifier weight norm can lead to overconfident predictions, which might be detrimental in practical applications. Consequently, there is potential for refining LP-FT by adjusting the classifier weight norm to enhance calibration.

Tuning the norm of the classifier after training can be effectively equated to applying temperature scaling [Guo et al., 2017] at test time. Temperature scaling adjusts the output logits with a temperature parameter T , thereby improving model calibration. Specifically, temperature scaling with parameter T , expressed as f ( x ) /T = V T ϕ ( x ) + b T , can be viewed as scaling the norm of classifier weight V and bias b by the temperature parameter T .

5 Numerical evaluation with transformer models

In this section, we numerically justify the following aspects obtained from our analysis:

The changes in features during training are smaller in LP-FT than in FT, and the norms of the classifier significantly increase during LP (Section 5.2).

The FT-effective component of the NTK matrix more effectively captures the input data than the pre-train-effective component (Section 5.3) and is more pronounced in LP-FT than FT.

A large classifier weight norm reduces the feature change during training, and its negative effects on calibration can be improved by temperature scaling (Section 5.4).

Details on the datasets, setup, and additional results, including performance evaluations for the experimental and practical application, are available in the Appendix (Appendices A.3 and A.4).

5.1 Setup

Datasets and models We used a total of 13 classification datasets from various benchmarks: SuperGLUE [Wang et al., 2019], GLUE [Wang et al., 2018], BOSS [Yuan et al., 2023], and PubMed 20 k RCT [Dernoncourt and Lee, 2017]. The breakdown of the datasets is as follows: five datasets from SuperGLUE (BoolQ, CB, RTE, WiC, and WSC), three datasets from GLUE (CoLA, MRPC, and SST-2), four datasets from BOSS (Amazon, Dynasent, SemEval, and SST-5), and PubMed 20 k RCT. Following experimental settings in studies that analyze FT dynamics from NTK perspectives [Malladi et al., 2023, Jang et al., 2024] and the study with similar settings Chen et al. [2022], we employed the RoBERTa-base model [Liu et al., 2020] as our Transformer-based model.

100

102

三

10-4

6

10-6

Pre-train E

Sing

FT

10-8

LP-FT

LoRA

1010

LP-LoRA

0

100

200

300

400

500

600

674

Index

Implementation and training We used the Transformers library [Wolf et al., 2020] and AdapterHub [Pfeiffer et al., 2020] for our implementation. Our training protocol followed the experimental setup described by Chen et al. [2022]. Hyperparameter tuning, especially for learning rates during the FT stage of LP-FT, was conducted through a grid search based on the validation set performance. For LP, we used logistic regression with L2 regularization on pre-trained features.

5.2 Small feature changes during LP-FT and significant norm increase during LP

LP-FT achieves notable performance with Transformer-based language models, outperforming standard FT in both ID and OOD settings, as detailed in Appendix (Appendices A.4.1 and A.4.3). To understand the underlying reasons for these results and validate small feature changes suggested by our analysis (Section 4.2), we analyzed changes in both the classifier and the features.

According to statistics presented in Table 1, the features after LP-FT demonstrate smaller changes from those of the pre-trained model than FT. Consequently, LP-FT preserves high cosine similarity among its features and exhibits a low Fisher's discriminant ratio (FDR) [Fisher, 1936], which assesses linear separability. Conversely, the classifier norms after LP and LP-FT are substantially larger than those of the pre-trained model and after FT, suggesting a significant increase in classifier weights during LP. A similar trend is observed in training with LoRA.

5.3 Kernel analysis

We examined the overall NTK matrix and its pre-train-effective and FT-effective components to understand their properties. Kernel regression was performed on the train and test sets to evaluate the performance of each kernel matrix.

Analysis of NTK matrix components and effectiveness of LP-FT In Table 2, the FT-effective component of the NTK matrix for LP-FT shows a higher rank and greater kernel regression accuracy compared to the pre-train-effective component, and the overall NTK matrix has intermediate properties. Additionally, the FT-effective component contributes more significantly to the overall kernel in LP-FT than in FT, as indicated by a higher FT Ratio. This ratio, calculated as the average of ∥ ∑ N i =1 F ( x , x i ) δ i ∥ / ∥ ∑ N i =1 ( P ( x , x i ) + F ( x , x i )) δ i ∥ for the train set samples, reflects the enhanced influence of the FT-effective component in LP-FT than in FT. These results suggest that the NTK matrix of LP-FT better captures input data through the increased influence of the FT-effective component.

Similarities between LoRA and FT The ranks of the FT-effective components in LoRA and FT (or LP-LoRA and LP-FT) are similar, as indicated in Table 2. Their distributions of singular values normalized by the maximum singular value, also closely align, as shown in Figure 2. These results suggest that the FT-effective components of the NTK matrix in FT and LoRA differ only by a scalar factor. This consistency demonstrates that our analysis (Section 4.2), originally based on a two-layer linear model, is applicable to more complex Transformer-based models.

25

Norm of the feature difference

20

15

10

FT

LP-FT

10-1

100

101

102

Norm ofclassifierweight

5.4 Analysis of classifier weight norms and temperature scaling

We experimentally verified significant effects of classifier weight norms in training (Section 4.2) and at test time (Section 4.6) in the following.

Effects of classifier weight norms in training We scaled the classifier weight norms at the start of the FT stage of LP-FT. The results, shown in Figure 3, indicate that larger classifier weight norms almost monotonically lead to smaller feature differences in both FT and LP-FT. Notably, LP-FT consistently shows smaller feature differences than FT, particularly when the classifier weight norms are large, validating our analysis that larger classifier weight norms reduce feature changes.

Temperature scaling at test time We implemented temperature scaling at test time, which is equivalent to adjusting the classifier weight norms, as discussed in Section 4.6. We optimized the temperature parameters on the validation sets based on CE loss, following the methodology suggested by Guo et al. [2017]. Table 3 presents the results on the RTE datasets. We assessed the expected calibration error (ECE) and maximum calibration error (MCE) [Naeini et al., 2015], which quantify the absolute differences between predicted and actual probabilities, with lower values indicating better calibration. These results show that the improvements in calibration with temperature scaling are the largest in LP-FT for both ECE and MCE, with notably substantial improvements in MCE. This suggests that large classifier weight norms contribute to poor calibration of LP-FT, which can be effectively mitigated through temperature scaling. These results highlight the effectiveness of refining LP-FT by temperature scaling.

6 Conclusion

In this paper, we explored the LP-FT training dynamics in complex classification models using the NTK to analyze feature changes. Our analysis identified classifier weight norms at the start of the FT stage as a key factor influencing FT dynamics. These norms balance the NTK matrix components and help reduce feature changes. Our findings support the existing feature distortion theory from an NTK perspective and emphasize the role of classifier weight norms alongside prediction accuracy. We also found that increases in classifier weight norms, characteristic of training with CE loss, may negatively impact model calibration, and this can be mitigated by temperature scaling. Additionally, the approximation effectiveness of LoRA is theoretically validated in terms of the similarity of the NTK matrix components. Empirical experiments with Transformer-based language models supported our theoretical insights, validating our understanding of the NTK, feature changes, and the benefits of temperature scaling. Overall, our study substantiates the efficacy of LP-FT as a robust method for adapting pre-trained complex models while preserving their well-trained features.

Limitations The main limitation of our study is that it is based on the NTK regime, which might not fully capture the training dynamics. Additionally, we consider just one epoch of gradient descent in FT, which may not effectively represent the overall training. In our experiments, we specifically focused on validating the effectiveness of LP-FT on language models. Therefore, areas other than natural language processing are outside the scope of our experiments.

Acknowledgments and Disclosure of Funding

This work was supported by JSPS KAKENHI Grant Number 24H00709 Japan.

References

Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https://aclanthology.org/2021.acl-long. 568 .

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems , 32, 2019.

Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , 01 2006.

Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC , 7(8):1, 2009.

Jill Burstein, Christy Doran, and Thamar Solorio. Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.

Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. Revisiting parameter-efficient tuning: Are we really there yet? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 2612-2626, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.168. URL https://aclanthology.org/2022. emnlp-main.168 .

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300 .

Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop , pages 177-190. Springer, 2005.

Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung , volume 23, pages 107-124, 2019.

Franck Dernoncourt and Ji Young Lee. Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts, 2017.

Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005) , 2005.

Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics , 7 (2):179-188, 1936.

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel, 2020.

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing , pages 1-9, 2007.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/guo17a. html .

Seokhyeon Ha, Sunbeom Jeong, and Jungwoo Lee. Domain-aware fine-tuning: Enhancing neural network adaptability. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 12261-12269, 2024.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=0RDcd5Axok .

Richard Zou Horace He. functorch: Jax-like composable function transforms for pytorch. https: //github.com/pytorch/functorch , 2021.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/ houlsby19a.html .

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= nZeVKeeFYf9 .

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Uijeong Jang, Jason D. Lee, and Ernest K. Ryu. Lora training in the ntk regime has no spurious local minima, 2024.

Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv:1911.03437 , 2019.

Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access , 8:81674-81685, 2020.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk .

Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2661-2671, 2019.

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=UYneFzXSJWh .

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems , 32, 2019.

Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= APuPRxjHvZ .

Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning , 2012.

Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=B1g8VkHFPH .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro{bert}a: A robustly optimized {bert} pretraining approach, 2020. URL https://openreview.net/forum?id=SyxS0T4tvS .

Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning , pages 23610-23641. PMLR, 2023.

Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems , 36, 2024.

Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of ACM Conference on Recommender Systems , 2013.

Mohamad Amin Mohamadi and Danica J. Sutherland. A fast, well-founded approximation to the empirical neural tangent kernel, 2023. URL https://openreview.net/forum?id=HN0ehX-ov5Q .

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence , volume 29, 2015.

Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of International Workshop on Semantic Evaluation (SemEval) , 2016.

Roman Novak, Jascha Sohl-Dickstein, and Samuel S Schoenholz. Fast finite width neural tangent kernel. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 17018-17044. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/novak22a.html .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vuli´ c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. AdapterHub: A framework for adapting transformers. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 46-54, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.7. URL https://aclanthology.org/2020.emnlp-demos.7 .

Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. DynaSent: A dynamic benchmark for sentiment analysis. In Proceedings of ACL-IJCNLP , 2021.

Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing learning paths, 2022.

Yi Ren, Shangmin Guo, Wonho Bae, and Danica J. Sutherland. How to prepare your task head for finetuning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=gVOXZproe-e .

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP , 2013.

Daniel Soudry, Elad Hoffer, and Nathan Srebro. The implicit bias of gradient descent on separable data. In International Conference on Learning Representations , 2018. URL https://openreview. net/forum?id=r1q7n9gAb .

Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems , 33:7852-7862, 2020.

Puja Trivedi, Danai Koutra, and Jayaraman J. Thiagarajan. A closer look at model adaptation using feature distortion and simplicity bias. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=wkg_b4-IwTZ .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems , 32, 2019.

Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019. doi: 10.1162/ tacl_a_00290. URL https://aclanthology.org/Q19-1040 .

Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learning , 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.emnlp-demos.6 .

Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in nlp: Benchmark, analysis, and llms evaluations. arXiv preprint arXiv:2306.04618 , 2023.

Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=likXVjmh3E .

Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 698-714. Springer, 2020.

Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE , 109(1):43-76, 2020.

A Appendix / supplemental material

A.1 Abbreviation and notation

Table 4 and Table 5 show our abbreviations and notations, respectively.

A.2 Proof of theoretical results

Additional notation The parameters for a function g , a weight matrix A , and a vector a is denoted as θ g , θ A , and θ a . Given a function g ( · ; θ g ) : R d → R s trained on N training samples X = { x 1 , x 2 , . . . , x N } ⊆ R d , we denote the NTK matrix of g at time t as Θ g t , which is defined as Θ g t := ∂ g t ( X ) ∂θ g ( ∂ g t ( X ) ∂θ g ) ⊤ ∈ R Ns × Ns , where g t ( X ) := vec ( g t ( x i )) x i ∈X , and ∂ g t ( X ) ∂θ g ∈ R Ns × p

⊤

with p parameters. The sub-matrix Θ g t ( x i , x j ) is defined as Θ g t ( x i , x j ) := ∂ g t ( x i ) ∂θ g ( ∂ g t ( x j ) ∂θ g ) ∈ R s × s , describing the relationship between training samples x i and x j in X . In the infinite width limit with NTK parameterization and general assumptions, the NTK matrix converges to Θ g := ∂ g 0 ( X ) ∂θ g ( ∂ g 0 ( X ) ∂θ g ) ⊤ as shown by [Jacot et al., 2018]. Subscripts represent iteration or epoch, so g t ( · denotes the model g at time t . ⊗ denotes the kronecker product of two matrices defined as

where A = [ a ij ] is an m × n matrix and B is any matrix.

)

A.2.1 Proof of Proposition 4.1

Proposition 4.1. The NTK matrix of a model f ( x ) = V ϕ ( x ) + b , denoted by Θ f , can be decomposed as:

where the pre-train-effective component P ( x , x i ) and the FT-effective component F ( x , x i ) are defined using the classifier weight matrix V 0 and the feature extractor ϕ 0 at starting point of training as:

Consequently, assuming that one-epoch training within the NTK regime approximates FT, the logits and feature vectors for a sample x after FT, denoted as f FT ( x ) and ϕ FT ( x ) , to the starting point of training, f 0 ( x ) and ϕ 0 ( x ) , can be expressed as:

where δ i := e y i -σ SM ( f 0 ( x i )) represents the difference between the one-hot label and the predicted probability, and η is the learning rate.

Proof of Proposition 4.1

Proof. The parameters of f , denoted as θ f , consist of θ V , θ b , and θ ϕ . The derivative of the model f with respect to each parameter is given by:

Therefore, the NTK matrix of f , defined as Θ f ( x , x i ) := ∂ f 0 ( x ) ∂ θ f ( ∂ f 0 ( x i ) ∂ θ f ) ⊤ , can be expressed as:

For gradient descent, the update to the parameters θ f at time t is given by:

where δ i is defined as δ i := e y i -σ SM ( f 0 ( x i )) . Assuming that one-epoch training approximates FT, the model is expressed as f FT = f 1 . Therefore, the update to the model f in the linearized regime is given by:

Finally, replacing θ f with θ ϕ in Eq. (8), the update to the parameters θ ϕ at time t is given by

Therefore, the update to the feature extractor after FT, given by ϕ FT = ϕ 1 for the same assumption, is:

This completes the proof.

A.2.2 Proof of Corollary 4.3

Corollary 4.3. Within the context of the linear model (Definition 4.2), for any sample x ∈ Span( X ) ⊥ , the orthogonal complement of the subspace spanned by the training sample set X , the features after FT remain unchanged, expressed as:

where ϕ FT ( x ) and ϕ 0 ( x ) denote the feature vectors after and before FT, respectively.

Proof of Corollary 4.3

Proof. The feature extractor is given by ϕ ( x ) = Bx , where B is the weight matrix. The derivative of the feature extractor with respect to the parameters θ ϕ = θ B is:

so the empirical NTK matrix of the feature extractor becomes:

where ⊗ denotes the kronecker product.

From the Proposition 4.1, the feature update is given by:

where δ i = e y i -σ SM ( f 0 ( x i )) , V 0 is the classifier weight matrix at the start of training, and η is the learning rate. For any sample x ∈ Span( X ) ⊥ , ⟨ x , x i ⟩ = 0 for all x i ∈ X , so the feature update is 0 for OOD samples, namely:

This completes the proof.

A.2.3 Proof of Proposition 4.4

Proposition 4.4. Consider the linear model setting (Definition 4.2) and let f LoRA and f FT be the models obtained via one-epoch training with LoRA and standard FT in the NTK regime. Let r denote the rank of the LoRA hyperparameter, and σ 2 represent the variance of the low-rank weight matrix initialization. Assume the input samples x satisfy ∥ x ∥ ≤ c . Then, for each sample pair x i , x j ∈ X , the pre-train-effective components of the NTK matrix for LoRA and FT, P LoRA ( x i , x j ) and P FT ( x i , x j ) , are identical:

Moreover, with at least 1 -4 exp( -( ϵ 2 -ϵ 3 ) r/ 4) probability, their FT-effective components, F LoRA ( x i , x j ) and F FT ( x i , x j ) , satisfy:

Proof Approach To prove this theorem, we use a lemma from distributional properties:

Lemma A.1 (Corollary of the distributional Johnson-Lindenstrauss Lemma) . Given vectors u , v ∈ R d with ∥ u ∥ , ∥ v ∥ ≤ c , and a random matrix A ∈ R k × d with i.i.d. entries from a distribution with mean 0 and variance 1, for any ϵ > 0 :

Proof of Proposition 4.4

Proof. The feature vector of LoRA is given by ϕ LoRA ( x ) = B 0 x + B LoRA A LoRA x , where pretrained feature weight matrix B 0 is fixed during training, and A LoRA ∈ R r × d and B LoRA ∈ R h × r are low-rank weight matrices in LoRA. A LoRA is initialized from a normal distribution with mean 0 and variance σ 2 , while B LoRA is initialized with zeros. The LoRA feature updates are represented as ϕ LoRA ( x ) = B 0 x + B LoRA A LoRA x , with B 0 fixed during training.

The pre-train-effective components of LoRA and FT, denoted as P LoRA ( x , x i ) and P FT ( x , x i ) respectively, are defined as:

where I C is the identity matrix of size C . These pre-train-effective components are identical since:

for all x ∈ X because B LoRA is initialized as a zero matrix i.e. B LoRA 0 = O .

For the FT-effective component of the NTK matrix, consider the derivatives concerning LoRA parameters B LoRA and A LoRA :

Here, θ B LoRA and θ A LoRA denote the parameters of B LoRA and A LoRA , respectively.

The FT-effective component of the NTK matrix for LoRA, denoted as F LoRA ( · , · ) , is derived by combining these partial derivatives:

where the last equality holds because B LoRA 0 is a zero matrix.

Similarly, the FT-effective component of the NTK matrix for standard FT, F FT ( · , · ) , is given by:

Using the Johnson-Lindenstrauss lemma, with a probability of at least 1 -4 exp( -( ϵ 2 -ϵ 3 ) r/ 4) :

which implies:

This completes the proof.

A.3 Experimental details

A.3.1 Datasets

From the SuperGLUE benchmark [Wang et al., 2019], we used the five datasets: BoolQ [Clark et al., 2019], CB (CommitmentBank) [De Marneffe et al., 2019], RTE (Recognizing Textual Entailment) [Dagan et al., 2005, Bar-Haim et al., 2006, Giampiccolo et al., 2007, Bentivogli et al., 2009], WiC (Words in Context) [Burstein et al., 2019], and WSC (Winograd Schema Challenge) [Levesque et al., 2012]. From the GLUE benchmark [Wang et al., 2018], we used the three datasets: CoLA (Corpus of Linguistic Acceptability) [Warstadt et al., 2019], MRPC (Microsoft Research Paraphrase Corpus) [Dolan and Brockett, 2005], and SST-2 (Stanford Sentiment Treebank, version 2) [Socher et al., 2013]. Four datasets from BOSS [Yuan et al., 2023] were used in OOD evaluation: Amazon Reviews [McAuley and Leskovec, 2013], Dynasent [Potts et al., 2021], SemEval [Nakov et al., 2016], and SST-5 [Socher et al., 2013]. Finally, we used the PubMed 20 k RCT dataset [Dernoncourt and Lee, 2017] for validation in practical settings. The dataset statistics are detailed in Table 7.

For the datasets from the GLUE, SuperGLUE, and BOSS benchmarks, we divided the original training set using a 9:1 training-to-validation ratio, using the original validation set as the test set, in accordance with Chen et al. [2022]. For PubMed 20 k RCT, we used the original training, validation, and test sets for their respective purposes.

A.3.2 Implementation and training details

When applying LoRA, LoRA was applied only to the query and value projection matrices of the attention mechanism in the Transformer architecture, following the approach described in the original paper by Hu et al. [2022]. The LoRA settings were fixed at α = 8 and r = 8 for all experiments.

The model was trained for 10 epochs without early stopping, and the one showing the best performance on the validation set was chosen for further evaluation. We used the Adam optimizer [Kingma and Ba, 2017]. Our code is built on PyTorch [Paszke et al., 2019], using the HuggingFace Transformers library [Wolf et al., 2020] and AdapterHub [Pfeiffer et al., 2020]. All experiments were run on a single NVIDIA A 100 GPU. The results reported are averages from 3 tuning seeds and 5 evaluation seeds.

For LP, cross-validation and automatic hyperparameter adjustment were used to find the optimal L2 regularization strength, using scikit-learn [Pedregosa et al., 2011] with its standard training parameters.

Details on the hyperparameters for our experiments can be found in Table 6.

A.3.3 Details of each experiment

Experiments on the GLUE and SuperGLUE benchmarks For the FT and LoRA methods, the learning rate and batch size were adopted from Chen et al. [2022], where these hyperparameters were optimized using grid search on the validation set. For LP-FT and LP-LoRA, batch size is fixed at 8 and we tuned the learning rate.

Experiments on BOSS benchmark and the PubMed 20 k RCT dataset For the experiments on BOSS benchmark and the PubMed 20 k RCT dataset, we tuned the learning rate and batch size using grid search based on the validation set performance.

Calculation of the NTK matrix We computed the NTK matrix for FT, LoRA, LP-FT, and LPLoRA as specified in Eq. (1). We separately calculated the pre-train-effective and FT-effective components of the NTK matrix. Following the methodology by Malladi et al. [2024], we used functorch [Horace He, 2021] and forward-mode auto-differentiation [Novak et al., 2022] for these calculations. To reduce computational costs, we randomly selected 10 %of the parameters from the word embedding matrix for derivative calculations. For datasets with more than 250 samples, we used a subset of 250 randomly selected samples to compute the NTK matrix.

Solving the kernel regression Following the methodology described by Malladi et al. [2024], we treated each output logit independently in our kernel regression model. This method is based on the representer theorem, where the empirical risk minimizer is expressed as a linear combination of kernel features from the training data: f ( x ) = ∑ NC i =1 α i K ( x , x i ) , with K representing the NTK matrix or its component for a training set of size NC × NC . We solved this optimization using logistic regression with L2 regularization and used the resulting coefficients α i to compute logits on the test set via its corresponding NTK matrix.

Effects of classifier weight norms in training We scaled the norms of the classifiers within the range of [0 . 1 , 0 . 5 , 1 , 2 , 5 , 10 , 50 , 100] before proceeding to the FT stage of training, specifically after random initialization in FT and after LP training in LP-FT. We conducted this experiment using the CB and RTE datasets and Boss benchmark. We apply the LoRA method on the CB and RTE datasets. We averaged the results over 5 seeds for the CB and RTE datasets and 3 seeds for the Boss benchmark, plotting these with their standard deviations.

Temperature scaling We applied temperature scaling [Guo et al., 2017] to the logits of the model at test time. Following the methodology of the original paper [Guo et al., 2017], we tuned the temperature parameter using the validation set to minimize the negative log-likelihood. For implementation, we employed the Adam optimizer [Kingma and Ba, 2017] with a learning rate of 1 × 10 -3 , optimizing the temperature for 1 × 10 5 steps. We incorporated early stopping based on the negative log-likelihood, with a patience of 10 iterations starting from an initial temperature value of 1 . 0 . The number of the bins to calculate ECE and MCE is set to 15 .

A.4 Additional experimental results

A.4.1 Results on the SuperGLUE and GLUE benchmarks

Table 8 shows the test results for the SuperGLUE and GLUE benchmarks. We report accuracy and its standard deviation on the test sets, except for the CoLA dataset, which uses the Matthew's correlation coefficient for the performance metric.

Figure 4 shows the increase in the classifier weight norm during training on the CB dataset. With more iterations or epochs, there is a noticeable increase in both accuracy and the classifier weight norm.

Figure 5 and Figure 6 display t-SNE visualizations of the feature vectors from the CB dataset. After FT, the features are distinctly separated by class. In contrast, the classifier row vectors remain nearly identical to those of the pre-trained model. After LP-FT, the features retain the structure of the pre-trained model, but the classifier row vectors deviate from their initial state. A similar pattern is observed with the LoRA method.

f classifier weight

(a) LP (CB)

(b) FT (CB)

(c) LoRA (CB)

(d) LoRA (RTE)

(e) After training (CB)

1.0

Trainaccuracy

2500

60

2000

weigh

0.8

1500

classifie

T

0.7

1000

Norm

0.6

500

0.5

Classifierweight

0

0

20

40

60

80

100

Number of iteraions

1.00

Train accuracy

0.990

classifierweight

0.95

0.985

0.90

0.980

0.85

0.975

Norm

0.80

Classifier weight

0.970

0

10

20

30

40

50

Number of epochs

1.0

Train accuracy

1.15

Train accuracy

60

Norm of clasifier weight

60

1.00

1.10

0.95

1.05

0.7

0.7

0.90

1.00

0.6

0.6

0.85

0.5

Classifier weight

0.95

Classifierweight

0.5

0.80

0

10

20

30

40

50

0

10

20

30

40

50

Number of epochs

Number of epochs

LP-LoRA

LP-FT

LP-

LoRA

Ld

Rand.

5

10

15

20

25

30

35

40

Norm of classifier weight

Norm of

(a) FT

(b) LP-FT

Feature

X

Classifier

Pretrain feature

Pretrain classifier

Feature

X

Classifier

Pretrain feature

Pretrain classifier

(a) FT

(b) LP-LoRA

Feature

Classifier

Pretrain feature

米

Pretrain classifier

米

X

Feature

Classifier

Pretrain feature

十

Pretrain classifier

A.4.2 Results of NTK analysis

Table 9 displays the kernel statistics, while Figure 7 shows the distribution of singular values. Figure 8 and Figure 9 visually depict the trace norms of sub-matrices within the NTK matrix. For the kernel matrix K ∈ R NC × NC , we calculated the trace norms of the sub-matrix K ( x i , x j ) ∈ R C × C for each sample pair ( x i , x j ) in the training sets.

Figure 8 reveals a consistent pattern in the FT-effective component of the NTK matrix across all datasets: pairs of identical samples in diagonal positions typically exhibit higher trace norms. This suggests that the FT-effective component is more effective at capturing relationships among samples compared to the pre-train-effective component. Additionally, in the CB dataset, certain sample pairs, particularly in classes 1 and 3, show notably high trace norms, indicating that the pre-trained model effectively differentiates between these class samples.

(a) RTE

(b) BoolQ

100

100

10-3

10-3

Singular value

Singular value

10-6

106

109

10-9

Pre-train E

Pre-train E

L

L

1012

1012

LP-FT

LP-FT

1015

LoRA

10-15

LoRA

LP-LoRA

LP-LoRA

0

100

200

Index

000

400

499

0

100

200

Index

300

400

499

10-1

10-3

Singular value

10-5

10-7

10-9

Pre-train E

10-11

FT

LP-FT

10-13

LoRA

10-15

LP-LoRA

0

100

200

Index

300

400

499

(c) WiC

(a) Pre-train E (CB)

(b) FT E (FT, CB)

(c) FT E (LP-FT, CB)

(d) Pre-train E (RTE)

(e) FT E (FT, RTE)

(f) FT E (LP-FT, RTE)

(g) Pre-train E (BoolQ)

(h) FT E (FT, BoolQ)

(i) FT E (LP-FT, BoolQ)

(j) Pre-train E (WiC)

(k) FT E (FT, WiC)

(l) FT E (LP-FT, WiC)

405

-400

- 395

2

390

385

380

2

m

272

270

-268

266

264

262

260

258

2

115.0

112.5

-110.0

107.5

2

105.0

102.5

100.0

3

97.5

2

m

76

74

2

72

70

68

2

72

- 70

68

66

64

62

60

2

-44000

-42000

2

-40000

-38000

36000

3

-34000

2

m

16000

15750

-15500

15250

-15000

14750

14500

14250

2

262.5

260.0

257.5

255.0

252.5

250.0

247.5

245.0

-280

275

270

265

260

2

17500

-17000

16500

-16000

-15500

-15000

14500

2

-330000

320000

-310000

300000

290000

280000

2

-330000

-320000

-310000

300000

290000

280000

2

A.4.3 Experiments on BOSS benchmark

Table 10 shows indicate that LP-FT surpasses FT in OOD robustness and achieves higher accuracy in ID settings than LoRA. This suggests that LP-FT is effective in enhancing model robustness to OOD samples with reduced feature changes.

Table 11 displays the statistics of feature and classifier changes on the Amazon, Dynasent, SemEval, and SST-5 datasets. The FDR within the ID is lower for LP-FT than for FT, whereas the FDR for OOD is higher for LP-FT than for FT. This indicates that LP-FT is learning robust features that are less sensitive to OOD data.

A.4.4 Change of feature and classifier norms

Table 12 shows the changes in features during the FT stage, indicating that the changes are smaller during LP-FT compared to FT. Table 13 shows the classifier norms, which increase during training, with a more noticeable increase observed during LP than during FT.

(a) LoRA, CB

(b) LoRA, RTE

(c) LoRA, BoolQ

(e) LP-LoRA, CB

(f) LP-LoRA, RTE

(g) LP-LoRA, BoolQ

(h) LP-LoRA, WiC

0.185

0.1500

0.140

460

0.180

0.1475

0.1450

0.135

440

0.175

2

2

0.1425

0.130

2

0.170

0.1400

420

0.165

0.1375

0.125

0.1350

400

0.160

0.120

0.1325

34

32

90

33

460

85

31

32

440

2

31

80

30

30

420

29

75

29

400

28

N

A.4.5 Effects of classifier weight norms in training

Figure 10 (Boss benchmark) and Figure 11 (the CB and RTE datasets) illustrate the changes in features from the pre-trained models. Except for the CB dataset, the change in features in LP-FT is generally smaller than in FT when using large classifier norms. The CB dataset has a smaller sample size, which could be an exception.

A.4.6 Temperature scaling

The result of temperature scaling on SuperGLUE and GLUE is presented in Tables 14 and 15.

A.4.7 PubMed 20k

In addition to the natural language understanding benchmarks, we also evaluated LP-FT on the PubMed 20 k RCT dataset to evaluate its effectiveness in practical applications. The PubMed 20 k RCT dataset, a subset of PubMed 200 k [Dernoncourt and Lee, 2017], comprises 20,000 medical abstracts from randomized controlled trials, categorized into five classes. Efficient tools for navigating extensive medical literature are essential for the medical community.

(a) Amazon (ID)

(b) Dynasent (OOD)

(c) SemEval (OOD)

25

25

Norm of the feature difference

Norm of the feature difference

20

20

15

15

10

10

5

FT

5

FT

LP-FT

LP-FT

10-1

100

101

102

10-1

100

101

102

Norm of classifier weight

Norm of classifier weight

Norm of the feature difference

20

15

10

FT

LP-FT

10-1

100

101

102

Norm of classifier weight

(a) CB

(b) RTE

25

25

FT

LP-FT

Norm of the feature difference

difference

LoRA

20

20

LP-LoRA

Norm of the feature

15

15

10

10

FT

LP-FT

5

5

LoRA

LP-LoRA

0

10-1

100

101

102

10-1

100

101

102

Norm of classifier weight

Norm of classifier weight

The results are presented in Table 16. The LoRA model outperforms other models, although the performance of FT, LP-FT, and LoRA models are relatively similar.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that our paper analyzes a fine-tuning method, specifically linear probing then fine-tuning, from a neural tangent kernel perspective. The abstract succinctly summarizes the main contributions, and the introduction provides a thorough overview of the paper's scope with our motivation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explicitly discuss the limitations of our theoretical analysis in the limitations section of our paper, highlighting the need for further investigations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We clearly state our assumptions alongside the propositions and provide complete proofs in the appendix. This ensures that our theoretical results are well-supported and verifiable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include all essential details needed to replicate our main experimental results within the paper. This includes hyperparameters and data splits to ensure that our findings are reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release the code and associated data before the review process begins. This release will include comprehensive instructions to ensure faithful reproduction of our experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so ' No ' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We detail all necessary training parameters, including data splits and hyperparameters, to ensure that our experimental results can be faithfully reproduced.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include error bars and standard deviations in our results where applicable, ensuring that the statistical significance of our findings is clear and well-documented.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Detailed descriptions of the computational resources used, including hardware specifics and implementation details, are provided in the Appendix to aid in reproducing our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: After thoroughly reviewing the NeurIPS Code of Ethics, we confirm that our research adheres to all the specified guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Given the theoretical nature of our work, we assess that it does not directly engage with societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research does not involve the release of data or models that pose high risks for misuse, hence specific safeguards are not required.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our study does not use any external assets, thus no licensing or attribution issues are applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are introduced in our paper, so there are no associated documentation requirements.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors analyze the training dynamics of LP-FT for classification models based on NTK theory. They decomposed NTK matrix and highlight the importance of linear head norm. Additionally, they highlight the increased linear head norms can negatively affect the model calibration and can be fixed by temperature scaling. Finally, they extend their analysis to the LoRA method.

STRENGTHS:
1. The application of NTK theory to analyze LP-FT in complex models like Transformers is interesting.
2. The theoretical analysis is thorough, well-supported by mathematical proofs, and aligns with empirical observations.

WEAKNESSES:
1. The experiments primarily focus on the classification tasks. Since the authors also extend their analysis to LoRA, exploring additional domains and harder tasks (reasoning, math, code generation) combined with LLMs could siginificantly strengthen the generalizability of the findings.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
The authors present a theoretical analysis of the linear probing and fine-tuning framework based on neural tangent theory, supported by experiments with transformer-based models on natural language processing benchmarks. Their analysis decomposes the NTK matrix into the FT-effective component and the pre-train-effective component, demonstrating an approximation between the fine-tuning matrix and the LoRA matrix.

STRENGTHS:
1. The paper is very well written, with detailed technical content. The authors have provided enough mathematical details to support their theoretical analysis. The proofs for theorems and corollaries are detailed delineated in the appendix.   

2. A timely topic that bridges established theory and recent advancements in natural language processing.

3. The authors have provided comprehensive benchmarks to support the claims of the paper.

WEAKNESSES:
Although the experiments included an exhaustive list of benchmarks, I noticed that you only implemented one transformer model in your setup. I believe it would be more effective to include multiple types of transformer-based models to better illustrate the practical advantages of your approach.

QUESTIONS:
Minors: In Figures 3 and 11, it would be helpful to explicitly indicate the purpose of the shaded area in the caption to enhance readability.

LIMITATIONS:
The authors have made a clear statement about the limitations of the proposed analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

comments: SUMMARY:
This paper studies the dynamics of Linear probing and fine tuning by means of NTK theory. The authors provide a connection between the  Frobenius norm of  linear probing weights and FT-effective component of the NTK matrix.

STRENGTHS:
The NTK  sees the model as an  Gaussian process, making it a  powerful tool for analysis of neural networks’ convergence and generalization. This paper  derives an interesting connection between NTK  theory and  fine tuning language models. The NTK is decomposed into  pre-trained and fine tuned terms.  Feature distortion theory is employed to interpret performance of FT.

WEAKNESSES:
-The organization of  material   can be improved. for instance, you  used the linear model $f(x)$ in Proposition 1, however it is introduced later in Definition 1. 

-Table 2 is very uninformative and the numbers are not in line, making it hard to read. please provide  precise description of each table and  figure in the text. 

-Same for Fig.3, some numbers are floating on it and the message of the figure is not clear.

please  further enhance the presentation of  results,  I found it hard to  infer  your  achievements from the numerical results section.

QUESTIONS:
In Fig.3, What does “number of feature difference”? please provide the definition.

-How did you calculate the NTK matrix for numerical simulations?  There is a body of literature for only  approximation of NTK for transformers.

-Did you use empirical NTK for simulations? if yes, then can you say if your results hold in the finite width regime too?

LIMITATIONS:
NTK theory in general holds in the infinite width limit. Further discussions are needed to adapt the methods in this paper to finite regime.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel application of neural tangent kernel (NTK) theory to analyze the training dynamics of the linear probing then fine-tuning (LP-FT) method for large language models, demonstrating its effectiveness and extending the analysis to include the low-rank adaptation (LoRA) method.\n\nSTRENGTHS:\nThe paper stands out for its originality by applying NTK theory to the LP-FT method and extending it to include an analysis of LoRA. It's strong in terms of quality because it combines solid theoretical work with thorough experiments to back up the claims. The writing is clear and well-organized, making it easy to follow the arguments and findings. The significance of the research is evident in its potential impact on fine-tuning practices, not just in NLP but across various fields that use large pre-trained models. Overall, this paper makes a valuable contribution to advancing our understanding and methods in transfer learning.\n\nWEAKNESSES:\nOne weakness of the paper is that while it presents a solid theoretical foundation and empirical evidence, it could benefit from more detailed explanations in certain sections, particularly the experimental setup and hyperparameter tuning. This would enhance transparency and reproducibility. Additionally, the paper could further strengthen its claims by including a broader range of experiments across different domains beyond NLP. Another point for improvement is the discussion of the impact and practical applications of the findings, which could be expanded to provide more actionable insights for practitioners. Lastly, while the paper briefly mentions temperature scaling to address calibration issues, a deeper exploration of this aspect could provide more comprehensive recommendations for improving model performance.\n\nQUESTIONS:\nCan you expand on the practical applications of your findings? More specific examples of how practitioners can apply your recommendations in real-world scenarios would enhance the practical value of your work.\n\nLIMITATIONS:\nhe paper does not discuss the computational resources required for implementing LP-FT and NTK analysis. Providing information on the computational cost and efficiency of the proposed methods would help readers understand the practical feasibility of adopting these techniques.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}