Linearization Explains Fine-Tuning in Large Language Models
Authors: Zahra Rahimi Afzal
Tara Esmaeilbeig
Mojtaba Soltanalian
Mesrob I Ohannessian
Venue: NeurIPS 2025 poster ()
Keywords: Parameter-Efficient Fine-tuning
LLMs
Neural Tangent Kernel
Linearization
Low Rank Adaptation
Primary area: deep_learning
TL;DR: This paper analyzes PEFT of LLMs via linearization, showing how an explicit inductive bias connects fine-tuning dynamics to NTKs and offering theoretical and empirical insights into when linearization accurately predicts adaptation performance.

Abstract: Parameter-Efficient Fine-Tuning (PEFT) is a popular class of techniques that strive to adapt large models in a scalable and resource-efficient manner. Yet, the mechanisms underlying their training performance and generalization remain underexplored. In this paper, we provide several insights into such fine-tuning through the lens of linearization. Fine-tuned models are often implicitly encouraged to remain close to the pretrained model. By making this explicit, using an $\ell_2$-distance inductive bias in parameter space, we show that fine-tuning dynamics become equivalent to learning with the positive-definite neural tangent kernel (NTK). We specifically analyze how close the fully linear and the linearized fine-tuning optimizations are, based on the strength of the regularization. This allows us to be pragmatic about how good a model linearization is when fine-tuning large language models (LLMs). When linearization is a good model, our findings reveal a strong correlation between the eigenvalue spectrum of the NTK and the performance of model adaptation. Motivated by this, we give spectral perturbation bounds on the NTK induced by the choice of layers selected for fine-tuning. We empirically validate our theory on Low Rank Adaptation (LoRA) on LLMs. These insights not only characterize fine-tuning but also have the potential to enhance PEFT techniques, paving the way to better informed and more nimble adaptation in LLMs.

Linearization Explains Fine-Tuning in Large Language Models
Zahra Rahimi Afzal"a
Tara Esmaeilheig Mojtaba Soltanalian'
Mesrob I. Ohannessian"
Universsity of Illinois Chicago "Nokia Bell Labs
"(zrahim2, msol, mesrob]@uic.ent Saara.esmaeillel haaaaeeeeeallleiiiggoii......m
Abstract
Parameter-Efficien Fine-Tuning (PEFT) is popular class of techniques that strive to adapt large models in scalable and resource-efficient manner. Yet, the mechanisms underlying their training performance and generalization remain underexplored. In this paper, we provide several insights into such fine-tuning through the lens of linearization. Fine-tuned models often implicitly encouraged to remain close to the pretrained model. By making this explicit, using an lz- distance inductive bias parameter space, show that fine-tuning dynamics become equivalent to learning with the positive-definite neural tangent kernel (NTK). We specifically analyze how close the fully linear and the linearized fine- tuning optimizations are, based on the strength of the regularization. This allows be pragmatic about how good a model linearization is when fine-tuning large language models (LLMs). When linearization is a good model, our findings reveal strong correlation between the eigenvalue spectrum of the NTK and the performance of model adaptation. Motivated by this, we give spectral perturbation bounds on the NTK induced by the choice of layers selected for fine-tuning, We empirically validate our theory on Low Rank Adaptation (LoRA) on LLMs. These insights not only characterize fine-tuning but also have the potential to enhance PEFT techniques, paving the way to better informed and more nimble adaptation in LLMs.
Introduction
Foundational large language models (LLMs) [1,2] have emerged then adapted to various NLP tasks through fine-tuning. Due to the enormous number of trainable parameters in these models, full fine-tuning can be expensive in terms of time and other computational resources [3]. Parameter-Efffiieet Fine-tuning (PEFT) [4-8] is a particularly popular set of techniques that strive tackle this computational burden. They aim reduce the effective number of trained parameters, making choices such as focusing on certain layers only or applying rank-limited updates, all while maintaining how well the model adapts the task. However, these methods often lack fundamental understanding of the dynamics behind these choices, making informed exploration of the algorithmic space difficult.
purppose models that
In this paper, we introduce linearized fine-tuning, which establishes a foundation for rigorously understanding adaptation in large models through the lens of Neural Tangent Kernel (NTK) regression. Said simply, linearizing the fine-tuning process reduces the problem to one closely aligned with NTK regression. This, in turn, allows to predict the performance of various fine-tuning decisions using the properties of the NTK kernel. Linearization can be achieved by regularizing fine-tuning to remain
"Equal contribution.
Our code is publicly available at http:////ggguuubb..omaararaaraae
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

close to the original model, which is empirically found to be non-restrictive, because most methods already encourage this proximity between the fine-tuned and original models [9].
I Linearized approximations to guide fine-tuning have previously been explored in [10-14]. In [10], a linearized approximation is used introduce two scores for selecting models from zo0 of models to be fine-tuned for specific task. In [11]. a linear model is constructed that combines network activation with the gradient features. In [12], it is demonstrated that linearized models, governed by the NTK, outperform their nonlinear counterparts in fine-tuning. All these works, however, do not quantify the closeness of finettuning dynamics to the linear approximation. Instead, they only hypothesize that the model will remain close to the pretrained model and that, therefore, the fine-tuning dynamics can be approximated by a Taylor expansion of the model around the pretrained parameters. This shortcoming motivates the present paper to introduce an explicit inductive bias that enables quantifying the extent to which linearity is preserved during fine-tuning. In particular, we provide a theoretical upper bound on the distance between the fine-tuned model and its linearized approximation, which in turn firmly supports predictions of fine-tuning performance based on linearization. The work in [13] explores the loss landscape in the NTK regime; however, it does not determine under what conditions fine-tuning falls within this regime, In contrast, our work proves both theoretically and empirically that, although necessarily in the NTK (linearization) regime, fine-tuning can be constrained to it by regularizing the process towards the pretrained model.
Our work builds on prior results that establish conditions under which linearity holds. Jacot et al. [15] showed that in the infinite width limit, the network function follows a linear differential equation during training. Moreover, they proved that for wide networks, the NTK remains constant during training; hence, they called this regime lazy training. Later, Bach et al. [16] suggested that lazy training is not limited to infinite-width networks.
More recently, and of particular relevance here, Malladi al. in [17] extended the NTK theory to characterize kernel-based dynamics specifically for fine-tuning language models. They adapted infinite width analysis to account for pretrained initialization and proved that prompt-based fine- tuning in complex architectures like transformers can exhibit kernel behavior under certain conditions. They demonstrated the critical role of meaningful prompts in achieving kernel behavior and explained how these dynamics describe fine-tuning trajectories. Additionally, they proved the effectiveness of Low Rank Adaptation (LoRA) by bounding the difference between the kernel of full fine-tuning and the kernel of LoRA. In [17], the authors assume that fine-tuning can be explained by linearization, and their results also confirm that linearization does not always appear in tuning (see Fig. of [17]). In our work, however, by introducing an explicit inductive bias for fine-tuning, which is essentially a weight decay toward the original model parameter, we induce lazy training. Consequently, the fine-tuning performance can be approximated by the linearized model [9, 18]. The evolution of the model is described by the neural tangent kernel (NTK), which stays constant during fine-tuning. This property makes it possible to study the generalization properties of a fine-tuned model based on the spectral properties of the NTK. Specifically, our main contributions are:
We show that having an explicit inductive bias toward the pretrained model results in linearized fine-tuning: more precisely, it brings fine-tuned and linearized models closer.
Leveraging the linearization of fine-tuning, we formulate the fine-tuning problem as a neural tangent kernel regression and use the eigenvalue decomposition of the neural tangent kernel to derive bounds on the empirical risk of the end result of fine-tuning, prior to its execution.
We provide bounds on the spectral perturbation on the NTK when a set of trainable parame- ters is added to fine-tuning. In particular, to the best of our knowledge, we present the first spectrum perturbation results due to layer selection and introduce an algorithm that provides new insights to guide fine-tuning design.
Through extensive experiments, we validate our theoretical results. We evaluate the condition number of NTK as an at-initialization metric to anticipate the performance of LoRA before training. Even though our experiments focus on LoRA, the technical tools we introduce could be equally used in the context of other PEFT methods.
The paper is organized as follows. In $2, we present the problem formulation and notation. In $3, we detail the linearization framework. In $4, we give an NTK regression formulation for the linearized fine-tuning and related performance to the kernel's 's spectrum. $5, we show how layer choices affect this spectrum. We then validate our findings through experiments in $6 and conclude in $7.

---

2 Problem Formulation
In the fine-tuning problem, we are given a pretrained model fo. (-), a target task dataset Dr (x;, Y.) for the downstream task, and a loss function -) R x IR - R. The most basic fine-tuning solution is minimize the target empirical risk via gradient descent, when initialized at Sa, (-). This implicitly finds solutions close to the pretrained model. In this paper, we make this explicit via an inductive bias toward the pretrained model. We denote the regularized fine-tuned model by Sa- (+) I2 R, and we define it as minimizing the regularized empirical risk.
0- = minimize R(0)= + 2 210 0ul12.
where
R(0) = (fo((x))
(2)
denotes the trainable fine-tuning parameters, 0n denotes the parameters of the pretrained model, (0) the original objective function, and À the regularization strength hyperparameter. When 0g = 0, (I) reduces to ordinary weight decay. This regularization reduces the deviation between the fine- tuned and pretrained models, and is a simple instance of proximal methods often used in fine-tuning. We assume ((;,)) is the squared loss, and thus the risk is the MSE. By representing Dr (X,, Vt) ] E 1a and Y = ly Y2.......... ..] E IR x*, we write the risk as
as X = X2.........
R (X) Y
where (X) = [So (x1),... ..., So (x.)] E R1x=, We split the optimization (1) using gradient descent at step with learning rate 17 into two steps:
0. = 0 17 V
(4)
0: = 0. - An
0u) -0.-
(5)
where the first step is the gradient descent on R()) and the second step the gradient descent on the regularization term à e 0. 2
Proximity to the Pretrained Model Promotes Linearity
We first justify why we regularize for proximity to the pretrained model by showing that this promotes similarity between the final fine-tuned solution and its linearized counterpart. For any yx,y) E Dr, the latter is defined
V fe
foo
H
fa
fa.
Ja, (x) = So. (x) + (V (x), 0. -0) (6)
F
where @ denotes the parameters of the linearized model. We set 0g = 0g and, therefore, for any E Dr. Jä, (x)
(x).
We list and discuss our main theorems toward this goal, and defer the proofs to the appendix. In Theorem we provide conditions for mono- tonicity of updates in regularized fine-tuning, and use to slightly modify gradient descent to ensure monotonicity. In Theorem 2, we de- rive an upper bound on the distance between the parameters at step t of the regularized fine-tuned model and the pretrained model, i.e.. ||0, - Ooll In Theorem 3, we find the distance between the inference of the regularized fine-tuned model (X) and the linearized model (x) i..., (X) (x)) which is the primary contribution of this section.
Figure 1: The NTK defines linear function space H tangent the non-linear function space F defined by the model. Regularized fine-tuning in the lazy regime is close to kemel regression on the tangent space. fo (x) is the fine-tuned model obtained by em- pirical risk minimization. If fine-tuning remains in the linearized regime, then after T steps of training for (x) xe fe, - (Vojo, (x), 0r - -b)) is a good approximation.
3

---

Theorem 1. Under the squared loss. for any t > 0, if A > 0 and VoR(0,) (0, - -00) 2 0, then
हi Il. (X) Y -
Moreover, if (oR(()) (0, 0o) 0, then À 0 is sufficient condition for (7) to hold.
Proof. See Appendix B.
In fine- tine-tuning. the objective in (1) non-convex in the parameters. However, according Theorem sufficient condition for regularized fine-tuning to have non-increasing |o (X) Y is that at
step have Ve, Vo, R(e,) (6. Ou) or = 0. Therefore, we modify the regularization step
in (5) as
0. An 0, 0o) if Vo, R(0,) (0. 0n) 2 0,
0:
O₂
V.. R(0,) (0, - 0,) <0
This is selective regularization scheme, which decides on regularization of each trainable parameter based on the direction of its gradient. Intuitively, the inner product Vo, (o,)) ) (0, 0o) reflects how aligned the current progress is with the negative gradient update direction. A negative inner product signals consistent improvement in loss, while a positive value suggests movement toward a higher-loss region. As result, in order have non-increasing loss at step t, (8) regularizes only the trainable parameters with positive inner product values [9].
Theorem 2. Consider the selectively regularized fine-tuning solution, under squared loss. Denote the instantaneous value of the regularization parameter by >t, which can be either or X. If fo(X) Lip(f ()) Lipschitt in an b-zball of radius r around pretrained parameters G. we have
|00 - Ooll < 22ip(s (xx -YI L e((A--Ao) da, where As = 1/" A
(9)
In the special case when X X (remains constant), we obtain
0 Oull s 2ipp(f))lfe -YI 1 At
Y1I A
(10)
Proof. See Appendix C.
Intuitively, one can think of (9) as placing the "average" X, 1 A1 in (10). In what follows, we adhere to (10), for simplicity. Note that conventional fine-tuning, À Since limx-s0 1-e-Ar t the bound in (10) recovers
00 - 0oll s 2Lip(f) Ilfe, (X)
Y|
(11)
which follows Theorem 2.3 in [16], in the when the rescaling parameter is In other words, regularization and rescaling both manifest the lazy training phenomenon, but regularization maintains
much closer proximity, since - is bounded, unlike t.
Theorem 2 primarily a building block for Theorem 3; it simply says that while the solution may deviate from the origin under regularization, we can bound that deviation. Theorems 3 and 4 show that we can jump from deviation from the initialization bound, which is the direct objective of regularization, to a bound on how far the NTK solution and the regularized fine-tuning solutions are, i.e. the indirect but main benefit of regularization. This is what allows us to place the approximation of Section $4 on a stronger footing than prior works [1-13) that assume that such proximity simply holds. For simplicity, we consider the X constant case to prove Theorems 3 and 4.
Theorem 3. Under the squared loss, if So(X) and V (X) are Lip( L-ipschtss and Lip (-))- Lipschitz in an by-llll of radius r around Og respectively, we have
1x), So. (X)1l 5 (6(1--
where
(12)
- x(|+1)
6 b= (=2iip(f)(lfee(xx)
(13)

---

Proof. See Appendix D.
In the following theorem, we show that, for a proper choice of the regularization parameter X,
linearization of fine-tuning only depends on the local properties fo (X) around Og.
Theorem Let the loss be the squared loss, (x) and V (X) ) be Lip(fh-Lipschitz and Lip V f)-
Lipschitz, respectively, in an by-ball of radius around 0y. Define
X 2115o, (X) -Yl Lip(f)
If 2 Do, then for all t, the following holds.
If X < Xo, then the following holds for t $ 1/² in i-A/Ao
life (X) fo. (x)1 (2 Y|l (+ XII -XI++
In particular, for X 22 Xo. the bound from Theorem 3 always holds and simplifies to:
fo (X) - (x)1l 1 (2rlip(()) + lip(s)) t.
Proof. See Appendix E.
While the distance between the pretrained and fine-tuned models can be substantial, the results illustrate that this gap can effectively constrained through regularization. Under this condition, fine-tuning can be modeled by NTK regression [15], as discussed in the following section.
The takeaway of Theorem 4 is that the regularization determines the time scale which the ap- proximation holds, as well as how tight the approximation is. Theorem 4, at first glance, seems to suggest that a larger value of À better bring NTK and regularized trajectories together. However, larger À means that we effectively force ourselves to remain in an even smaller ball around 6, and may suffer in terms of fine-tuning performance (see À 50 in Table 1). The "sweet spot" regularization parameter can be read from Theorem 4. It should be proportional to the smoothness of J, the pretrained error, and inversely proportional to the radius where the smoothness assumptions hold. While X is in terms of Lip(J), Theorem 3 and (13) show that there are also diminishing returns for pushing X beyond Lip(V due to the +1 in this term.
4 Fine-Tuning Meets Neural Tangent Kernel Regression
We formally define the fine-tuning problem as regularized function estimation in the reproducing kernel Hilbert space (RKHS), H. generated by the NTK, k(x, x') = (oo(('\/((x'''
As shown in the previous section, regularized fine-tuning is in the linearized regime. Therefore, fe. (-) can be approximated by its dual in the tangent space, as shown in Figure 1. We can interpret the linearized model as kernel method with feature map given by Φ(x) = Vefe, (x). The corresponding kernel induced by this feature map is the NTK. Jacot al. in [15], showed that gradient descent with an infinitesimally small learning rate is equivalent to performing kernel regression with the fixed NTK. Thus, the empirical risk minimization (1) is approximated by kernel regression when the kernel is the NTK [15].
Despite the fact that NTK analysis often considers the entire model, it straightforward to specialize it to the fine-tuning case, which we do here for completeness. When using mean squared error, fine-tuning with the linear model is similarly equivalent solving the kernel regression problem presented in (14). Let H. be the reproducing kernel Hilbert space (RKHS) endowed with positive definite kernel function k(-,)) i.c..
n
= 1() 10) Σ ack(,,x.)
5

---

Assuming the solution lies in or close to this Hilbert space, then as an alternative to (1), we solve
minimize SEH
1f(x) + (xy))Dr
Σ
(14)
where || 11 is the norm corresponding to the inner product ->x defined on the RKHS H. The parameter 0 regularizes the problem toward minimum-nont solutions and is qualitatively inversely related to the number of gradient descent steps in the original problem. We thus effectively formulate the fine-tuning problem as regularized function estimation in the RKHS, H. generated by the NTK, k(x,x' V fee, (x) V (x')
According to the representer theorem [19], the problem possesses the closed-form solution
(14) on training dataset Dr = (xi, yi 1
(()) Σ aak = (e,,K(
(15)
where X) k((,x)) x1), ................................
R
Substituting (15) in (14), we have
minimize Bxx,y)-Dr a| (((,X) Y +
(16)
which is a convex problem with a K
oI Y as the solution. Equivalently,
(-) K (, X K X X oI Y
(17)
where [KX,, X)).. = k(x;,x,) For the sake of brevity, hereafter, we use K(X, X) and K interchangeably.
Under linearity, we now show through Theorem 5 that the spectrum of the NTK directly affects the empirical risk, making it tool that can be used for making fine-tuning decisions.
Theorem 5. The empirical risk is bounded as
σ I (
(
Y
SR(O))
S
(18)
Ameax K)
Aumin K
where Amin (K) and Amor (K) are the minimum and maximum eigenvalues of K(X, X), respectively.
Proof. See Appendix F.
Theorem 5 motivates us to study the regularized condition number R(K + ol)
Aun Amn(K)+ (K)+a as an at-initialization metric for predicting the performance of fine-tuning. To show how this can be useful, for example, in selecting what subset of parameters tune, we next study the randomized kernel specifically for fine-tuning, which enables us to investigate the training dynamics based on the influence of each individual trainable parameter on the bounds in Theorem 5.
Auaa +o
5 Spectral Perturbation of Layers
We have shown that (i) regularized fine-tuming is well-approximated by its fully linearized counterpart, (ii) that optimizing the linearized fine-tuning is equivalent NTK kernel regression, and (iii) that the kernel's 's spectrum is predictive of the fine-tuning performance. To put these facts to use, in this section, we study the effect of layer selection on the spectrum of the NTK, and consequently on the empirical risk of fine-tuning. Let 0' be the parameters of the layer from the pretrained model and {0', ( 0²) be the set parameters of the layers from the pretrained model. The NTK, when the parameters in are chosen for fine-tuning, is [K],, = E'= Vorse (x,) Vo The kernel induced by 0¹ is S E R=x=, where
[ilks = Vo (xi) Vo TT
(19)
One can infer that the NTK induced by all layers is K Σls St.
The following theorems quantify the effect of adding a set of trainable parameters on the spectrum of the NTK.

---

Dataset
Hyper-Paramenn À
10
0.5
0.1
0.0
10. Bulla
0.280
0.350
0.404
0.5263
0.6148
0.6946
0.960
[l,,(X) (r.()))Ja (x)
1.12
1.39
1.25
1.32
CoLAA
KL Divergence
0.1060
0.1377
0.200
0.1613
0.1788
0.1961
0.1599
0.210
Evaluation Accurncy of fe. (x)
74.59
79.57
80.44
79.38
80.24
80.15
80.15
79.67
110, Golla
0.292
0.336
0.369
0.424
0.520
0.700
1.589
2.519
[x,,(X)- -
12)
2.303
2.635
2,957
3.217
3.331
3.397
2.791
SST-2
KL Divergence
0.320
0.433
0.476
0.517
0,545
0.560
0,578
0,540
Evaluation Accuracy of fe. (x)
89.3
91.2
91.5
92.4
92.8
93.0
92.4
91.6
Table 1: Sweep over the hyperparameter (A). Increasing regularization strength, i.e, larger A, reduces the deviation between the regularized fine-tuning and linearized models at one snapshot of fine-tuning step t. Accuracy is largely unaffected by regularization.
Theorem 6.
Let K. be the NTK with respect to the of selected fine fine-tuning parameters, and S be the kernel with respect to the parameters of the candidate layers, to add to the fine-tuning parameters. Then
(1 (1- (K) ((K+ (K))))
(20)
where = |K-1/25 K- K-1/
Proof. See Appendix G.
The spectral perturbation bounds of Theorem 6 can interact with the risk bounds given by Theorem to help us further understand the relative merits of parameter choices. We denote by R(0) and R U 0), respectively, the empirical risk of the model fine-tuned with parameters 0 and U U.
Theorem Let K be the NTK induced by the trainable parameters in 0, then if n(K + ol) S c, we have
Amaz (K oI)
R(OUO) R(
a.Amas (Ka
(21)
ammo (K oT)
Amor (K oI)
where (1-1)2+ 1 Vofe (x:) V Volo(x;)
KK-S/SK -1/1| and S is the kernel induced by O with [S]a.
Proof. See Appendix H.
The above result implies that, by incorporating an additional layer to the parameters being fine-tuned, one can expect log R(0L)) R(Ø)
log ( S)+ log a, log (m=)+++) + log (K)+
Auas (K)+o
Corollary 1. When we have two candidate layers @) and 0 mutatis mutandis, we have
Amaz KK + S1 +
RiO J0)
bdmar K+ +ol
(22)
bXmax (K S2
R(0 U 02)
Amar (K + S2 ol)
where a1 a, (I-n, 11: = ||| 1/25, K-1/2 1/]
{1,2), and St is the kernel induced
by 0' with [Si]k, = V (x:) V Vi Tx
As such, one can evaluate the candidacy of layers for fine-tuning based on Corollary 1. In order to anticipate the empirical risk of fine-tuning, at initialization we look at a confined interval, i.e.,
log R(QU02)
R(OL0t) [oo
Arno (K+ log b, log Am.. (knn(K+ -log
Aena (K+S2)+0
Aan (K+S2)+o
Experiments
We now demonstrate that insights from our theory apply in practice, despite moving from squared loss to cross-entropy and from gradient descent
AdamW W optimizer, which we use for all fine-tuning in conjunction with the selective regularization (8). In our experiments, we implement LoRA on RoBERTa base and evaluate its performance on the General Language Understanding Evaluation (GLUE) benchmark [20], IMDb [21], and Yelp [22] datasets.

---

6.1 Model, Datasets and Optimizer
RoBERTa base incorporates several key modifications to the pretraining process, such as using larger batch sizes, longer sequences, and more diverse data than its antecedents like BERT [23]. Despite its relatively compact size of 125M parameters, RoBERTa base has proven to be one of the most powerful models for various NLP tasks, including text classification, question answering, and named entity recognition, especially on the GLUE benchmark.
The GLUE benchmark is collection of diverse tasks that test model's natural language understand- ing abilities. The tasks included in our experiments are linguistic acceptability judgment (CoLA [24]) and sentiment analysis (SST-2 [25]). The GLUE benchmark provides comprehensive evaluation of a model's 's performance across various NLP challenges, assessing its ability to understand and reason about language in different contexts.
The IMDb dataset is a large dataset for binary sentiment classification, containing 50k highly popular movie reviews from the Internet Movie Database (IMDb). The Yelp dataset contains customer reviews from Yelp, a popular platform for crowd-sourced reviews about businesses, primarily restaurants. This dataset originally contains reviews with ratings from to 5. To convert it into binary classification task, we consider reviews with ratings less than as label 0 (negative sentiment) and those with ratings greater than equal to label (positive sentiment).
Table Appendix shows specific hyperparameters for RoBERTa base across various benchmarks, including GLUE tasks (CoLA, SST-2), Yelp, and IMDb. For all experiments, we use LoRA on the RoBERTa-base model from the Hugging Face transformers library [26], and report its performance on different tasks. We implemented our code NVIDIA Tesla V100 GPUs. Following Hu et al. (2022), we mostly use the weights of the query and value layers, W, E IR' and W, E gmxp for fine-tuning. In experiments, trainable parameters per selected layer for each of the query, key, and value projection matrices the self-attention mechanism in the RoBERTa base model.
apply LoRA with 8, which has (m + p)
x 768
6.2 Linearization of Fine-Tuning
In Table I, analyze how regularization impacts the fine-tuning of models the CoLA and SST-2 datasets, using several metrics, including the deviation of fine-tuned parameters from the pretrained parameters, 0, 0oll. KL divergence between predictions of the regularized fine-tuned model and its linearized counterpart, the & norm fl, (X) - fe. (X) | evaluation accuracy. The results indicate that increasing the regularization strength, X, reduces both the deviation in parameters and prediction divergence, measured by KL divergence and || fe. (X) fo, (X)ll, implying that stronger regularization keeps the fine-tuning behavior closer its linearized approximation. Conversely, as regularization weakens, À - 0, parameter deviations and divergences from linearization increase, suggesting significant departure from the linear regime. This is well aligned with the bounds
Dataset Selected Layers
Selected Parameters
Condition Number
Train Loss
Evaluation Loss
Evaluation Accuracy
[0]
11.618
0.5271
0.5265
73.15
[0,11]
W
W.
9,490
0.5174
0.5293
73.15
CoLA
(0.5.11)
W
W.
7,503
0.5093
0.5272
73.44
(0,5.11)
W
2,320
0.5128
0.5357
73.25
W W
3,195
0.4794
0.387I
83.48
[0,11]
W W.
6,413
0.4677
0,3916
82.79
SST-2
(0.5.11)
W.
6,792
0.5078
0.3913
82.68
(U.5,1T)
W
4511.4
I14717
U.3893
83.60
[0]
W W.
274
0.29
0.2597
88.20
[0.11]
W. W.
4.167
0.2882
0.2596
88.24
Yelp
(..
W W.
1,336
0.2885
0.2596
88.21
(0,5.11)
W.
39.33
0.2865
0.2596
88.23
[0]
179
0.3512
0,2702
89.56
[0,11]
W. W.
5,899
0.3597
0.2717
89.50
[0.5. 113
W. W.)
1,277
0.3709
0.2727
89.49
(0,5.11)
(WE)
9.605
0.3642
0.2719
89.49
Table 2: RoBERTa-base model's performance GLUE tasks, condition number of the NTK, train loss, and evaluation loss snapshot of the training 10-th epoch. LoRA with is used for fine-tuning The Ama (K)+
condition number is calculated as x(K + oT)
and o 1e fixed among all tasks.
Auai (K)+7*

---

Exalnatton Accuraty (N)
Co CoudBiie Namtthe
Balaaik
2072
Datamet
304
(a) Train loss over 10 epochs
(b) Condition number
(c) Evaluation accuracy and Condi- tion number
Figure 2: (a)-(b) Illustrate the positive correlation between the convergence rate of optimization steps of LoRA over 10 epochs and k(K + aI) of NTK at initialization. {W g: W.) of layers {0, 5, 11) are fine-tuned. (c) Illustrates the negative correlation between evaluation accuracy after 10 epochs of training and the condition number of NTK. LoRA with r is used to fine-tune [W&} of the layers {0, 5, 511..
in Theorems 2 and Notably, the overall accuracy remains largely unaffected by the regularization, and its fluctuations are within the error margin for À up to 10.
The key takeaways from Table are that, long as don't overregularize (A not too large, e-g.. smaller than 50), we simultaneously: (I) encourage the model not to deviate (the direct objective of regularization), (2) encourage the NTK solution and the regularized fine-tuning solutions become closer (the indirect main objective of regularization), and (3) do not sacrifice much on performance. We conjecture that the reason we don't 't sacrifice much on performance that the fine-tuning solution is already relatively close to the initial model and that there are equivalently good NTK solutions in the same neighborhood. (The choice of is similar searching for solutions within sequentially smaller neighborhoods around the initialization, as À increases.) The trifecta of easy implementation, no performance sacrifice, and theoretical compatibility with the NTK framework makes our proposed regularization an especially attractive choice.
6.3 NTK Evaluation
We verify our proposition that by calculating the condition number of the NTK matrix for the LoRA model at initialization, we can predict the generalization error, including evaluation loss and accuracy. Table 2 presents training loss, evaluation loss, accuracy, condition number of the NTK before fine- tuning, based on the snapshot at epoch 10. We vary LoRA parameters across different layers ((0), (11), (0,11), 10, for query and value parameters, and layers (0,5,11) } for key parameters, various tasks and datasets. We collected selected from the training datasets and computed k(x;, xj) with respect to trainable parameters, A and B of LoRA. The final empirical NTK matrix is K(X, X) E R32x32 Our work is not specifically empirically illustrate
[x1, [x, ......, X,,] using n 32 samples, randomly
studying optimal sketching of the kernel matrix; however, in Appendix that our numerical results are robust to the choice of NTK samples. Note that the number of samples used for calculation of the empirical NTK orders magnitude smaller than the training dataset for sampling. This shows that the results remain valid even with sketch of the full kernel. This finding highlights the great potential of kernel methods for large language models (LLMs), particularly in terms of efficiency.
At first glance, in Table the highest condition number does not correspond to the lowest accuracy. However, one may notice that this is an issue primarily when the nature (query/value vs. key) and number of parameters change. When this is the case, the effective complexity the fine-tuned model changes, and since the NTK parameter a is inversely proportional to the complexity, it affects the regularized condition number. In the table, we have fixed and so this change not being reflected and it is only fair to compare instead cases where the nature and number of parameters are the same, e.g., fixing a layer and varying tasks among CoLA, SST2, IMDB, YeLP. The correlation is perfect for {0, 11}, but also mostly tracks for other fixed choices. This also holds when varying the same number of layers like (0, 7). (0, 11) for CoLA and SST2 Fig. 3. It would be very interesting to factor the change of into the layer selection methodology, but that is great direction for future research. More extensive numerical results for decoder-only models, such as GPT-2 and OPT-125M, provided in Appendix J.

---

Training time and NTK calculation time are reported in Table 3 for fine-tuning { {W&} in layers {0, 5, 11]. For this scenario, the number of total trainable parameters (TTPs) 0.628M, and the number of chosen NTK parameters is 36.8k. As shown in the table, fine-tuning, even with just 10 epochs, has significantly higher computational overhead than computing the NTK. This finding supports the advantage of the present approach in terms of time complexity when comparing the risks of different datasets without training. Additionally, since Yelp and IMDb are larger datasets, it is evident that fine-tuning on them requires more time compared to the others. Figure 2(a)-(b) illustrates the positive correlation between the condition number the NTK matrix at initialization and training loss for different tasks. In all datasets, the attention parameters of layers {0, 5, 11] are fine-tuned, and evaluation accuracy was reported. Although for CoLA, it is customary to report Matthew's 's correlation coefficients [27], we adhere to reporting the evaluation accuracy for all tasks in Figure 2(c), to maintain consistency in the evaluation metric across different datasets. Figure 2(c) starkly illustrates inverse relationship between the condition number of the NTK and the model's evaluation accuracy. In our experiments, we observed that Amin (K) is almost always close to zero and the regularized condition number, s(K + oI), is tracing the spectral norm or Xmax (K). For instance, the CoLA task, which exhibits the highest training loss, also shows the largest condition number. This suggests that by computing the NTK matrix before training, we can identify which tasks are well-conditioned, i.e., a lower condition number indicates lower training and evaluation loss. Figure 3 in Appendix evaluates the bounds of Theorem 7, where 0 represents the weights {W of layer {0), and ê denotes the candidate layers. The empirical risk ratio and maximum eigenvalue ratio closely follow each other in most cases. Theorem conveys that the spec- trum of the NTK directly affects the empirical risk. Therefore, we propose to select the layers based on the eigenvalues they induce in the NTK. For instance, as seen in Figure 3, for the CoLA dataset, the eigenvalue analysis at-initialization suggests that layers {0, 7} should be selected for fne-tunnng, and it also shows the least empirical risk compared to others. Similarly, for SST-2, {0, 5, 11) } is the optimal choice by eigenvalues and is once again consistent with the empirical risk after fine-tuning.
Dataset Fine-tuning Time NTK Calculation Time
CoLA
187
33
SST-2
794
63
Yelp TVDb
46,096
245
1,541
55
Table 3: Fine-tuning time(s), NTK calculation time(s),
{W& J of layers {0, 5, 11] are fine-tuned. In all datasets, only 32 random samples from the training calculating the NTK.
used
7 Conclusion and Limitations
In this paper, we tackled the challenge of guiding design decisions parameter-efficien fine-tuning by offering insights through linearization. Fine-tuned models are often implicitly close to the pre-trained model. We made this proximity explicit and showed that it results in a model that remains close to the linearization of the pre-trained model. We used the linear model's ' NTK regression formulation to show that the NTK kernel spectrum can be a predictor of fine-tuning performance, at-initialization (before actual tuning). We then analyzed how this spectrum is affected through layer selection and showed how this can experimentally guide the decision of which layers to tune, saving valuable time and computational resources. Some of the limitations of the theory are that it applies to squared loss and gradient descent, though the experiments show that the insights carry to other losses (such as cross-entropy) and stochastic solvers (such as AdamW). The experiments on RoBERTa with LoRA across multiple datasets (GLUE, IMDb, and Yelp tasks) demonstrate reasonable correspondence between theory and practice, despite some theoretical assumptions differing from experimental conditions. Discussion on the implications of the assumptions is provided in Appendix M.
Acknowledgments and Disclosure of Funding
This paper is based upon work supported in part by the National Science Foundation, through the NSF CAREER Program under Award No. CCF-2146333 (From Rare Events to Competitive Learning Algorithms),, the NSF HDR TRIPODS Phase II Program under Award No. SCCS-221702 (IDEAL Institute). Computational infrastructure was supported in part by the NSF MRI Program Award No. 182-182 (COMPaaS DLV).
10

---

References
[1] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar al., "LLaMA: Open and efficient foundation language models," arXiv preprint arXiv:2302. 13971, 2023.
[2] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, Gehrmann et al. "PaLM: Scaling language modeling with pathways." Journal of Machine Learning Research, vol. 24, no. 240, pp. 1-113, 2023.
[3] J. Howard and S. Ruder, "Universal language model fine-tuning for text classification," Associa- tion for Computational Linguistics, pp. 328-339, 2018.
[4] E. Ben-Zaken, S. Ravfogel, and Y. Goldberg. "BitFit: Simple parameter-efficient fine-tuning for transformer-based masked manguage-models." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, 2022.
[5] N. Houlsby, A. Giurgiu, S. Jastrzebkki, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, "Parameter-effiiien transfer learning for NLP," International conference on machine learning, pp- 2790-2799, 2019.
[6] N. Ding, Y. Qin, G. Yang. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen et al., "Paraaeter-efffciient fine-tuning of large-scale pre-trained language models," Nature Machine Intelligence, vol. 5, no. 3, pp. 220-235, 2023.
[7] B. Lester, R. Al-Rfou, and N. Constant, "The power of scale for purameter-efficien prompt tuning," " Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
[8] A. Tomihari and I. Sato, "Understanding linear probing then fine-tuning language models from ntk perspective." arXiv preprint arXiv:2405. 16747, 2024.
[9] J. Tian, C. Huang, and Z. Kira, "Rethinking weight decay for robust fine-tuning of foundation models," in The ehirty-eighth Annual Conference Neural Information Processing Systems.
[10] A. Deshpande, A. Achille, A. Ravichandran, H. Li, L. Zancato, C. Fowlkes, R. Bhotika, S. Soatto, and P. Perona, "A linearized framework and new benchmark for model selection for fine-tuning," arXiv preprint arXiv:2102. 000004 2021.
[11] Mu, Y. Liang, and Y. Li, "Gradients as features for deep representation learning," in Interna- tional Conference on Learning Representations.
[12] G. Ortiz- Dimizeiimneez, A. Favero, and P. Frossard. "Task arithmetic in the tangent space: Improved editing of pre-trained models," Advances in Neural Information Processing Systems, vol. 36, pp. 66 767-666777 754, 2023.
[13] U. Jang. J. D. Lee, and E. K. Ryu, "LoRA training in the NTK regime has no spurious local minima," Forty-first International Conference Machine Learning, 2024.
[14] Z. Afzal, T. Esmaeilbeig, M. Soltanalian, and M. I. Ohannessian, "Can the spectrum of the neural tangent kernel anticipate fine-tuning performance?' in Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning.
[15] A. Jacot, F. Gabriel, and C. Hongler, "Neural tangent kemel: Convergence and generalization in neural networks," Advances in neural information processing systems, vol. 31, 2018.
[16] L. Chizat, E. Oyallon, and Bach, "On lazy training differentiable programming." Advances in neural information processing systems, vol. 32, 2019.
[17] S. Malladi, A. Wettig, D. Yu, D. Chen, and S. Arora, "A kernel-based view of language model fine-tuning." " International Conference on Machine Learning, pp. 23 61061023366 641, 2023.
[18] L. Xuhong, Y. Grandvalet, and F. Davoine, "Explicit inductive bias for transfer learning with convolutional networks," in International conference on machine learning. PMLR, 2018, pp. 2825-2834.
[19] B. Ghojogh, A. Ghodsi, F. Karray, and M. Crowley, "Reproducing kernel Hilbert space, Mercer's s theorem, eigenfunctions, Nyström method, and use of kernels in machine learning: Tutorial and survey," CoRR, vol. abs/2106.08443, 2021.

---

[20] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, "GLUE: A multi-task bench- mark and analysis platform for natural language understanding," Association for Computational Linguisties, pp. 353-355, 2018.
[21] A. L.
Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, "Learning word vectors for sentiment analysis," Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
[22] X. Zhang, Zhao, and Y. LeCun, "Character-level convolutional networks for text classification," Advances in neural information processing systems, vol. 28, pp. 649-657, 2015.
[23] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding." Association for Computational Linguisties, 2019.
[24] A. Warstadt, A. Singh, and S. R. Bowman, "Neural network acceptabilityy judgments." Transac- tions of the Association for Computational Linguistics, vol. 7, pp. 625-641, 2019.
[25] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, "Recursive deep models for semantic compositionality over sentiment treebank," Proceedings of the 2013 conference on empirical methods in natural language processing, pp- 1631 1631-1642, 2013.
[26] Wolf, Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz et al., "Transformers: State-of-the-art natural language processing," Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 38-45, 2020.
[27] B. W. Matthews, "Comparison of the predicted and observed secondary structure of T4 phage lysozyme," Biochimica et Biophysica Acta (BBA)-Protein Structure, vol. 405, no. pp. 442 451, 1975.
[28] M.-Y. Chung, J. Fan, H. Ye, Q. Wang, W.-C. Shen, C.-M. Yu, P.-Y. Chen, and S.-Y. Kuo, "Model reprogramming demystified: A neural tangent kernel perspective," arXiv preprint arXiv:2506. 00620, 2025.
[29] R. Mathias, "Spectral perturbation bounds for positive definite matrices," " SIAM Journal on Matrix Analysis and Applications, vol. 18, pp. 959-9880 1997.
[30] Y. Chen, E. N. Epperly, J. A. Tropp, and R. J. Webber, "Randomly pivoted cholesky: Practical approximation of kernel matrix with few entry evaluations," Communications on Pure and Applied Mathematics, vol. 78, no. 5, pp. 995-1041, 2025.
[31] L. Bottou, F. E. Curtis, and J. Nocedal, "Optimization methods for large-scale machine learning." " SIAM review, vol. 60, no. 2. pp. 223-311, 2018.
[32] S. Borade and L. Zheng, "Euclidean information theory," in International Zurich Seminar Communications. IEÉE, 2008, pp. 14-17.
[33] D. Savostianova, Zangrando, G. Ceruti, and F. Tudisco, "Robust low-rank training via approximate orthonormal constraints," Advances in Neural Information Processing Systems, vol. 36, pp. 66 064-66 083, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support our claims with theoretical proofs, which further validated by numerical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Appendix A, includes the definitions and lemmata that are repeatedly used in the theorems. Moreover, the proofs of all the theorems are provided in the Appendix. Where assumption is made, it is clearly stated in the proofs and in the main text.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The models, datasets, optimizer and hyperparameters are clearly explained in the main text and in appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The codes will be uploaded supplemental material and references to the datasets, which are open access, is provided the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
hwtpbbmmsssoooolo11m1omm onPolicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpps.ccpppliiiii/mmm/tao./nttc wwtppccppcc////ccccccccccccccclt htpplli//giieee http//uuiieeeeoeeeeee https//ooSSSuuiiseeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The optimizer and hyperparameters are clearly explained in the main text and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We support the main claims of the paper with a comprehensive set of experi- ments over different datasets and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: It is stated in the paper that "we implemented our codes on NVIDIA Tesla V V100 GPUs". Moreover, computation times are provided Table 3.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https: http::::///eerreeeeeeeeee httpa//ppp...ppppca/cc/ccccceeee
Answer: [Yes]
Justification:
This paper conforms to NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work contains foundational research and not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not contain models or datasets that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Code, data and models used in this paper are open access and explicitly mentioned and properly referred to as below:
RoBERTa-base Licensed under the [MIT Licunn//tttss//ppprsssrrrooooeor
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddataees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The assets are well-documented in the paper and the accompanied codes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used in this works.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettpa://neurdae for what should or should not be described.
19

---

A Definitions and Lemmas
In all the following proofs. we apply the notations and definitions below. Given the data matrices (X. Y), X E Bxxd and Y E IR 1x we define the models inference at time step t as
Y()) = fa, (X) E R
(23)
The gradient fe, (X). by convention, is in (n
(0) The gradient of (3), the mean squared error,
Ve. ((..) = (Y(() Y) ff,
(24)
On the other hand, the inference of the linearized model is defined as
(t) = Fo. (X) = fo, + (o (X), 0. 0o).
(25)
where 0 denotes the parameters of the linearized model, and we assume O = 0o and therefore
Jo. (X) = feo (X).
For all vectors, we use the f2-norm. For all matrices, we use the & induced norm, or spectral norm, defined as ||A||| = Aui(]|Au|| : ||u]12 S 1}, which is also equal to supil/uA]| ||ul|2 S 1} and the largest singular value of A.
Lemma 1. If for all 0 in a given range, (o(X) is norm) and (X) is Lip(Vf))Lippschis in 0 (w.r.. the lz induced norm), then kg (X.X)
in O (Wer. the lz-
So (X) [X)) E BRux is Lippk)t in 0 (wart. the l2 induced norm), with
Lip(k) & 2 (f) Lip(
(26)
Proof. First, since So is Lip( (bpissitiss it follows that the Eg induced norm of V fo is bounded by Lip( To see this, for any dim (0) -idiiennial vector w, introduce the single parametrized family ((6) = + 5w. Then d
do Secs) V Veeo w. By Lipschitzness, ([ - Seco, S Lip(f)o][ w|]-
Taking the limit + 0 we obtain that f//l S Lip(f)l[wl]
Recall that kg [X is the NTK matrix. Since all arguments of kg. fe. and Vee are X, we omit them for clarity. To show that kl is Lipschitz and compute its Lipschitz constant, given any n--dimensional unit vector v, we need to bound
I[kk - k = 1
V)(V() v/l-
(27)
We write
(ke - kcv fe - v.
(28)
which yields, by the triangle inequality, norm bounds, and Lipschitzness,
11ke k<)v/l < V/o (v vf()v] [| - Vfcc) V
(29)
< f/ || ( -Vf( | VV Vfcll
(30)
< Lip(f) V Vfq]l+ Lip(V/ <1I [Vfcll
(31)
< Lip(f)Lip(V/f)ll) <[| Lip((\)) - LI()ip())
(32)
Lip -Cll
(33)
which proves the claim.
L
Lemma 2. The NTK matrix K Ruxx, defined by [K] k (xx: Ys), is positive semidefinite.
Proof. For all v E IR", we have v Kv KV=vTV/oVVTV [ vl 2 0. Therefore, K is positive semi-definite.
Lemma 3. Let O, be the gradient limit of the regularized fine-tuning gradient descent described in (8), If we assume that X switches most countably often and denote its instantaneous value by X then O₂ satisfies the following differential equation:
d 0, - >r (0. -0)
(34)
dt
20

---

Proof. Subtracting Og from both sides of (5) yields
0 0o 0, - 00 -n 00)-
(35)
equivalently, (8) becomes
0, - 0o + A1) (0.-0))
(36)
where
X = A Vo, R(0,) (0, - 0o) 2 0,
(37)
otherwise.
On the other hand, by combining (4) and (5) for t + we have
O Oz+1 02 -V.()))) - -et1 Oc+1 - -00)
(38)
0: (V)())) 1111 (0c+1 - 0o)
(39)
A441
where we replaced 0z+1 - 0o 1-he+1 11 (0,-1 - -00) from (36) in (38) to obtain (39). By rearranging the terms we have
0t+1 (1 det1 11) (0. ( 1 x 17
A++1 0u
(40)
= (1 As11 0) 711 A4+1 7) V((()) T + 1
(41)
Subtracting O, from both sides and division by v) yields
0 0++1- O: ~1+110 (1 ---++1 VR(O.) + Ae4100.
(42)
d 0, vR(0,) de (0, Oo)
(43)
dt
To obtain the last equation, take the limit as 17 0 from both sides such that 0.-1 P.-1-0, and Xnj 1.
d0(()
di
B Proof of Theorem 1
From Lemma 3, we reproduce (34), assuming for the moment constant X:
d 0, = - A (0: 00)
(44)
Using chain rule on (23), and substituting for VR(0;) using (24), have
d
d
dt Y(t) v fa, (X) dt = -2 V fe. (X) V (X) (Y Y) fo, Oo),
Oc.
-22k; ((())= Y) fo, (X) (0, 0o)),
(45)
To reduce clutter, we again remove the explicit dependency of the kernel on input and denote (X, = V fo. (X) f (X) by k, hereafter. By replacing Y(t) = fa. (X) and (45) in below equation, we have
1 d di ((t) YI2 1 d
-) d Y(1), TVV (X) (0, - 0b), = --2( (Y - (Y) - - V (X) (0, 00)
d/ (Y(() - Y) (X())
(46)
=-2(Y(1) - (Y(() -)-X(Y()) -
Y)
(47)
(48)
21

---

Finally,
1 d
A VoR(00) (0:- 00). (49)
(t) EI -2( (Y Y)-6)--),,C (Y -)) - 2 di
Since k, Vo, (X) (X) is positive semidefinite, we have (k, (Y(1) (Y(I- Y))
To ensure that the error (())= Y| does not increase over time, it is sufficient that the gradient ((,)))) is aligned with the parameter update direction, i.e..
(o - 0o) 2 0. If this condition is not satisfied, then we should set À 0. This condition is aligned with the selective lg regularization introduced [9].
C Proof of Theorem 2
Once again, use (3), substitute = - Y) V (X) in (34), yielding = 2 0 = 2V So, (X) T (Y(1) - Y) - A (0. 0o)
(50)
dt
Let u, 00 and wt)) ||u 1|2 The dynamics can be rewritten as
dl u, 2 fo, (X) (Y(1) Y) - X us-
(51)
Since u, is continuous, w(l) is a.e. differentiable and, for u, + O.
u, d
i(t) = | 2| dt un
(52)
Substituting (51) into (52) and writing the unit vector ú, uu/l|u
it) 2 ú V fo. (X) (Y Y) A w(t).
(53)
By Stuchy-Schwar and recalling that the lg induced norm Ve (x)Il S Lip(f) (see the proof of Lemma 1).
0, v (X) (Y(()) - Y) s [x), - X1l2 < Lip(f) (t) Y112.
(54)
Therefore, we have
v(ll) S - ~v + 2Li(()) |Y(I) - Y| dt(t) + de w(() 2Lip(s) (I) Yl12.
(55)
(56)
Let
A 1 X dr.
(57)
We multiply (56) by the integrating factor A. and use the product rule
d
(c>x(()) S 2Lip(s) ( - Y|12.
(58)
We integrate (58) from 0 to t
^ ww)() w()) < Lip(ff) / (Y(s)) 21
(59)
Then we divide by d to obtain
w(t) S e Ar w(() + 2ip(ff) / -(Ai-))))) |Y(()) - Y ds.
(60)
< - ((0) + 2Lip(/) X112 1 ((Ae-dd)) ds,
(61)
22

---

where the last inequality follows from Theorem 1. Special case 1. If X >> then A, XL therefore have
||u/|2 - At Lolll 12 + 2 - Y1/2 / c c-A(t-o) ds,
(62)
$ e M 1ll12 + 21ip(() - Y112 Ah
(63)
A
Since u, 0, - -o. |o = 0, and (()) = foo (X), we have
100 0oll2 s 2 1150, (X)- - e -
(64)
A
sip |00, 0oll2 < Lip(f) llfe, - 1
(65)
t-+00
Special case 2. If A = 0, from (61), we have
|u: ll2 S |oll2 + 2iipp() |Y(O) -1 [ d ds.
(66)
Therefore,
|0: - 0oll2 22Lpp( (X) - 11124.
(67)
D Proof of Theorem 3
For simplicity, consider the lambda constant case to prove Theorems 3 and 4. For Y (X)
and (t) = So. (X), we define ((t) = (|Y - (1)l22. We have
1 d (tt) dt
d Y(t) (t)
(68)
d
(Y)() - Y't), Y(t) - Y(t)) + ((t)) (()-
=(Y'(I) Y'(), ((0)) Y()))
(- (-k;2(Y(()-)) -Y(()) A fo, (X) (0: 00) ko ( Y), Y(I)
Y()), (())-((())
where the last equality follows from (45) the proof of Theorem 1. We simplify the following term in the last equation as
k.2 (Y(t) - Y) + k. (Y(t) Y)
(69)
-;2 - Y)+ ko2 (Y(() Y) kn2 Y) + ko YY)
(ko 1Ko-ka (,)- (Y Y) ko (2 (Y( - Y) 2 Y)),
where in the first equality we added and subtracted k,2 (Y Y). Substituting (69) in (68) yields
1 d
2 ((ko 2)) Y) fo (X) (0. 0n) Y(I) Y
2 dl
+ <ko (X)
Y() - (
(70)
since Kg V.g. (X) Vo, (X) is positive semidefinite, the second term on the left hand side non-positive, i.e.,
(ko (2 (Y Y) - (X - Y)), Y() -(()) Y(!) s0.
= -(Y((tt) - (Y))
(o(Y(() (
(71)
As result of (99) and (71), we have
(t)² < ((k 2 - Y) - AV (X) (0, 6o). Y(t) - Y())).
(72)
2 dt
23

---

Noting that 1 4 (tt) = A(t) 2 and taking norms on both sides of the above, then clearly using (36), we have
d dt
(1) || (ko - ()) Y) - AV (0) 00)
(73)
< (kk - k2)
(X) 0u)
(74)
x Lip(k)]l (0: - -0) ( - Y) + Lip(f) |0: 0
(75)
2Lip(k)) ( - + 10: -00l
(76)
x 41ip(())ipp()))XX())) - Y1 (1 (1-e
(77)
A
+ 21ip Y|| (1 c
< (0) YII ( (()) Y| + Lip(s) e-r
(78)
+ ())(())) - YII (2 -Yl Lip(f))
(75) follows from Lipschitz properties of fe (X) and k, which was shown in Lemma 1. Due to the non-increasing property of (t) shown Theorem replaced (() with ( obtain the upper bound. We also used (10) in Theorem to obtain the last inequality.
Using Lemma and (78), have
d A(L) S b b- -1
(79)
dt
where
b = (== - (HLIDVVDINYO))- - -YI00+ x1I+1).
(80)
By taking the integral of both sides of (79), we obtain
- e _r -
A(t) s b
(+ )
(81)
Consequently, have (12).
E
Proof of Theorem 4
From Theorem 2, we have
00 0 s Y Lip(f) 21-
(82)
In order for the regularizer to satisfy the Lipschitz continuity assumptions, mainly that ||0 0u S this shows that there are two phases of behavior depending on how large is. The threshold given by:
2 Y Y Lip(f)
Ao
In particular, if À 2 >o, then O, remains in the r-ball around On for all I. Otherwise, it remains in this ball only as long as
1
1
t < In 1 A/2o
Based Theorem 3, get that
(t) < 21ip())) Yll (X) ++ {111)-
When A 2 Aa. A(t) grows roughly linearly, with coefficient given by: A(t) s ( Y|l (2rLip(V + t.

---

Proof of Theorem 5
Let UEUT denote the eigenvalue decomposition of K (X, where E
Diag (Amin (K),... Amax (K)) and U = Due the proximity of the regularized fine- tuned model to the pretrained model, which promotes linearity, we have
1
n (fo
R(0)-
(x,)
n2
1
y1 K (x, X) + o1
Y
T1
1
Y (X, (X, o1) Y
[I (X)
Y
1
[I -1- UUUUUUT (UEUUUUUUUU. [ 112
R
1
(I- U
7
1
-(-+ U Y
(83)
n
Since I E(E + o1) is diagonal matrix, have
Amia [I- (I-
2 U YI2 R(0) Amax - EI- -(E "
Y!?
(84)
Noting that
Amin (I-E(E + o1)
(85)
Amax (K)
Amax
+ +l)-
(86)
0 + Amin (K)
we have
2² Y
0² Y 2
< R(0)= < (a Amax (K))2
(87)
(0 + Amin (K)
which is also shown in [28].
G Proof of Theorem
Since K is positive semidefinite (see Lemma 2), have
[I+K-1/2S K S = K1/2 (I+K 1/28 1/2S/11/2/ K-122 K2/2.
(88)
Considering 7 = |K- 1/25 K K-1/21|, we have
I < K 1 --12
1/2 < nI.
(89)
Consequently,
(1 (1-1) (1-n)I I I+ K 1/28 K -1/2
1/2 < (1+n)I,
(90)
and, from (88) and (90), we obtain
(1 (1- n K < K + + (1+
(91)
Note that for any two positive semidefinite matrices A and B, if A S B, then M( S AB)... Therefore, (91) is equivalent to (20) [29].
25

---

H
Proof of Theorem 7
From Theorem we already know
(Amin K) + x)²
(Amax (K) + 00)²
(92)
0² Y
R(00)
0²1 Y|1
Theorem 6, states that
Amax (K + S o1)
Amax (K) + >maxx (K + o1) S
(93)
1 1-
and
Amax (K + 01)
Amax (K + + +
min (K) +
(94)
iiK + 01)
<(K + o1)(1+1)
Therefore, we have
Amax
K + S 5 01) [< + a1)(1 7)
1
Amax (K +
(95)
R(0)
|| -1)
On the other hand, it follows from Theorem that and, from Theorem 6 that
o||||
Y
max K +0 <R(OU0)=
(96)
Amin KK + S) +
Amin (K + 2 - 17) Amin K + o
(1 11) mmx K + aI)
k K + ol)
mma (K ()) + S (1 + 17) Mma KK + oI).
(97)
From (96) and (97), conclude
a Y
< o /|| 0T)
S R(
(98)
(1 + 1) Amax (K + ø1)
(1 Amaa (K + o1)
Considering that s(K oI) the inequalities (98) and (95) imply
1
Amax KK + S + 01)
R(OU0) R(0)
Cdmax (K + S ol)
(99)
c(1 + 1)) K + oI)
2 (K + oI)
Note that (1 - n)² S (1 1))
By defining
(1- we can rewrite (99) in the desired form:
Amaa (K + <
R(OV0) R(0)
aAmax (K + +
(100)
a ammax K o
Amax (K + ol)
26

---

I Robustness to Sampling for Estimating the NTK
We empirically illustrate that the eigenvalues of the NTK are robust the choice of NTTK samples. Sketching of the kernel matrices is studied in the literature, for instance, in [30]. In [17], which also modeled fine-tuning as an NTK regression problem, the number NTK samples is fixed 16 and 64 (see Table 2 of [17)).
Random Seed
Amin
Amax
Condition Number
2.04 x 10 6
0.0030
32.43
123
6.45 x 10-? 0.0035
36.85
7.37 x 10
0.0025
26.22
99
3.19 x 10
0.0024
25.82
2024
6.78
10
0.0025
26.86
Table 4: Condition numbers and eigenvalues for different random seeds. We of layer 11 is fine-tuned.
J Additional Experiments
We fine-tuned decoder-only models GPT-2 and OPT- for 10 epochs using the Adam optimizer. LoRA with r = 8 is used to fine-tune W. of the layers {0, 5, 11j. The negative correlation between evaluation accuracy after 10 epochs of training and the condition number of the NTK illustrated below. In GPT-2, Yelp with the lowest condition number, possesses the highest accuracy, and in OPT-125M, IMDb and Yelp, with higher accuracies, have lower condition numbers than the other
tasks.
Model
Dataset
Eval Accuracy (%)
Condition Number
CoLA
71.0
83
GPT-2
IMDb
87.4
36
Yelp
87.6
35
SST-2
64.8
910
CoLA
69.1
720
OPT-125m
IMDb
70,0
210
Yelp
82.1
310
Table 5: Evaluation results for GPT-2 and OPT-125M across different datasets.
K Lipschitzness
To empirically estimate Lipschitzness for all models in balls around the initial model 6g. need to choose models in this ball and calculate the Lipschitz constants for each using pairs of feature points (x, x'). Since it's not feasible choose all models and all pairs, we have sample them reasonably. In the table below, we report on O sampled uniformly on consecutive spherical shells with increasing radius and (x, x') samples from pairs of training data points. We calculate the estimated Lipschitz constant for each 0 and report both the average and maximum estimates, for each radius. The accurate Lipschitz constant (fp(f), technically, the maximum. However, the average is also meaningful in that it represents more typical practical behavior.
We conducted the following procedure. We first capture the maximum fluctuation, Rmux in parameters as in Table 6.
Dataset
Layer
R
CoLA CoLA
WA E (0,5, 0.174
Wa E 11,
0.189
Table 6: Rmux = Max
0 during 10 epochs of fine-tuning.
27

---

Algorithm Computation of Lavg and Lupper Vs. r
Input: Set 8 of data samples (x, x')
1: Initialize list max (indexed by 0)
2: Initialize lists Lovg. Lupper (indexed by r))
3: for to Amax do
4:
Generate a new set (r) of fyr models using
-00 where distortion(r)
v 10
|
5:
for all 0 Tr) do Initialize empty list Liss Set model params 0 for all (x, x') E S do
6:
7:
8:
Compute (xx) fx(x)]l ||x' x||
9:
and append to List
10:
end for
11:
Append max Liss) to Lmna end for
12:
13:
Append mean( Lmax) to Lave
14:
Append max [mux) Lupper
15:
end for
Output: Lavg and Lupper vs.
Then we used Rmax to calculate the Lipschitz continuity as in Algorithm We used |S| 1000 pairs of data points, Ny 100 models, i.e., 100 different Os, with Rmax 0.174 according to Table 6, 10 steps to vary distortion average, Travg and upper bound, Tupper: Lipschitz constants show a gradual and consistent increase as r grows, indicating that the model's sensitivity to perturbations increases mildly with distance from the original parameters 0g. On average, the model remains stable under small to moderate distortions.
step of the algorithm, and regularization value
Both the
Laeg
Lypper
0.000000
0.020724
0.020724
0.019333
0.020725
0.020764
0.038667
0.020726
0.020803
0.058000
0.020727
0.020843
0.077333
0.020727
0.020883
0.096667
0.020728
0.020923
0.116000
0.020729
0.020962
0.135333
0.020730
0.021002
0,154667
0.020731
0.021042
0.174000
0.020732
0.02 0.021081
Table 7: Lipschitz ratio of the model with selected layers {0,5, 11} and parameter type key (Wx) x) for 1000 pairs of data samples (x, x') with R 0.174.
Luvg
Lapper
0.000000
0.014479
0.014479
0.021000
0.014480
0.014505
0.042000
0.014482
0.014530
0.063000
0.014483
0.014556
0.084000
0.014484
0.014582
0,105000
0.014486
0.014608
0,126000
0.014487
0.014634
0,147000
0.014488
0.014660
0,168000
0.014490
0.014686
0.189000
0.014491
0.014712
Table Lipschitz ratio of the model with selected layers (0, 11] and parameter type key (We) for 100 pairs of data samples (x,x)) with Rmax 0.189.
28

---

L Layer Selection Algorithm
Building upon the foundation established by Theorem Corollary and Figure 3, we distill our proposed methodologies into the following PEFT layer selection strategy informed by our spectral perturbation bounds.
Algorithm 2 Trainable Parameter Selection via Spectral Perturbation
Input: Pretrained parameters 0; scalar o > 0; training samples X = [xI,..., Xn. J candidates
parameter subsets (Ø(1) ((2)); C = (1....,L) L}
Compute base NTK matrix K R=
2: for = L, L do
3:
Compute kernel contribution S₁ for candidate (())
4: end for
5: for each subset C E C do
6: Compute combined kernel SC
Elec St.
7: Compute spectral ratio
Amax K + S + +ol)
rc
Amax (K +ol
8: end for
9:
Select C + argmin Te-
Output: Selected parameters :
Cola
5553
Nasieaan Raprovane audta
00
1L.T
Coulilie
CoLA
(b) SST-2
((OLIA) Figure 3: Empirical risk ratio log (()0)
and maximum eigenvalue ratio log Ame (K+S+al) Amas (K+ol)
used
evaluate the impact of candidate layers. Here, 0 is fixed as the weights (W ) of layer {0}, while & represents the candidate layers. The horizontal axis represents the combination of layer {0} and different candidate layers.
29

---

M
Discussion and Implication of Assumptions
Stochastic vs. deterministic gradient descent:
Our theoretical derivations are based on deterministic gradient descent, while the AdamW optimizer, which is stochastic, is used in our experiments. Our theory designed make few key aspects precise: (1) that is possible to directly elicit NTK regime behavior through regularization and (2) that, once we operate in the NTK regime, the spectrum of the kernel determines training performance. Note that (2) does not rely any particular optimization scheme. As for (I), while it is true that the noisy gradient aspect of stochastic approaches is not part of the theory, gradient descent/flow captures the local-search approach of gradient-based methods, including SGD and its variants. Applying selective regularization (Eq. (8)) is as straightforward as applying gradient clipping with SGD and Adam. The bounds of Theorems 3 and transfer to stochastic variants through standard bounds that link their respective trajectories. (These typically need strong convexity assumptions the loss functions and bounds on the variance of the stochastic gradient [31].
Regularized vs. non-regularized fine-tuning is crucial to emphasize that our linearity analysis only applies to the explicitly regularized variant of fine-tuning that present. Even though we believe that this regularization may sometimes exist implicitly during fine-tuning, establishing bounds between non-regularized fine-tuning and linearized fine-tuning remains outside the scope of this paper and is an interesting avenue for future investigation.
Cross-entropy vs. mean squared loss: Our theoretical results are derived for the squared loss, while our experiments succeed with cross-entrope According to [31]. we provide intuitions on why our theoretical insights generalize well to different loss functions. Note that optimizing cross-entropy is equivalent optimizing KI Kl-diivegeence, and Kl--divergence and squared loss are intimately related, given that they're both Bregman divergences. For two outputs that are close to each other, i.e., in the high-accuracy regime, the KI-divergence behaves very similarly to the squared loss [32]. As such, while the theory doesn't apply directly cross-entropy, the behavior that we demonstrate qualitatively supports the behavior that we experimentally observe with cross-entropy.
Lipschitzness The Lipschitz continuity assumption in Theorems is not limiting. Almost every architecture and training methodology used in practice limits the complexity of the networks. Of particular relevance to our paper are the robust variants of low-rank adaptation, which explicitly enforce this kind of Lipschitz condition [33]. In Appendix K, we empirically estimate the Lipschitz constant for models balls around the initial model.
N Hyperparameters
Dataset
CoLA
SST-2
Yelp
IMDb
Optimizer
AdamW
Warmup Ratio
0.06
LR Schedule
Lincar
Max Sequence Length
512
LoRA Rank r
8
LoRA a
8
Number of Epochs
10
Batch Size
32
16
32
16
Learning Rate
4e-4
5e-4
4e-4
4e-4
Table 9: Hyperparameters used for the RoBERTa-base model on various benchmarks, including GLUE (CoLA, SST-2), Yelp, and IMDb.
30


Reviews:
[{"review_id": "Q3m7qQ6Ly1", "reviewer": "Reviewer_bqdr", "summary": "This paper proposes a theoretical analysis of fine-tuning through the lens of linearization and NKT theory. The authors observe that by adding explicit L2 regularization to the pretrained model parameters, fine-tuning dynamics become equivalent to learning with the NTK, and establish theoretical connections between the NTK's eigenvalue spectrum and fine-tuning performance. They validate the theory empirically on LoRA fine-tuning of RoBERTa across GLUE benchmarks.", "questions": "Your theoretical results are derived for squared loss and gradient descent, yet your experiments succeed with cross-entropy and AdamW.  Could you provide more intuition on why your insights generalize so well to stochastic settings and different loss functions? For example, does the proposed selective regularization scheme in Equation 8 transfer straightforwardly to AdamW? What about the bounds in Thms 3 and 4? \n\nThe NTK computation uses a small sketch of only 32 samples from the training data.  How stable is the condition number to the specific choice of these 32 samples? \n\nTable 1 shows that model accuracy is largely insensitive to the regularization strength lambda across a wide range of values. Could you explain why we expect this to be the case? Is your main takeaway here that one should always use a sufficiently large lambda that encourages linearity ?", "limitations": "Yes, the authors have adequately addressed the limitations of their work. In the conclusion, they explicitly state that their theory is limited to squared loss and gradient descent, while the experiments use other configurations. They are also clear that their analysis only applies to the explicitly regularized fine-tuning they introduce, and that analyzing standard, non-regularized fine-tuning remains an open question for future work.  This is a transparent and fair assessment of the work's scope", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\nThe paper provides a theoretical contribution connecting regularized fine-tuning and NTK. The spectral perturbation bounds for layer selection are particularly interesting and may provide guidance for PEFT design. Similarly, the connection between NTK condition numbers and fine-tuning performance may offer a practical prediction tool for model selection. The experiments on RoBERTa with LoRA across multiple datasets demonstrate reasonable correspondence between theory and practice, despite some theoretical assumptions differing from experimental conditions.\n\nWeaknesses\n\nA primary weakness is the disconnect between the theoretical assumptions and the experimental setup. The theorems are derived for (deterministic) gradient descent with a squared loss function, while the experiments use the AdamW optimizer with a cross-entropy loss. The authors acknowledge this limitation, but it makes it unclear how tightly the theoretical bounds apply to the practical scenarios shown.\nAlso, the linearization analysis only applies to the explicitly regularized variant of fine-tuning introduced by the authors, and the connection to standard fine-tuning practices where such regularization may exist only implicitly is not clearly established.\n\nThe NTK condition number calculations use only 32 samples, which may not capture full kernel behavior. The paper doesn't explore how sensitive the condition number and the resulting performance predictions are to the choice of these few samples.\n\nWhile the paper does a decent job of positioning itself, the distinction from previous work that also uses linearization and NTK to analyze fine-tuning (e.g., Malladi et al., https://arxiv.org/abs/2210.05643) could be sharpened. The paper's core novelty seems to be in enforcing linearity through regularization rather than just assuming it, but the practical implications of this difference could be explained more forcefully.\n\nThe experimental validation focuses on LoRA and RoBERTa-base; further investigation across different PEFT methods and model architectures would be beneficial.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal and discussion. The authors should implement many additions to the current version to reflect the rebuttal and discussion. I am convinced that the reasons to accept outweigh those to reject, despite some of the weak points identified here and by other reviewers.", "rebuttal": "Dear Reviewer bqdr,\n\nThank you very much for your time and effort reviewing our paper and for voicing a positive assessment. We appreciate it a lot. We hope that with what follows we address all your comments.\n\n---\n\n**Theory vs. experiments** Our theory is designed to make a few key aspects precise: (1) that it is possible to directly elicit NTK regime behavior through regularization and (2) that, once we operate in the NTK regime, the spectrum of the kernel determines training performance. Note that (2) does not rely on any particular optimization scheme. As for (1), while it is true that the noisy gradient aspect of stochastic approaches is not part of the theory, gradient descent/flow captures the local-search approach of gradient-based methods, including SGD and its variants. Applying selective regularization (Eq. (8)) is as straightforward as applying gradient clipping with SGD and Adam. The bounds of Theorem 3 and 4 transfer to stochastic variants through standard bounds that link their respective trajectories. (These typically need strong convexity assumptions on the loss functions and bounds on the variance of the stochastic gradient. [3]) Regarding the adherence to the squared loss in both (1) and (2), note that optimizing cross-entropy is equivalent to optimizing KL-divergence, and KL-divergence and squared-loss are intimately related, given that they're both Bregman divergences. (For two outputs that are close to each other, i.e., in the high-accuracy regime, KL-divergence behaves very similarly to squared loss. [4]) As such, while the theory doesn't apply directly to cross-entropy, the behavior that we demonstrate qualitatively supports the behavior that we experimentally observe with cross-entropy. \n\n---\n\n**Sensitivity of Performance to NTK Samples** Our  work is not specifically studying optimal sketching of the kernel matrix, however, below we empirically illustrate that our  numerical results are  robust to the choice of NTK samples.  Sketching of the kernel matrices is studied in the  literature, for instance in [2]. In [1], which also modeled  fine-tuning as an NTK regression problem the number of NTK samples are fixed to $\\{16,64\\}$ (see Table.2 of [1]).\n\n| **Random Seed** | $\\lambda_{\\text{min}}$ | $\\lambda_{\\text{max}}$ | **Condition Number** |\n|----------------|------------------------|------------------------|-----------------------|\n| 42             | 2.04e-6                | 0.0030                 | 32.43                |\n| 123            | 6.45e-7                | 0.0035                 | 36.85                |\n| 7              | 7.37e-8                | 0.0025                 | 26.22                |\n| 99             | 3.19e-9                | 0.0024                 | 25.82                |\n| 2024           | 6.78e-8                | 0.0025                 | 26.86                |\n\n*Table: Condition numbers and eigenvalues for different random seeds $\\mathbf{W}_k$ of layer 11 is fine-tuned.*\n\n\n**Regularization and accuracy** The key takeaways from Table 1 are that, as long as we don't overregularize ($\\\\lambda$ not too large, e.g., smaller than 50) we simultaneously: (1) encourage the model not to deviate (the direct objective of regularization), (2) encourage the NTK solution and the regularized fine-tuning solutions to become closer (the indirect **main** objective of regularization), and (3) without sacrificing much on performance. We conjecture that the reason we don't sacrifice much on performance is that the fine-tuning solution is already relatively close to the initial model and that there are equivalently good NTK solutions in the same neighborhood. (The choice of $\\\\lambda$ is similar to searching for solutions within sequentially smaller neighborhoods around the initialization, as $\\\\lambda$ increases.)\n\n---\n\n**Distinction from previous work** In [1] the authors assume that fine-tuning can be explained by linearization and their  results also confirm that linearization does not always appear in  fine-tuning (see fig.1 of [1]). In our work however, we prove linearization appears as result of proximity of the fine-tuned model and the pretrained model, and a regularization is  required to impose linearization. Additionally, we  study the layer selection performance by means of spectral perturbation bounds.   \n\n---\n\nThe experimental validation focuses on LoRA and RoBERTa-base; further investigation across different PEFT methods and model architectures would be beneficial.\n\n**Other model architectures** We fine-tuned decoder-only models GPT2 and OPT-125m for 10 epochs using the Adam optimizer. LoRA with $r=8$ is used to fine-tune $\\mathbf{W}_k$ of the layers $\\{0,5,11\\}$. The negative correlation between evaluation accuracy after 10 epochs of training and condition number of the NTK is illustrated below. In GPT2 Yelp with lowest condition number possesses the highest accuracy and in OPT-125M, IMDb and  Yelp with higher  accuracies have lower  condition  number than the other  tasks. \n\n\n\n| Dataset | Eval Accuracy (%) | Condition Number |\n|:--------|:------------------|:-----------------|\n| CoLA    | 71.0              | 83               |\n| IMDb    | 87.4              | 36               |\n| Yelp    | 87.6              | 35               |\n\n*Table: Evaluation results for GPT2.*\n\n| Dataset | Eval Accuracy (%) | Condition Number |\n|:--------|:------------------|:-----------------|\n| SST-2   | 64.8              | 910              |\n| CoLA    | 69.1              | 720              |\n| IMDb    | 70.0              | 210              |\n| Yelp    | 82.1              | 310              |\n\n*Table: Evaluation results for OPT-125m.*\n\n---\n\nThank you once again and we look forward to discussing further!\nBest,\nAuthors\n\n---\n\n[1] Malladi, S., Wettig, A., Yu, D., Chen, D. and Arora, S., 2023, July. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning (pp. 23610-23641). PMLR.\n\n[2] Chen, Y., Epperly, E.N., Tropp, J.A. and Webber, R.J., 2025. Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluations. Communications on Pure and Applied Mathematics, 78(5), pp.995-1041.\n\n[3] Bottou, L., Curtis, F.E. and Nocedal, J., 2018. Optimization methods for large-scale machine learning. SIAM review, 60(2), pp.223-311.\n\n[4] Borade, S. and Zheng, L., 2008, March. Euclidean information theory. In 2008 IEEE International Zurich Seminar on Communications (pp. 14-17)."}, {"review_id": "Y3lrGMGlOV", "reviewer": "Reviewer_LXhB", "summary": "The paper shows that if fine-tuning is explicitly regularized to stay near the pre-trained weights, the optimization behaves almost linearly and can be analyzed through the Neural Tangent Kernel (NTK). ,Under this linearization regime, the paper proves bounds on the empirical risk of fine-tuning as a funciton of the kernel’s eigen-spectrum. Crucially, the paper demonstrates, both theoretically and empirically, that the regularized condition number $\\kappa(K + \\sigma I)$ at initialization is a strong indicator for downstream performance, that could work as an effective heuristic for the selection of layers to tune.", "questions": "Would theorems 2-4 apply to SGD or Adam training?\n\nWhat new insight does the regularization-based proximity argument provide, beyond the intuitive fact that L2 regularization encourages proximity to the pretrained model?", "limitations": ".", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "While the paper provides and interesting theoretical framework and solid proofs, the theoretical analysis contains several assumptions or logical leaps that are not sufficiently justified in the paper.\n1. In section 3, the paper defines a proximal setup where the empricial risk is regularized with the L2-distance between the pretrained and fine-tuned parameters. The main argument of section 3 is that the distance between the pretrained model and the fine-tuned model is constrained by the regularization. However I see no significance of the result in this section for the following two reasons.\n(i) The bounds in theorem 2~4 don't have much of a quantative meaning, because it is not for actual SGD or reasonable algorithms, but it is for an aritificial ad-hoc algorithm that is defined soley for the convinience of the proof, with no practical usage reported in prior works or through experiments in the paper. Thus the bound only has it's meaning as in theoretical demonstration for intuition.\n(ii) However, the argument that adding the L2-distance of the the parameters of the pretrained model and the fine-tuned model constrains the difference between the two models is already intuitively straightforward, and thus a theoretical demonstration in a toy setup isn't really significant.\n2. In sections 4 and 5, the paper implicitly equates 'performance' and the 'empirical risk'. Especially, in theorem 5, I think there should be more explanation about why the model performance should get better when the condition number is small.\nRegarding the empirical results, it is indeed quite interesting that the generalization performances are predicted through the actual condition numbers of the NTK kernel, but there could have been much more the authors could have done to make a more robust and persuasive experiment. For example, for figure 2-(c), instead of comparing between different datasets, sampling 32 datasamples from the same dataset multiple times and comparing between these minidatasets could have been a more fair comparison. Also, the choice of the layers {0, 5, 11} seem quite arbitrary, it would be better to see an experiment spanning all layers of the RoBERTa-base model.\n\n\nOverall, the results of the paper are interesting as it connects theoretical insight into empirical results on generalization performance and an actual methodology with practical usage, but lacks theoretical solidness and extensiveness of its experiments. \n\n\nTypos\nline 22: 'particularly popular methods [4, 5, 6, 7, 8] set of techniques that strive to'\nline 214: 'The spectral perturbations bounds'\nline 222: 'See Apendix H'\nline 322: 'close to the linearization of the the pre-trained model.'\nline 731: notations and definitons\nline 804: 'inquality 79 will be reduced to'", "was_revised": true, "final_justification": "This work provides a novel kernel spectrum perspective to parameter-efficient fine-tuning. The results and insights are valuable and distinct from those offered by prior work. Therefore, this work should be published.", "rebuttal": "Dear Reviewer LXhB,\n\nThank you for taking the time to review our paper. We would like to address your comments below and hope that you will consider revising your assessment upward.\n\n---\n\n**Significance of Section 3** (i) Regarding the applicability to SGD,  while it is true that the noisy gradient aspect of stochastic approaches is not part of the theory, gradient descent/flow captures the local-search approach of gradient-based methods, including SGD and its variants. Applying selective regularization (Eq. (8)) is as straightforward as applying gradient clipping with SGD and Adam.The bounds of Theorem 3 and 4 transfer to stochastic variants through standard bounds that link their respective trajectories. (These typically need strong convexity assumptions on the loss functions and bounds on the variance of the stochastic gradient. [3]) (ii) Regarding these latter bounds being \"intuitively straightforward\", we would like to correct a misunderstanding. The *direct* objective of regularization is to encourage the model not to deviate but the *indirect* benefit we get is that it encourage the NTK solution and the regularized fine-tuning solutions to become closer (to answer your question, *this* is the additional insight and it is *not* straightforward.) This is what allows us to claim that the results on Section 4 apply and its direct validity is demonstrated clearly in Table 1, where we also see an additional advantage that, as long as we don't overregularize ($\\\\lambda$ not too large, e.g., smaller than 50), we don't sacrifice much on performance either. This advantage makes this very simple regularization very attractive for practical use (easy to implement + no sacrifice + NTK theory holds.) Note that this regularization has been proposed in an entirely different setting in reference [16] in the paper, so it is not \"artificial\" or \"ad-hoc\". We have demonstrated a fully novel way in which it can be used.\n\n---\n\n**Condition Number vs. Performance** Theorem 5, taken as it is, does not say that the condition number dictates performance. It does say that the condition number determines the tightness of the bounds (how close the upper and lower bounds are). However, if we assume that the spectrum is mostly stable (say the arithmetic or geometric midpoint of $\\\\lambda_{\\\\textsf{min}}$ and $\\\\lambda_{\\\\textsf{max}}$ remains roughly constant), then the changes in condition number would indeed dictate the magnitude of the bounds.\n\n\n---\n\n**Robustness to Sampling for estimating the NTK kernel** Our  work is not specifically studying optimal sketching of the kernel matrix, however, below we empirically illustrate that our  numerical results are  robust to the choice of NTK samples.  Sketching of the kernel matrices is studied in the  literature, for instance in [2]. In [1], which also modeled  fine-tuning as an NTK regression problem the number of NTK samples are fixed to $\\{16,64\\}$ (see Table.2 of [1]).\n\n| **Random Seed** | $\\lambda_{\\text{min}}$ | $\\lambda_{\\text{max}}$ | **Condition Number** |\n|----------------|------------------------|------------------------|-----------------------|\n| 42             | 2.04e-6                | 0.0030                 | 32.43                |\n| 123            | 6.45e-7                | 0.0035                 | 36.85                |\n| 7              | 7.37e-8                | 0.0025                 | 26.22                |\n| 99             | 3.19e-9                | 0.0024                 | 25.82                |\n| 2024           | 6.78e-8                | 0.0025                 | 26.86                |\n\n*Table: Condition numbers and eigenvalues for different random seeds $\\mathbf{W}_k$ of layer 11 is fine-tuned.*\n\n---\n\n**Extensiveness of experiments** We fine-tuned decoder-only models GPT2 and OPT-125m for 10 epochs using the Adam optimizer. LoRA with $r=8$ is used to fine-tune $\\mathbf{W}_k$ of the layers $\\{0,5,11\\}$. The negative correlation between evaluation accuracy after 10 epochs of training and condition number of the NTK is illustrated below. In GPT2 Yelp with lowest condition number possesses the highest accuracy and in OPT-125M, IMDb and  Yelp with higher  accuracies have lower  condition  number than the other  tasks. \n\nThe only exception is that IMDB in OPT-125M, despite having the smaller conditional \n\n| Dataset | Eval Accuracy (%) | Condition Number |\n|:--------|:------------------|:-----------------|\n| CoLA    | 71.0              | 83               |\n| IMDb    | 87.4              | 36               |\n| Yelp    | 87.6              | 35               |\n\n*Table: Evaluation results for GPT2.*\n\n| Dataset | Eval Accuracy (%) | Condition Number |\n|:--------|:------------------|:-----------------|\n| SST-2   | 64.8              | 910              |\n| CoLA    | 69.1              | 720              |\n| IMDb    | 70.0              | 210              |\n| Yelp    | 82.1              | 310              |\n\n*Table: Evaluation results for OPT-125m.*\n\n\n\nThank you once again. We hope you will consider raising your score and we look forward to discussing further!\nBest,\nAuthors\n\n---\n\n[1] Malladi, S., Wettig, A., Yu, D., Chen, D. and Arora, S., 2023, July. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning (pp. 23610-23641). PMLR.\n\n[2] Chen, Y., Epperly, E.N., Tropp, J.A. and Webber, R.J., 2025. Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluations. Communications on Pure and Applied Mathematics, 78(5), pp.995-1041.\n\n[3] Bottou, L., Curtis, F.E. and Nocedal, J., 2018. Optimization methods for large-scale machine learning. SIAM review, 60(2), pp.223-311."}, {"review_id": "8B8zjYo5aP", "reviewer": "Reviewer_Ptpw", "summary": "This paper presents a theoretical framework for understanding Parameter-Efficient Fine-Tuning (PEFT) in large language models (LLMs) through linearization and NTK theory. The authors introduce explicit $\\(l_2\\)$-distance regularization toward pretrained parameters, showing that this induces \"lazy training\" dynamics equivalent to NTK regression. Key contributions include: (1) Theoretical bounds on the distance between fine-tuned and linearized models, (2) Characterization of fine-tuning performance via the NTK eigenvalue spectrum, and (3) Spectral perturbation bounds for layer-wise adaptation.", "questions": "1. **Loss Function Generalization**: Your theory assumes MSE, but experiments use cross-entropy. Can you discuss whether the NTK-performance correlation holds *theoretically* for cross-entropy? If not, what adjustments are needed?  \n2. **Methodological Extensions**: Could your spectral perturbation bounds (Thm 7) directly inform a *layer selection algorithm* (e.g., prioritizing layers that minimize $\\lambda_{\\text{max}}(\\mathbf{K} + \\mathbf{S})$? Have you tested such an approach?  Moreover, some more novel methodological innovations are preferred. (See Weakness 2)\n3. **Regularization Discrepancy**: In Table 1, higher $\\lambda$ reduces $\\|\\theta_t - \\theta_0\\|$ but inconsistently affects accuracy (e.g., CoLA peaks at $\\lambda=5$). How do you reconcile this with the claim that accuracy is \"unaffected\"?  \n4. **Condition Number Correlation**: Table 2 shows CoLA’s best accuracy (73.44%) occurs at $\\kappa = 7,503$ , which is not the smallest one. What explains cases where low $\\kappa$ *does not* yield the best performance? In fact, it seems that the performance is more likely to be *unaffected* by condition numbers, a more precise explanation is favored.\n5. Typo: Line 167, *infinitesimally*", "limitations": "See Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. **Theoretical Rigor**: The paper rigorously formalizes PEFT dynamics through NTK linearization, deriving novel bounds for model deviation (Theorems 2–4) and risk minimization (Theorem 5). The spectral perturbation analysis (Theorems 6–7) provides actionable insights for layer selection.  \n2. **Practical Utility**: Demonstrates that NTK condition number (computable beforehand) correlates with downstream task performance, enabling efficient layer selection. Table 3 highlights significant time savings.  \n3. **Innovative Regularization**: The selective $\\(l_2\\)$-regularization scheme, activated based on gradient alignment, is a nuanced methodological contribution that ensures monotonic loss decrease.  \n4. **Strong Validation**: Experiments across GLUE, IMDb, and Yelp tasks consistently support theoretical claims, particularly the NTK-performance correlation and regularization efficacy.\n\nWeaknesses:\n1. **Loss Function Gap**: Theory relies on **MSE loss**, while experiments use cross-entropy. The generalization of NTK spectral analysis to classification losses remains unproven.  \n2. **Methodological Novelty**: While offering theoretical insights, the work does **not propose a new algorithm**. The selective regularization is underexplored as a standalone method. Moreover, a linear fine-tune strategy in [11](author's citation number) seems to be further justified by this work, which should be used as a analyzed baseline, or even should be referred to design a new tuning method.\n3. **Inconsistent Empirical Claims**:  \n   - The claim that \"*accuracy remains largely unaffected by regularization*\" (Sec 6.2) contradicts Table 1 (SST-2 accuracy varies by **3.7%** with $\\lambda$, variations are even larger on CoLA).  \n   - Table 2 shows weak correlation between $\\kappa(\\mathbf{K} + \\sigma\\mathbf{I})$ and accuracy.  \n4. **Limited Scope**: Theoretical bounds assume Lipschitz continuity and small parameter deviations, which may not hold for aggressive fine-tuning.", "was_revised": true, "final_justification": "All of my concerns have been addressed. I believe this is a work of broad interest for researchers concerning PEFT and NTK analysis on LMs, thus warranting an acceptance. The authors should supplement their rebuttal explanations in their final version of the manuscript.", "rebuttal": "Dear Reviewer Ptpw,\n\nThank you for taking the time to review our paper. We would like to address your comments below and hope that you will consider revising your assessment upward.\n\n---\n\n**Loss Function** (Q1) Regarding the adherence to the squared loss in both (1) and (2), note that optimizing cross-entropy is equivalent to optimizing KL-divergence, and KL-divergence and squared-loss are intimately related, given that they're both Bregman divergences. (For two outputs that are close to each other, i.e., in the high-accuracy regime, KL-divergence behaves very similarly to squared loss. [1]) As such, while the theory doesn't apply directly to cross-entropy, the behavior that we demonstrate qualitatively supports the behavior that we experimentally observe with cross-entropy. \n\n---\n\n**Methodological Novelty** (Q2) While we didn't explicitly present it algorithmically, we can use Theorem 7, Corollary 1 and Figure 3 to distil our suggested methodologies into the following PEFT layer selection strategy, using our spectral perturbation bounds:\n\n> ### Algorithm: Trainable Parameter Selection via Spectral Kernel Perturbation\n>\n>**Input:**\n>- Pretrained model parameters $ \\\\boldsymbol{\\\\theta} $\n>- $ \\\\sigma > 0 $\n>- Training samples $ \\mathbf{X} = [x_1, \\\\dots, x_n]^{\\\\top} $\n>- Candidate parameter subsets $\\\\{ \\hat{\\\\boldsymbol{\\\\theta}}^{(1)}, \\\\dots, \\\\hat{\\\\boldsymbol{\\\\theta}}^{(L)} \\\\} $ and $\\mathcal{C}=\\\\{1,\\\\ldots,L\\\\}$\n>\n>**Step 1:** Compute base NTK matrix $ \\\\mathbf{K} \\\\in \\\\mathbb{R}^{n \\\\times n} $\n>\n>**Step 2:** For each candidate $ \\\\hat{\\\\boldsymbol{\\\\theta}}^{(l)}$, compute kernel contribution $ \\\\mathbf{S}_l $\n>\n>**Step 3:** For each subset $ C \\\\in \\\\mathcal{C}$:\n>- Compute combined kernel:  $\\\\mathbf{S}^C = \\\\sum_{l \\in C} \\\\mathbf{S}_l $\n>- Compute spectral ratio:   $r_c = \\frac{\\\\lambda_{\\\\max}(\\\\mathbf{K} + \\\\mathbf{S}^C + \\\\sigma >\\mathbf{I})}{\\\\lambda_{\\\\max}(\\\\mathbf{K} + \\\\sigma \\\\mathbf{I})}$ \n>- Select:  $C^* = \\\\underset{C}{\\\\arg\\\\min} \\\\; r_c $\n>\n>**Output:** Selected parameters  $\\\\{\\\\!\\\\{ \\\\hat{\\boldsymbol{\\\\theta}}^{(l)} : l \\\\in C^*\\\\}\\\\!\\\\} $\n\n\nWe believe this is the key novelty in our paper. We will also investigate using [11](paper reference) as baseline. Thank you for the suggestion.\n\n---\n\n**Regularization vs. Accuracy** (Q3) We would like to correct a misunderstanding regarding Table 1. Accuracies are negatively affected *only* when we overregularize ($\\\\lambda$ not too large, e.g., smaller than 50). Apart from this, our claim that accuracy is unaffected is true (the fluctuations are within the error margin for $\\\\lambda$ up to 10. The takeaway from this table is that whereas the *direct* objective of regularization is to encourage the model not to deviate, the *indirect* benefit we get is that it encourages the NTK solution and the regularized fine-tuning solutions to become closer. This is what makes the results on Section 4 apply. The trifecta of easy implementation + no performance sacrifice + enabling the NTK theory hold makes our proposed regularization an attractive option.\n\n---\n\n**Is Lipschitz continuity limiting?** The general answer is no. Almost every architecture and training methodology used in practice limits the complexity of the networks. Of particular relevance to our paper are the robust variants of low-rank adaptation, which explicitly enforce this kind of Lipschitz conditions. [2]\n\n---\n\n**Condition Number Correlation** (Q4) It is true that at first glance, in Table 2, the highest condition number does not correspond to the lowest accuracy. However, one may notice that this is an issue primarily when the nature and number of parameters change (query/value vs. key). When this is the case, the effective complexity of the fine-tuned model changes, and since the NTK parameter $\\\\sigma$ is inversely proportional to the complexity, it affects the *regularized* condition number. In the table, we have fixed $\\\\sigma$ and so this change is not being reflected and it is only fair to compare instead cases where the nature and number of parameters is the same, e.g. fixing a layer and varying tasks among CoLA, SST2, IMDB, YeLP, or (the correlation is perfect for $\\\\{0,11\\\\}$, but also mostly tracks for other fixed choices), or varying the same number of layers like  $\\\\{0,7\\\\}$, $\\\\{0,11\\\\}$ for CoLA and SST2 in Figure 3. It would be very interesting to factor the change of $\\\\sigma$$ into the layer selection methodology, but that is a great direction of future research.\n\n---\n\nThank you once again. We hope you will consider raising your score and we look forward to discussing further!\nBest,\nAuthors\n\n---\n\n[1] Borade, S. and Zheng, L., 2008, March. Euclidean information theory. In 2008 IEEE International Zurich Seminar on Communications (pp. 14-17).\n\n[2] Savostianova, D., Zangrando, E., Ceruti, G. and Tudisco, F., 2023. Robust low-rank training via approximate orthonormal constraints. Advances in Neural Information Processing Systems, 36, pp.66064-66083."}, {"review_id": "GVGQmrADvs", "reviewer": "Reviewer_xfCG", "summary": "The paper studies fine-tuning through linearization and NTK regression. The paper shows under an explicit regularized fine-tuning process, the behavior of model dynamics is close to a linearized model, which leads to the approximation by NTK. They then analyze how the eigenvalue spectrum of the NTK is related to the model performance and validated through empirical experiments.", "questions": "1. It is not clear what is the implication of Theorem 2. Comparing the bound for regularized fine-tuning (9) and conventional fine-tuning (10), the bound (9) could be larger than (10) given the exponential dependence on $t$ versus linear dependence on $t$. In this case, how such regularization $\\lambda$ would manifest lazy training? \n\n2. In Theorem 4, what is $N$? And it seems by setting $t = O(1/\\lambda \\log(N/\\text{Lip}^2))$ and requiring $\\lambda \\geq O(2N/(r \\text{Lip}))$ would bring $t = O(r\\text{Lip} \\log(N/\\text{Lip}^2 )/N)$. Could you comment on whether such requirement on $t$ is restrictive or not? \n\n3. In Theorem 4, can you comment on how large $\\lambda$ should be in order for the fine-tuning to be approximated by NTK regression?\n\n4. In Theorem 5, what is $\\mathcal{R}$? It seems only $\\widetilde{\\mathcal{R}}$ is defined in (3). I think $\\mathcal{R}$ is defined in terms of (14), is it correct?\n\n5. I do not quite understand how Table 2 correlates to your propositions. Section 6.3 does not have sufficient discussions of the results in Table 2. I do not understand why \"lower condition number indicates lower training and evaluation loss\". In your Theorem 5, the condition number seems only to suggest the range of your empirical risk.\n\n6. It seems the main benefit of the propositions is to determine what layers need to fine-tuned. However, I did not see much discussion in this regard, namely how can we determine the layer to be fine-tuned in practice. It is better to write explicitly an algorithm for such purpose.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper provides some insights on how fine-tuning is related to NTK and how the eigenvalue spectrum of NTK predicts the model performance. Such an insight could be useful for determining what layers to be fine-tuned by evaluating the corresponding NTK spectrum. However, analyzing fine-tuning dynamics with NTK is not new and has been studied in e.g., [1]. And it is unclear what is the main procedure for fine-tuning with layer selection via NTK. Also see weaknesses in the questions section. \n\n\n[1] LoRA training in the NTK regime has no spurious local minima. ICML 2024.", "was_revised": true, "final_justification": "My main concerns on the theoretical results are addressed. More discussions are expected in the revised version.", "rebuttal": "Dear Reviewer xfCG,\n\nThank you for taking the time to review our paper and for the incisive yet constructive remarks. We appreciate it a lot. We would like to address your comments below and hope that you will consider revising your assessment upward.\n\n---\n\n**Distinction from [1]** The work in [1] explores the loss landscape in the NTK regime; however, it does not determine under what conditions fine-tuning falls within this regime. In contrast, our work proves—both theoretically and empirically—that although fine-tuning is not necessarily in the NTK (linearization) regime, it can be constrained to this regime by regularizing the fine-tuning process towards the pretrained model. Additionally, we provide a spectral perturbation bound for layer selection, which, to the best of our knowledge, is the first of its kind. Therefore, our work is distinct from [1].\n\n---\n\n**Role of Theorem 2** (Q1) This theorem is primarily a building block for Theorem 3, it simply says that while the solution may deviate from the origin under regularization, we can bound that deviation. As you point out, this may not be the tightest way to bound the deviation, but it is still useful. We could relegate it to a lemma. Theorem 3 and 4 show that we can jump from a deviation from initialization bound (which is the direct  objective of regularization) to a bound on how far the NTK solution and the regularized fine-tuning solutions are  (the indirect but **main** benefit of regularization). This is what allows us to place the approximation of Section 4 on a stronger footing than prior works that assume that such proximity simply holds.\n\n---\n\n**Theorem 4** (Q2) The analysis Theorem 3 and 4 is based on gradient flow, which is a continuous process  (see Theorem 2.3 in [2]). $t$ represents the continuous time in the gradient flow model. What $N$ represents is a discretization relationship between the continuous time gradient-flow analysis and gradient descent, and is analogous to the number of iterations. We will define $N$ more explicitly in the statement of the Theorem. (To be more technical, one step of gradient descent approximates the gradient flow over a small time interval. The size of this time interval (the \"step-size\") is related to the properties of the function being optimized, specifically $1/\\\\text{Lip}(h)^2$ and regularization parameter  $\\\\lambda$. Moreover the choice of $t=\\\\frac{1}{\\\\lambda} \\\\log(1+\\\\frac{N}{\\\\text{Lip}(f)^2})$ results $|\\\\theta_t-\\\\theta_0|<r$ and ensures  Lipschitz continuity  in equation (79).)\n\n(Q3) If we sidestep $N$, we can express $t$ directly as a function of $\\\\lambda$. The result is an inverse dependence, up to the logarithmic term. The takeaway is that the regularization determines the time scale at which the approximation holds, as well as how tight the approximation is. Regarding the exact choice of $\\\\lambda$, Theorem 4, at first glance, seems to suggest that the bigger the better to bring NTK and regularized trajectories together. However, larger $\\\\lambda$ means that we effectively force ourselves to remain in a small ball around $\\\\theta_0$ and may suffer in terms of fine-tuning performance (see $\\\\lambda=50$ in Table 1). The \"sweet spot\" tradeoff can be read from Theorem 4 by noting that the right-hand is simply $t$ times the middle parenthesis factor $\\\\frac{2}{\\\\lambda} \\\\textrm{Lip}(\\\\nabla f) \\\\Vert f_{\\\\theta_0}(\\\\mathbf{x}^\\\\star)-\\\\mathbf{y^\\\\star}\\\\Vert+1$. Therefore the parenthesis term is the one that governs the accuracy. Since there is a $+1$, there is no point optimizing $\\\\lambda$ beyond $\\\\textrm{Lip}(\\\\nabla f)$ times the largest error of the base model.\n\n---\n\n**Empirical risk** (Q4) We apologize for this notational mismatch. It is true that in this context we are using (14), but the definition of empirical risk is the same (does not include the Hilbert norm). We will unify the notation across all sections:\n\n $​​\\\\mathcal{R}= \\\\frac{1}{n} \\\\sum_{i=1}^{n} \\\\mathcal{L}(f_{\\\\mathbf{\\\\theta}}(\\\\mathbf{x}_i), \\\\mathbf{x}_i)$\n\n---\n\n**Condition Number vs. Performance** (Q5) It is true that Theorem 5, taken as it is, does not say that the condition number dictates performance. It does say that the condition number determines the tightness of the bounds (how close the upper and lower bounds are). However, if we assume that the spectrum is mostly stable (say the arithmetic or geometric midpoint of $\\\\lambda_{\\\\textsf{min}}$ and $\\\\lambda_{\\\\textsf{max}}$ remains roughly constant), then the changes in condition number would indeed dictate the magnitude of the bounds.\n\nIt is also true that at first glance, in Table 2, the highest condition number does not correspond to the lowest accuracy. However, one may notice that this is an issue primarily when the nature and number of parameters change (query/value vs. key). When this is the case, the effective complexity of the fine-tuned model changes, and since the NTK parameter $\\\\sigma$ is inversely proportional to the complexity, it affects the *regularized* condition number. In the table, we have fixed $\\\\sigma$ and so this change is not being reflected and it is only fair to compare instead cases where the nature and number of parameters is the same, e.g. fixing a layer and varying tasks among CoLA, SST2, IMDB, YeLP, or (the correlation is perfect for $\\\\{0,11\\\\}$, but also mostly tracks for other fixed choices), or varying the same number of layers like  $\\\\{0,7\\\\}$, $\\\\{0,11\\\\}$ for CoLA and SST2 in Figure 3. It would be very interesting to factor the change of $\\\\sigma$$ into the layer selection methodology, but that is a great direction of future research.\n\n---\n\n**Main procedure for fine-tuning with layer selection via NTK** (Q6) While we didn't explicitly present it algorithmically, we can use Theorem 7, Corollary 1 and Figure 3 to distil our suggested methodologies into the following PEFT layer selection strategy, using our spectral perturbation bounds:\n\n> ### Algorithm: Trainable Parameter Selection via Spectral Kernel Perturbation\n>\n>**Input:**\n>- Pretrained model parameters $ \\\\boldsymbol{\\\\theta} $\n>- $ \\\\sigma > 0 $\n>- Training samples $ \\mathbf{X} = [x_1, \\\\dots, x_n]^{\\\\top} $\n>- Candidate parameter subsets $ \\{ \\hat{\\\\boldsymbol{\\\\theta}}^{(1)}, \\\\dots, \\\\hat{\\\\boldsymbol{\\\\theta}}^{(L)} \\} $  and $\\mathcal{C}=\\\\{1,\\\\ldots,L\\\\}$\n>\n>**Step 1:** Compute base NTK matrix $ \\\\mathbf{K} \\\\in \\\\mathbb{R}^{n \\\\times n} $\n>\n>**Step 2:** For each candidate $ \\\\hat{\\\\boldsymbol{\\\\theta}}^{(l)}$, compute kernel contribution $ \\\\mathbf{S}_l $\n>\n>**Step 3:** For each subset $ C \\\\in \\\\mathcal{C}$:\n>- Compute combined kernel: $ \\mathbf{S}^C = \\\\sum_{l \\in C} \\\\mathbf{S}_l $\n>- Compute spectral ratio:  $r_c = \\frac{\\\\lambda_{\\\\max}(\\\\mathbf{K} + \\\\mathbf{S}^C + \\\\sigma >\\mathbf{I})}{\\\\lambda_{\\\\max}(\\\\mathbf{K} + \\\\sigma \\\\mathbf{I})}$ \n>- Select:  $C^* = \\\\underset{C}{\\\\arg\\\\min} \\\\; r_c $\n>\n> **Output:** Selected parameters  $\\\\{\\\\!\\\\{ \\\\hat{\\boldsymbol{\\\\theta}}^{(l)} : l \\\\in C^*\\\\}\\\\!\\\\} $\n\n---\n\nThank you once again. We hope you will consider raising your score and we look forward to discussing further!\nBest,\nAuthors\n\n---\n\n[1] Jang, U., Lee, J.D. and Ryu, E.K., 2024, July. LoRA training in the NTK regime has no spurious local minima. In Proceedings of the 41st International Conference on Machine Learning (pp. 21306-21328).\n\n[2] Chizat, L., Oyallon, E. and Bach, F., 2019. On lazy training in differentiable programming. Advances in neural information processing systems, 32."}]

Author rebuttal:

Meta-review: (a) Summary

This paper proposes a theoretical analysis of Parameter-Efficient Fine-Tuning (PEFT) in large language models (LLMs) through the lens of linearization and NKT theory. The authors observe that by adding explicit L2 regularization to the pretrained model parameters, fine-tuning dynamics become equivalent to learning with the NTK, and establish theoretical connections between the NTK's eigenvalue spectrum and fine-tuning performance.  Key contributions include: (1) Theoretical bounds on the distance between fine-tuned and linearized models, (2) Characterization of fine-tuning performance via the NTK eigenvalue spectrum, and (3) Spectral perturbation bounds for layer-wise adaptation. Crucially, the paper demonstrates, both theoretically and empirically, that the regularized condition number  at initialization is a strong indicator for downstream performance, that could work as an effective heuristic for the selection of layers to tune. The paper validates the theory empirically on LoRA fine-tuning of RoBERTa across GLUE benchmarks.

(b) Strengths

The paper provides a theoretical contribution connecting regularized fine-tuning and NTK. The spectral perturbation bounds for layer selection are particularly interesting and may provide guidance for PEFT design. Similarly, the connection between NTK condition numbers and fine-tuning performance may offer a practical prediction tool for model selection. The paper rigorously formalizes PEFT dynamics through NTK linearization, deriving novel bounds for model deviation (Theorems 2–4) and risk minimization (Theorem 5). The spectral perturbation analysis (Theorems 6–7) provides actionable insights for layer selection. The experiments on RoBERTa with LoRA across multiple datasets (GLUE, IMDb, and Yelp tasks) demonstrate reasonable correspondence between theory and practice, despite some theoretical assumptions differing from experimental conditions. The results and insights are valuable and distinct from those offered by prior work. 

(c) Weaknesses 

Analyzing fine-tuning dynamics with NTK is not new and has been studied in e.g., [1]. And it is unclear what is the main procedure for fine-tuning with layer selection via NTK. 
[1] LoRA training in the NTK regime has no spurious local minima. ICML 2024.

A primary weakness is the disconnect between the theoretical assumptions and the experimental setup. The theorems are derived for (deterministic) gradient descent with a squared loss function, while the experiments use the AdamW optimizer with a cross-entropy loss. The authors acknowledge this limitation, but it makes it unclear how tightly the theoretical bounds apply to the practical scenarios shown. Also, the linearization analysis only applies to the explicitly regularized variant of fine-tuning introduced by the authors, and the connection to standard fine-tuning practices where such regularization may exist only implicitly is not clearly established.

The NTK condition number calculations use only 32 samples, which may not capture full kernel behavior. The paper doesn't explore how sensitive the condition number and the resulting performance predictions are to the choice of these few samples.

While the paper does a decent job of positioning itself, the distinction from previous work that also uses linearization and NTK to analyze fine-tuning (e.g., Malladi et al., https://arxiv.org/abs/2210.05643) could be sharpened. The paper's core novelty seems to be in enforcing linearity through regularization rather than just assuming it, but the practical implications of this difference could be explained more forcefully.

(d) Most important reasons for your decision to accept/reject. 

Theoretical rigor, with novel insights, combined with effective validation on a wide range of datasets.

(e) Summarize the discussion and changes during the rebuttal period.

Reviewer Ptpw considered that the rebuttal answered all of their concerns, and gave a high final rating of 5.
Reviewer xfCG stated that their main concerns on the theoretical results were addressed by the rebuttal, but that more (explanatory) discussions are expected in the final version.
Decision: Accept (poster)