A Kernel-Based View of Language Model Fine-Tuning
Authors: Sadhika Malladi
Alexander Wettig
Dingli Yu
Danqi Chen
Sanjeev Arora
Venue: Submitted to ICLR 2023 (2023)
Keywords: language models
fine-tuning
neural tangent kernels
tensor programs

Abstract: It has become standard to solve NLP tasks by  fine-tuning pre-trained language models (LMs), especially in low-data settings. There is minimal theoretical understanding of empirical success, e.g., why fine-tuning a model with $10^8$ or more parameters on a couple dozen training points does not result in overfitting. We investigate whether the Neural Tangent Kernel (NTK)--which originated as a model to study the gradient descent dynamics of infinitely wide networks with suitable random initialization--describes fine-tuning of pre-trained LMs.  This study was inspired by the decent performance of NTK  for computer vision tasks (Wei et al., 2022). We also extend the NTK formalism to  fine-tuning with Adam.  We present extensive experiments  that suggest that once the task is formulated as a masked language modeling problem through prompting, the NTK lens can often reasonably describe the model updates during fine-tuning with both SGD and Adam.
This kernel view also suggests an explanation for success of parameter-efficient subspace-based fine-tuning methods. Finally, we suggest a path toward a formal explanation for our findings via Tensor Programs (Yang, 2020).

Under review as a conference paper at ICLR 2023
A KERNEL-BASED VIEW OF LANGUAGE MODEL FINE- TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
It has become standard to solve NLP tasks by fine-tuning pre-trained language models (LMs), especially in low-data settings. There minimal theoretical un- derstanding of empirical success, e.g.. why fine-tuning model with 10 or more parameters on a couple dozen training points does not result overfitting. We in- vestigate whether the Neural Tangent Kernel (aTK)--hick originated to study the gradient descent dynamics of infinitely wide networks with suitable random initialization destribbs fine-tuning of pre-trained LMs. This study was inspired by the decent performance of NTK for computer vision tasks (Wei et al. 2022). We also extend the NTK formalism to fine-tuning with Adam. We present extensive experiments that show that once the downstream task is formulated as a language modeling problem through prompting, the NTK lens can often rea- sonably describe the model updates during fine-tuning with both SGD) and Adam. This kernel view also suggests an explanation for success of parameter-efficeen subspace-based fine-tuning methods. Finally, we suggest a path toward formal explanation for our findings via Tensor Programs (Yagg 2020b).
model
INTRODUCTION
is now customary to solve most supervised natural language processing (NLP) tasks such as topic classification and textual entailment by fine-tuning a pre-trained language model (e.g.. (Devlin et al.. 2019) Liu et al. 2020b] Clark et al. 22000 Raffel et al. [2020] Joshi et al. 2020)). We lack theoretical understanding of this fine-tuning paradigm. Why do we not see over-fitting when fine-tuning very large language model using couple dozen instances of the supervised task? Why is fine-tuning so sensitive to details such as whether or not we include prompt (e.g., adding "It was [great/terrible]"
for sentiment analysis (Schick & Schütze 2021 Gao et al. [202T) Why does restricting optimization to a low-rank subspace model parameters (Hu et al. 2021 Li et al. 2018: Aghhjanyynn et al.) 2021) still result in performance comparable to full tunint? Answering such questions requires understanding how the sequence of parameter updates changes in various scenarios, e.g., the addition of a prompt, the introduction of randomly initialized parameters. The current theory of deep learning, at first sight, seems too primitive to address such questions, especially since fine-tuning has to start from a parameter initialization inherited from pre-training.
Recently, Wei et al. (2022) suggested replacing fine-tuning with Neural Tangent Kernel (NTK), an idea invented for study of infinite-width deep neural networks (Jacot al. 2018 Du al. 2019a) and previously applied to solving vision tasks with infinitely wide ConvNets (Arora et al. (2019b))
They note that NTK can be defined for any neural model f and any initialization Bg by representing an input by the gradient it induces V f(: 9o). which yields a kernel matrix:
K((. = (V (f:((E:00) 0o). f (o)).
(1)
This kernel is well-defined for any parameter vector lg- However, for an infinite-width network initialized with 6n sampled from a suitably-scaled Gaussians, it can be shown that the kernel matrix is unchanged during gradient descent, which turns the classification task into a form of kernel regression with respect to this kernel (Jacot et al. 2018). In the fine-tuning setting, however, the initialization dg is inherited from the pre-trained network, and not sampled from the Gaussian distribution. Nevertheless, Wei et al. (2022) found that kernel regression using this "empirical NTK" (eNTK) defined with the inherited On performs well, achieving classification accuracy within 6%

---

Under review as a conference paper at ICLR 2023
absolute of actual fine tuning several image recognition tasks. In other words, their work hints that mathematical understanding of the fine-tuning phenomenon (e.g., its sample efficiency) could go via the theory of kernel classifiers.
The current paper furthers an empirical and theoretical understanding of the pre- pre-training (PT) and fine-tuning (FT) paradigm for NLP tasks. Our contributions are:
1. We formally extend the standard NTK theory developed for gradient descent to character- ize kernel-based dynamics when training with Adam (Section 4.. We propose and rigorously prove the correctness of a new kernel formula relying on the sign of the gradient to describe early-stage training (e.g., fine-tuning) with Adam.
2. We perform extensive empirical analysis on 12 diverse NLP tasks to reveal when and to what extent fine-tuning exhibits kernel behavior (Section (5). We find that using a prompt is crucial for the eNTK achieve good performance, suggesting that prompting induces well-characterized optimization benefit for fine-tuning. Further experiments reveal that the trajectory of prompt- based FT can often be described by kernel-based dynamics when the eNTK succeeds. The cNTK often achieves comparable performance to FT in the prompt prompt-based setting but struggles to solve multi-class and entailment tasks.
3. We straightforwardyy apply the kernel view of FT dynamics to formally analyze the success tone-tunin methods that update in a low-rank subspace of model parameters (e.g., LoRA, Hu et al. (022)). These results in Section 6 highlight how kernel-based understanding of FT can aid in the practical design and theoretical analysis of efficient variants.
4. We formally extend infinite-widthh analysis account for a pre-trained initialization and characterize conditions under which fine-tuning can exhibit kernel behavior. Using insights into the importance of prompting, we formally prove the existence of a rigorous mechanism through which prompt-based FT of complex architectures (e.g.. Transformers) can exhibit kernel behavior (Section 7 Analysis proceeds in the context of networks whose widths go to infinity (i.e., through the Tensor Programs framework), but unlike standard infinite-width NTK theory, it allows a non-random initialization (i.c., one that results from pre- mre-trainngg..
2 RELATED WORK
Kernel view of training. The infinite-width limit is a well-studied theoretical model for deep network optimization. JJacot et al. (2018) introduced NTK to capture training deep and infinitely wide neural network from random initialization. Subsequent experiments showed that the kernels underperformed for standard tasks (Arora et al. (2019)) but performed well on small datasets (i.e.. hundreds of examples) (Arora et al.] (020). Many works (Allen-Zhu et af. 2019a by Arora et al. 2019a Du et al. 2019b a Li & Liang, 2018 Zou et al., 2018] & Gu 2019) have since applied this lens to understand the optimization and generalization behavior of deep networks. However, these analyses of optimization and generalization do not directly apply to the pre-training and fine-tuning framework because (1) the network trained during FT is inherited and non-random; and (2) LMs are often trained with Adam, and the NTK formula only describes training infinitely wide network with SGD. In this work, we handle the issue of a non-random (i.e., pre-trained) initialization by assuming that the pre-training task is sufficiently related to the downstream task (Definition 773). and we derive new kernels to model carly-stage training with Adam (Section
Theory of self-supervised learning and transfer learning. Existing theoretical works on transfer learning focus on linear probing and use the representation to provide guarantees on various tasks related to the original training data (Du et al., 2021 Tripuraneni et al. 2020 Wu et al. 2020). Additionally, /aunshi et al. (222T) studied autoregressive language models to nigoruusly characterized why prompting can improve zero-shot task performance, but their analysis precludes an investigation of FT. We focus on the masked language model pretraining objective, but it is worth noting that there are many works (Saunshi et al., 2019 Tosh et al. 2 Lee et al., 2022 Tsai et al.) 202I studying transfer when pre-training with contrastive objective. However, experiments
language modeling (Abnar et al. 2021) and contrastive learning (Saunshi et al. 2022) recently demonstrated that properties of transfer between self-supervised pre-training and supervised FT cannot be fully captured by model-agnostic analyses that directly relate the pre-training and downstream task errors. Kernel theory provides principled optimization- and archifecture-aware framework to analyze FT.
2

---

Under review as a conference paper at ICLR 2023
Optimization of transformers. Several works (Zhang al. 2020 Liu et al. 2020a Li al.
2022) have documented issues with optimizing Transformer baseo architectures with SGD instead of Adam. To study the unique properties of optimizing transformers with Adam, we derive new kernel formula (Theorem 4.3) to capture early-stage training with Adam. We show results with this kernel and FT with Adam and SGD in Table
Variants of fine-tuning methods. A standard way of the-tuning pre-trained LMs as introduced in Radford et al. (2018); Devlin et al. (2019) is to add linear classifier on top of PT encoder and update all the parameters together. Subsequent work (Schick & Schütze, 2021 Gao et al. 202T) formulated downstream tasks as a language modeling problem (i.e., prompt-based FT) and demonstrated empirical suuccess in low-data scenarios (see Liu et al. (2022) for a comprehensive survey). Another line of research studies parameter-efficien fine-tuning methods in which only a subset of model parameters or the parameters updates are restricted to a www-dimensional subspace (Hu et al. 2021 Anghaaanyan al. 202T). We show that good eNTK performance arises only when studying prompt- based FT in Section 5 (Figure [T and we later show in Section that subspace-based fine-tuning methods such as LoRA et.. 2021) have simple interpretation through the kernel.
updated (Lester al. 2021 Ben Zaken al. 2022 Li & Liang 2021
PRELIMINARIES
3.1
KERNEL BEHAVIOR
It has been mathematically proven that training infinitely wide deep networks (with large Gaussian initialization) on small datasets can cause deep learning to turn into kernel-based learning. We are interested in identifying kernel behavior arising when training from an arbitrary initialization. Below, we adapt the definition of lazy regime (Woodworth et al. 2020) to an arbitrary initialization.
Definition 3.1 (Kernel Behavior). Consider neural network f(:: 0) that takes input and computes a scalar output") using 0 as the parameters. Let 0g be the parameters after t steps of training by gradient- based optimization algorithm. We say this training process of the network demonstrates kernel behavior if the following properties are satisfied.
1. Linearization: The change of the network can be approximated by its first order Taylor expansion, i.e.,
0.): - [(S: 0(-) ~ (V 0,-1), 0,-1);
2. Fixed Features: The gradient at step is approximately the same as before training, i.e.,
V (.) = (E)
f denotes the gradient of f w.r.t. 0. "Closeness to kernel behavior" is quantified using the difference in the quantities on the two sides of the As symbol.
Definition 3.2 (Kernel Analog). Suppose optimization of the parameters of model f using the gradient-based update algorithm A to minimize a loss E R2 - IR exhibits kernel behavior (Definition 3.1). Then, we say that kernel (A) is the kernel analog of the optimization algorithm
A if
S(E: I(E: 0z-1) R - (E)))
where & the training input of step t, O, is the parameter at step x(E. 0) ae(f((:)).v())) is the 8f
derivative of the loss with respect to the model output, and v(E) is the label of g.
We illustrate the dynamics of an optimization algorithm that demonstrates kernel behavior relates to the kernel analog. Let A be stochastic gradient descent (SGD). If SGD exhibits kernel behavior, then we can write how the function changes for chosen input & as
f(E: (15:0++1--11 ((E)))((((()))) xe (Vf(E;
0.+1 0,)) = (V 0,), - V (f(;;;)))) - (SGD) (E.E:) Ee),
Note that for C-way classification, f is vector in RC We say f exhibits kernel behavior if the Linearization and Fixed Features properties hold for every component of f. The subsequent definition of kernel analog also generalizes vector output component-wise.
3

---

Under review as a conference paper at ICLR 2023
where the approximations follow from the Linearization and Fixed Features property, respectively, This construction immediately yields the standard neural tangent kemnel (NTK) formula for K(SGi) derived in Jacot et al. (2018), which represents an input & as the resulting gradient V f(;: (o). Definition 3.3 (Neural Tangent Kernel K(SGDD), ((sSD") (E: 0o), (E'
Given an kernel K, one can solve the classification problem by learning kernel coefficients a; to minimize the empirical risk of E; a,K((, Ei)) where {5x} is the training data (see Appendix A). In Section 4 we derive the kernel analog for SignGD (i.c., an early-stage approximation of Adam), and in Section 5 we compare its eNTK performance against Adam FT. The eNTK computation relies on two design choices for the setting: (1) what the model output f(: 0) is, and (2) which optimization algorithm is being studied. For a given setting, the eNTK can be computed directly using the kernel analog (Definition 3.2) of AA We run experiments choosing A as SGD or Adam and choosing I based on the fine-tuning setting.
3.2 TRA-TIGINN AND FINE- TUNING PARADIGM
We focus our attention on masked language models (MLMs), such BERT (Devlin et 2019) and RoBERTa (Liu et al. 2020b), which are trained to minimize the cross-entropy loss on independently predicting masked tokens (i.e., a |V|-way classification task, where V is the vocabulary). Given text input s of length T from the pre-training distribution Sp1, replace small percentage (e.g.. 15%) of tokens with [MASK] tokens. This masked input is then fed into the representation function h: SPT - T x Rhth (e.g.. a Transformer encoder) to produce a low-dimensional contextual embedding for each position in the input. The contextual embeddings are independently multiplied by a classifier head (i.e., word embeddings)
RT* to produce logits that will be used to compute the probability of token filling each masked position.
Using a PT model solve downstream tasks effectively has been a highly active area of research. We focus on fine-tuning (FT) methods, which adapt the pre-trained model to a new input distribution SET using additional training on the C-way downstream classification task.
1. Standard FT (Devlin et al. 2019 Liu 2020b): To solve C-way downstream classification task, initialize and learl a new classifier head I'; R" + R on top of the contextual embedding of the [CLS), denoted hyaLs)- In this case, the choice of f SFT + Rº for the eNTK construction is fs) [(h(c1s(s)).
2. Prompt-based FT (Schick& & Schüitze| 2021 Gao et al. 202 :: Add natural language prompt (e.g. "This is [MASK].) in addition to the task input to the downstream task, and use the pre-trained MLM to fill in the masked token. Compute the logits over task-relevant words (e.g.. "great" and "terrible") using the corresponding columns of Φ denoted & E _ These logits will serve as surrogates to solve the downstream task. In this case, the choice of f Spr + RC for the eNTK construction is f(()) + hMAAKK] (s).
4 KERNEL DERIVATION FOR ADAM
Computing the eNTK requires using the kernel analog (Definition 3.2 of the chosen optimization algorithm A. However, is difficult to construct a long-term kernel analog for Adam, because the adaptivity causes the each update to depend on the entire gradient history. Previous work has shown that in the early stages of training, full-batch (Ma et al. (2022) and mini-batch (Malladi et al. 2022) Adam updates can be approximated as using the sign of the gradient. In particular, the moving averages for the moment estimates are computed in small neighborhood when the learning rate is small, so the Adam update is similar to performing coordinate-ve wiste normalization on the gradient. This gradient- based optimization algorithm is called SignGD, defined below.
Definition 4.1 (SignGD). SignGD is a gradient-based optimization algorithm that updates the parameters as 0.+1 0. - sign( ( ((r; 0.)), where sign applied element-wise.
II our experiments, Standard FT corresponds to initializing at the linear probing solution (i.c., training I' on the downstream task while freezing all other layers) and then performing FT. We do this because when FT exhibits kernel hchavior (Definition 3.1) it finds solutions close to initialization, and we hypothesize that the r learned during FT is closer the linear probing solution than random initialization.
4

---

Under review as a conference paper at ICLR 2023
We define the sign-based kernel below and prove that it is the correct kernel analog for SignGD. Definition 4.2 (Asymmetric SignGD Kernel), ((A-SiguCD)(EE
(V 0o), sign f 8o)).
Theorem 4.3 (Informal version of Theorem C.4). If network is trained with SignGD (Defini- tion ( and the training exhibits kernel behavior (Definition) 3.7), then the training dynamics can be characterized
f(g: f(;: ((E:0)-1)== TXeK (5,Et,
where is the derivative of the loss with respect to f at step
Proof sketch. By the Linearization property in Definition 3.1
[(; 0+) (E: 01-) (VI(E:0.),0, 0.-1) - -V.(s "1
Then by the Fixed Features Definition 3.1
(V sign V ((xz; 0,-1))) ~ (V (O) sign (V (E{;)))))) - ACTA-SinnDD (E.
We solve the asymmetric kernel regression by building augmented system modified from He et al. (2022b) (Appendix A.3), but the difficulties of solving the kernel regression problem with an asymmetric kernel motivate us to also use the symmetric SignGD kernel, though it is as theoretically sound as the asymmetric one.
Definition 4.4 (SignGD Kerne).. ((SignGD) = (sign (V ff(E:))), sign(V (
The kernel analog for Adam differs from the standard NTK formula for SGD because the sign function is agnostic the relative scales of the gradients.
5 EXPERIMENTS
We compute the eNTK described in Section 3 for different optimization algorithms and FT settings. eNTK performance being comparable to FT performance is a necessary but not sufficient condition also directly measure if the Linearization and Fixed Features properties hold (Section 5.3 Overall, we find that only prompt-based FT exhibits kernel behavior, although the eNTK still struggles with multi-class classification and exhibits anomalous behavior on entailment tasks.
for FT to exhibit kernel behavior (Definition 3.1.
5.1 SETUP
Our experiments follow the few-shot setting from Gao al. (2021) and use their manual prompt templates. We consider 12 NLP tasks, divided equally into 6 single sentence and sentence pair datasets, which cover: sentiment analysis (SST-2, MR, CR); classifying opinion's polarity (MQPA) or subjectivity (Subj) or question type (TREC); natural language inference (MNLI, SNLI, QNLI, RTE); and paraphrase detection tasks (MRPC, QQP). For each task, we randomly sample k-shot datasets with i training examples for each label. We use a pre-trained RoBERTa Tasebase (Liu et al. 2020b) for FT and eNTK. Appendix conotains further details on datasets and implementation.
5.2 KERNEL PERFORMANCE ON DOWNSTREAM TASKS
Prompting is critical for eNTK to match FT performance. We measure the eNTK performance in the standard and prompt- based FT settings across SST-2, MR. QNLI, QQP (Figure D.. In the standard FT setting, exhibit only I % gap the prompt-based setting.
and SGD FT demonstrate gap of up to 15% absolute on tasks that
SGD performs comparably to Adam in prompt-based FT. We focus on the prompt-based FT setting (Table [ We note that when doing prompt-basen based FT, Adam and SGD perform within 4% absolute of each other, furthering the discussion around the optimization of transformers with Adam
versus SGD (Li al. 2022
Zhang al. 2020 Liu al. 2020a).
5

---

Under review as a conference paper at ICLR 2023
k 16
100
80
60
Standard SGD FT
40
Standard k
k == 64
100
Prompt SGD FT
80
Prompt g(
60
40
SST-2
MR
CR
QNLI
QQP
Figure 1: Comparing SGD-FT and KXSGD) performance in the standard and the prompt-based FT settings (Section
3) suggests that kernel behavior (Definition 3.1 can only arise when using a prompt.
Prompt-based eNTK matches FT in most tasks. To study if the eNTK can solve given task in the prompt- based FT setting, we compare SGD-FT to K(SGD) and Adam-FT to ((A-SignGD)) and ("(ignDD) in Table I We observe that for 8 of 12 tasks, JC(SGD) can achieve accuracy within 5% absolute of SGD FT for k = 16 and A: = 64. Similarly, ((Sigg) or ((A-SignGD) can achieve accuracy comparable to Adam FT for out of the 12 tasks. The difference between ((SignGD) and K(A-SignGD) is negligible on most tasks. We suggest the asymmetry of the latter may cause ((A-SignGD) to sometimes perform worse than ((Siga(iD) despite being the theoretically sound kernel analog (Theorem 4.3
eNTK struggles with multi-class tasks. The eNTK performs much worse than FT on all of the multi-class (i.e., TREC, MNLI, and SNLI) tasks, which we believe warrants further investigation. We conjecture that the eNTK cannot solve MNLI and SNLI despite solving other entailment tasks (i.e., QNLI and RTE) because the prompt is less natural when considering the label word "Maybe". One explanation why the kernel analog sometimes outperforms FT that certain batches may induce anomalous gradients that disrupt the FT trajectory, the effect of which the kernel can mitigate by downweighting these examples.
5.3 MEASURING KERNEL BEHAVIOR
The eNTK matches the performance of prompted FT for many tasks (Table suggesting that these tasks may induce kernel behavior (Definition 3.T). However, the kernel's s success may just be coincidence. We take additional measurements to provide further empirical evidence that could be modeled kernel behavior.
Measuring the Linearization Property of Kernel Behavior If FT exhibits kernel behavior (Definition 3. then the function output after FT should be close the first order Taylor expansion around the pre-trained model:
f(g: 0pT) = f(5: 0pr) + (V f(:: 0pt), OPT - 0pT)
where Opr is the model parameters after pre-training, OFT is the model parameters after fine-tuning on the downstream task, and sampled from the test set. Figure summarizes the results.
Pre-trained models perform significantly better than random on many single-sentence downstream tasks (e.g., SST-2, MR, and CR) but close random on most sentence-pair tasks (e.g.. QNLI RTE, MRPC, and QQP). Subj. MNLI, and SNLI are outliers to this trend. The linearized model recovers substantial amount FT performance for SST-2, MR, CR, Subj. RTE, and QQP, of which the eNTK could solve [T.
Although pre-trained models perform much better than random on MNLI and SNLI, we find that the eNTK cannot solve these tasks very well (Table D. Similarly, although the pre-trained model demonstrates near-random performance on QNLI and RTE, we find that the eNTK can solve these tasks. Moreover, although QNLI and RTE could be solved by the eNTK, the results suggest they do not induce the Lincarization property kernel behavior very strongly. Altogether, these findings suggest deeper mystery around entailment tasks in particular.

---

Under review as a conference paper at ICLR 2023
Measuring the Fixed Features Property of Kernel Behavior We also empirically test if the Fixed Features property (Definition 3.1) holds for tasks that the eNTK can solve. We measure the relative distance between IC(GDD) computed before and after FT and record the average clement-wiss distance in Table 3 A smaller distance indicates that the Fixed Features property is more likely to hold. We that tasks that the eNTK solve exhibit relatively low (i.e., less than 1) distances.
s-shot
Method
SST-2
MR
CR
MPQA
Subj
TREC
SGD-FT
(1.5)
(3.2
(3.3
(3.3
88.5
(7.3
K(SGD)
88.3(..3)
84.7 (1.6)
89.5(0.6)
76.4(2.7)
88.6(1.3)
56.0(9.2)
Adam-FT
88.3
(3.0 (1.3
82.8
87 (2.4(
79.6(8.1)
K(SigiiiD)
(8.3
84.3 (1.5)
89 (4.0(a.))
76.7
(2.2)
58.9(1.1)
CCAA-Sigaiiy
88.3
84.9 (1.1)
88.0 88.0(1.8)
74.6(3.5)
88.6(1.8)
20.0(2.5)
64
SGD-FT
(0.7
85.6(1.1)
94.3
84.8 (0.8)
92.9 (0.3)
93.2
KCSGD)
89.2
(6.4(0.0)
89 (9.8().3)
81.2
91.4(0.7)
77.8(2.3)
Adam-FT
(0.3
(6.0))
(0.7
84.6
92.7(0.0)
92.6(1.3)
SiginGD
(9.3)
85.6(1.0)
90.0
78.6(6.4)
92.4(0.3)
82.0(1.4)
KCA-SSgsKiD)
(8.9
85.6(1.0)
90.1
81.8
(L.1)
21.0(4.3)
(a) Singlo-sentenee tasks
k-shot
Method
MNLI
SNLI
QNLI
RTE
MRPC
QQP
16
SGD-FT
(2.2
(5.7
(2.1
60.0
73.9
62.1
KSGGD)
53.0
57.8(233
60.1
60.0
73.4(6.0)
58.2
Adam-FT
56.8(2.9)
64.6(1.1)
(3.1
57.6(6.3)
77.6(3.1)
61.8(1.5)
((SignGD)
(3.2
(3.7
59.5(3.1)
55.4
75.6
60.7 (0.7(2.2)
((A-SigaGD)
51.9
54.9
56.0
59.8
75.2
59.4(2.0)
SGD-FT
(8.7
(7.3
72.8(2.2)
(8.9
(1.2)
69.2
K(SGD)
60.4(..8)
65.5
67.3(1.6)
66.5(2.5)
79.2
66.4(1.7)
Adam-FT
(7.9
76.9 (1.4)
(4.2
67.3
80.9 (1.2)
(9.8().6)
10 (iginDD
(9.9
(3.9
65.4(..)
63.8
77.4
63.7(1.4)
((A-SigaGD)
(8.5
66.8
66.5
63.8(2.2)
77.3
66.1(3.4)
Sentence-pair tasks
Table 1: Performance achieved by prompt-based FT and prompt-based eNTKs with different formulas on the LM-BFF test set (Gao et al. 2021). The eNTK performs comparably to the analogous FT on many tasks but fails on multi-class tasks (i.e., TREC, SNLI, and MNLI). Performance is measure by average test accuracy over 5 k-shot splits for all tasks except MRPC and QQP, where is Fl.
k = 16
100
k= 512
Lin
100
FT
SST-2
MR
CR
MPQA
Subj
TREC
MNLI
SNLI
QNU
RTE
MRPC
QQP
Figure 2: Accuracies of zero-shot pre-trained model (PT), linearized model (Lin., see Definition and fine-tuned model (FT). Tasks that induce the Linearization property of kernel behavior (Defini- tion 3. will show that Lin. performance recovers substantial amount of the FT performance. For each lt we plot the median and range of the test accuracies across seeds and data splits.
3.1
7

---

Under review as a conference paper at ICLR 2023
6 EFFICACY OF SUBSPACE- BASED TUN--UNNIG METHODS
We study parameter-efficient fine-tuning methods that roughly preserve performance but reduce the overhead of fine-tuning and saving large language model (He al. 2022a One such method is LoRA (Hu et al. (022), which restricts tiee-tnning updates to be rank as defined below.
Definition 6.1 (A-LoRA FT (Hu et al. 2021). I). Let A be gradient-basee optimization algorithm. R with i.i.d. mean-0 Gaussian values and B E Ix*x as O. Set the weight to be W + AB. To fine-tune, fix W at its pre-trained value and train only A and B using A.
For every weight matrix W R"*x choose < m and initialize
We also consider fine-tuning variant that projects the parameter vector to low-dimensional subspace. This method was originally proposed to characterize the difficulty of downstream tasks (Li al. 2018), and recent LM experiments in Aghajanyan al. (202I) have shown that fine-tuning the
projected parameters can recover most of the performance of standard fine-tuning
Definition 6.2 D4-mntiinsicDimmnsion FT (Li et al. 2018 Aghajanyan et al. (021)). Fix a random projection II x where M the number of parameters in model f. To fine-tune using a loss É on a downstream task, replace the gradient in the update formula of A with 11 V)(E;
Although theoretical characterization of these methods seems complex, the kernel view admits a simple interpretation. We straightforwardly apply the classical in inJoonson ( 1984), which guarantees inner product preservation under random projections, to show that these methods approximately preserve the SGD kernel (Definition 3.3
or JL, lemma
Theorem 6.3 (LoRA and IntrinsieDimension FT preserve KCSGD) informal version of Theorem D.5 Let K(SGD) be the kernel analog (Definition 3.2 to SGID FT, LLRA be the kernel analog to SGDD
LoRA FT (Definition 6.11 and k(SGD) be the kernel analog to SGDD-nntinsicDimmnniin FT on downstream task E. Then, with high probabililty, (KCEA (i.5) -KISGDI (i, (i,j)))/3/S0D zo 0 and X-SeP) (i,j) z KISGD) (i, for all i.j [.].
KnD
Proof sketch. Consider an individual layer in the network and imputs E E to the downstream task. LoRA randomly projects Vgf 8) and V f (E'E:')), f). where Vg denotes the gradient with
respect to B, and does not modify the gradient to A, since B is initialized zero. The rest of the proof for LoRA and the proof for IntrisicDimension FT follows from applying JL to all such pairs E' to show the inner product, which determines the kernel entry, is preserved.
O
Remark 6.4. Theorem 6.3 states that the kernel analog of SGD FT is unchanged by LoRA both prompt-based and standard FT. Therefore, the theorem only applies when A FT exhibits kernel behavior, which we find to only be in the prompt-based setting (Figure !
Experimental results in Table 7 verify that prompted SGD FT and prompted LoRA-SGD FT achieve similar performance on several downstream tasks, and K achieves performance similar to ((S))) We leave it future work account for the success these methods when FT does not exhibit kernel behavior.
7 THEORY: PROMPT-BASED FINE-TUNING CAN EXHIBIT KERNEL BEHAVIOR
We give a plausible mechanism for how prompt-based FT can exhibit kernel behavior (Definition 3.1 as the network width grows large. We start by defining a pre-training scheme, which formalizes how changing the architecture width impacts tre-training.
Definition 7.1 (Pre-Training Scheme). A pre-training scheme (X, A, F²) with width contains the dataset X, optimizer A and its hyperparameters, and a model architecture F Let fr ~ (X, F") denote model resulting from training the architecture J the dataset X' with optimizer A.
Remark 7.2. The concrete reliance of the architecture on the width is given by Tensor Programs: for example, in a Transformer, increasing n corresponds to increasing the embedding dimension.
We now connect pre-training to the downstream task. Analogous to Saunshi et al. (02T)) we reason that prompting transforms the downstream task into fill-i-tthe-blank problem, and thus the
8

---

Under review as a conference paper at ICLR 2023
downstream task can be viewed as subease of the pre- pre-training task. We then assume that wider pre-trained network will be better at filling in masked tokens and that an infinitely wide pre-trained network can solve the downstream task perfectly when using suitable prompt.
Definition 7.3 (Natural Task in the Sfinite-Width Limit). We say that downstream task E is natural with respect to a pre-training scheme AX F if, for any 8 (X),, and any { E E,
lim X(E, (E)) n-000
Remark 7.4. Note that task may only be natural in the infinite-wiet limit when using a prompt, since standard FT will always require training randomly initialized head (i.e., will not vanish at infinite width).
Although Definition 7.3 is asymptotic, we design a cheap empirical test. We require access to two models of different widths resulting otherwise identical pre- pre-training schemes: f AA A. F and f ~ (X' A. F*2). Then, we can check if x decreases with width by measuring x(5, f (E)) and x with n, # n22 without making any gradient updates (Table 8).
To study the behavior of fine-tuning one also needs to make assumptions about parameters that resulted from the pre-training. In particular, we assume that the network can be written a Tensor Program (Yang) 2019; (b). which is sufficiently general to allow our theory to describe many complex architectures (e.g., Transformers). To allow the analysis to proceed by way of Tensor Programs, we require that the network is (1) stable: its output does not grow with width (i.e., the infinite-widit limit is meaningful), and (2) non-trivial: its output can be updated during fine-tuning (i.e., learning can occur).
Assume the downstream task E is natural in the infinite-width limit with a pre-trained model J. aandd I is stable, non-trivial, and can be written as Tensor Program. Then prompt-based FT of f will exhibit the Linearization and Fixed Features properties of kernel behavior (Definition 3.1.
Theorem 7.5 (Informal version of Theorem C.5
The proof of the theorem formalizes the intuition that if the pre-trained network is already decent at solving the downstream task, the network needs to only mildly adapt to solve the downstream task. Notably, we extend standard NTK theory to account for arbitrary initialization and to characterize early-stage training with Adam (see Section 4 for kernel).
8 CONCLUSION
We use NTKs to mathematically formalize the general intuition that fine-tuning pretrained language models solve downstream tasks requires only "small change." Extensive experiments 12 NLU tasks demonstrate that prompt-based FT is much more likely to exhibit kernel behavior (Definition (3.]) than standard FT (Figure [].. Further experiments in the prompt-based FT setting using a newly derived kernel for Adam (Definition (42)) see Theorem 4.3) demonstrate that the eNTK can match the performance of FT on many tasks. On the tasks that eNTK can solve, measurements in Section 5.3 suggest that prompt- based FT does exhibit kernel behavior. We demonstrate one possible use of the kernel view to explain empirical phenomena by applying to understand subspace- subspace-based fine-tuning methods (Section 6), and we note that the kernel has many mathematically useful properties that can aid design and study of parameter-efficien fine-tuning methods. Moreover, one can use the kernel to study the inductive bias of FT, as done for gradient descent from a random initialization in the past (Allen-Zhi et al. 2019b a Li & Liang] 2018). We provide a first-cut thcoretical analysis in Section
to why prompt-basee fine-tuning can exhibit kernel behavior.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training, 2021.
ptattts::/axiiin wwtp://gggaaaa////eee aaaaabs//211 02095
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 1 URL https
V/aclantho Logy. org/ 2021 acl-long..5000
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized
neural networks, going beyond two layers. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a. URL https: http:///poceeeeiigg...e
www.unnnn..nuiuinnne meurips. cc/
Paper. pdf
Zeyuan Aluen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterizzation. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 242-252, PMLR, 09-15 Jun 2019b.
URL httphhtsss//pprcceeciiiiic
ml www.r.ppessssee /v97 aat//alleennaa zhul 9a html
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 322-332, PMLR, 09-15 Jun 2019a. URL httttttts///ooeeeeee mlr. wll.ppees/v97/aaa https/////arrrraaaaaa html
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019b. URL http:::tpps.........t....p http:://////oceediiipeeo www.fffffe2aabaaaccm be f pdf
neurii ps
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. International Confer- ence on Learning Representations, 2020. URL https: /ottps:///penreview net net/forummid
Ek18sJBYvE
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006. URL htttps://cittseeerr ist. psu. edu /vi ewdoc/ http://wwwwwoddoooooooooo 85526 reppee
oad?doi=10 1............................... 1.660
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficiet fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of
the Association for Computational Li Linguistics (Volume 2: Short Pupers), pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1/2222.coooo URL org/ 2022.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL PeSCALLecogniinngg textual entailment challenge. In TAC, 2009. URL https::/ccitett
ist psu. edu/
1. 232 1231 repperreepe
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September
2015, Association for Computational Linguistics. doi: 10.18653////D15-10007 URL https / orw.la/.hooooy..ogg/o111-------.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: http:tppppppppoooeciiiin.ee wtw.:eedwinnnn..neesses......
F3 orF http://ppppppcpcaaaaaaaaaaaaaaaa
10

---

Under review as a conference paper at ICLR 2023
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020. htth::t::: wwt./ppeeeviiieee wwt.:/eeeeoommmmi/1ee.....
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In the First International Conference on Machine Learning Challenges: Evaluat- ing Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, 2005. URL httpt:/tt cs. ksu. www.eoouor///aaaaaaaaaaaa Fa01- (2.00//CIS990 9 06 -o-dagannSpaacal ppdf
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), pp. 4171-4186, 2019.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In the Third International Workshop on Paraphrasing (IWP2005), 2005, URL
org/ 105-5002- pd.
Simon Du Jason Lee, Haochuan Li. Liwei Wang, and Xiyu Zhai Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1675-1685. PMLR, 09-15 Jun 2019a. URL htt.hhtss:pprrccedi.gge mlr wwr.tppeessssse paress/v97/uur du 9c html
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2019b. URL /
net forum? http:/iiwwwee//fourrrrrrr
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. International Conference on Learning Representations, 2021. [ttp::tt::///peeeeeeee
ew. net/ forum? dd=pm TMD.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Association for Computational Linguistics (ACL), pp. 3816-3830, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recog- nizing textual entailment challenge. In the ACL-PASCAL Workshop on Textual Entailment and Paruphrasing, 2007. haattss:/aclantttoo.co
pdf
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor KinkKKirkpatrick, and Graham Neubig. Towards unified view of parameter-effiieet transfer learning. In International Conference on Learning Representations, 2022a.
Mingzhen He, Fan He, Lei Shi, Xiaolin Huang, and Johan A. K. Suykens. Learning with asymmetric kernels: Least squares and feature interpretation, 2022b. URL httttts//axxx....///// 22C2. 01397
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https
7/arxiv. org/ abs 2ttbb///11..000 09685
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In ACM SIGKDD interna- tional conference Knowledge discovery and data mining, 2004.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and general- ization in neural networks. In Advances in Neural Information Processing Systems, volume 31, Cur- Associates, Inc., 2018. htt.htttss//pprcceeinnae wew.ssnnuuuiii lps. ccppp/er//pap/ / htle://aeeaa33ee222aeeeeee222223
pdf
William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math., 26:
189-206, 1984.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre- pre-training by representing and predicting spans. Transactions of the Associa- fion of Computational Linguistics (TACI.), 2020.

---

Under review as a conference paper at ICLR 2023
Jason D Lee, Qi Lei, Nikunj Saunshi, and JIACHENG ZHUO. Predicting what you already know helps: Provable self-supervised learning. In Advances in Neu- ral Information Processing Systems, volume 34, pp. 309-323 Curran Associates, Inc., 2021. URL https::/procseednng www.rrooceeeeineeeuue...ee neurips 1ps cc cttccc//pppp.e htce/ppper/222//F166
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Empirical Methods in Natural Language Processing (EMNLP), pp. 3045-3059, 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. URL ftt.::::/openreviie
net/ Forumnid--yuup---ee
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/
URL wtt.t//ppp///a....ttoooooooo...p awt.:/22221.a..oooooo.o 353
Yuanzhi Li and Yingyu Liang. Learning overparameeterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Informattinn Processing Systems, volume 31. Cur- Associates, Inc., 2018. URL URL/httss: /ptpooeeppe ngs newr ps catsscc/ppe cc/ppppr///11. file/ (tlee/5ff97765 6bal 70c1 1ac.4fa6766611111eeeeee333333333 Paperr-aappppppp .pff
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 12656-12684, PMLR, 17-23 Jul 2022. URL
www. mwt.press..pp..11///
1i22b. html
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the dif- ficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 5747-5763, Online, November 2020a. Association for Computational Linguistics. doi: 10. 11.11865550 1225:////200..mmmmmmmmoo https::/acclathooll. www./..rrg..... emn11 ma 463
URL
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., aug 2022. ISSN 0360-0300. doi: 10. 10.1145/3560815. URL
https://dol www.oogg/ooo....ooo 111111..11133333 00114////588111
Just Accepted.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [bertt) A robustly optimized (bert) pretraining approach, 2020b. URL httpstt:: www.owwweww.w....oo www. wtweet/fformmmmmmysssssss
Chao Ma. Lei Wu, and Weinan E. A qualitative study of the dynamic behavior for adaptive gradient algorithms. Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings Machine Learning Research, pp. 671-692, PMLR, 16-19 Aug 2022. URL htttps:/pproceecingg htw.:///w..........eess.s.e..... wwwppe/eeee//naeee
mlr
html
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDEs and scaling rules for adaptive gradient algorithms. and Kyunghyun Cho (eds.), Advances in Neurul Information Processing Systems, 2022. URL https: /ttp::///ppereeee/ ew wwt. forteeooommmiiie F JHKQP)
Alice H. Oh, Alekh Agarwal, Danielle Belgrave,
Roman Novak, Jascha Sohl-Dickstein, and Samuel S Schoenholz. Fast finite width neural tangent kernel. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17018-17044, PMLR, 17-23 Jul 2022. URL ht htttp:://procestt ngs .am.s.mm....... www.gngs.m............ee
wwwpsr/ssssee vv 62/nooaa222a .html
Bo Pang and Lillian Lee. sentimental education: Sentiment analysis using subjectivity summariza- tion based minimum cuts. In Association for Computational Linguistics (ACI.), 2004.
12

---

Under review as a conference paper at ICLR 2023
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Association for Computational Linguistics (ACL), 2005.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn. Rex., 21 2(114)):1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. Empirical Methods in Natural Language Processing (EMNLP), 2016. URL httltttss///aatthlllllo
org/D16 D16 016-1 (6666/126
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar, A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5637. PMLR, 09-15 Jun 2019. URL
mlr.
html
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. mathematical exploration why language models help solve downstream tasks. In International Conference on Learning Representations, 2021. URL
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19250-19286. PMLR, 17-23 Jul 2022. URL
http://dddiinn.......reee////aaa hhmlllll
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. European Chapter of the Association for Computational Linguistics (EACL), pp. 255-269, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), 2013. BtLhttps pdd
//aclantho www.enhhoooy....oo wwwpyoorgD11311111
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pp. 1179-1206. PMLR, 16-19 Mar 2021a. URI. httpt:ttpp/.omm /atttps://prooeeddinee ml wtt..reeres////////2222222 htm
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. Journal of Machine Learning Research, 22(281):1-31 2021b. URL httt::://mir..rg///////a
2atpee///v22221100020 21-0089 .tm00.99.lmllm.mmmmmmmmm.mmmm...
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. In Advances in Neural Information Processing Systems, volume 33, pp. 7852- 7862. Curran Associates, Inc., 2020. URL httts::/ppooceedinnntt
www.wps.......... wweurips.cc/pppes cc/paper/
Yao- Hto-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Self-supervised learning from multi-view perspective. In International Conference on Learning Representations, 2021. BtRp:htts:///peeee
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. In the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 2000.
13

---

Under review as a conference paper at ICLR 2023
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019. URL /tpptttt////eeee....e.... wwwpoewwee......... net/
forum? 1 3==.J44m22557
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learming, volume 162, pp. //proceedings..o mlr wrwssppsspeee 62/ we http///e2222....mmm html
PMLR. 17-23 Jul 2022. URL https
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3), 2005.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-IILT), 2018. URL https
//aclanchology.oom org.h///1ooooo../1B888 11011pp0.p..
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 3635-3673. PMLR, 09-12 Jul 2020. URL
Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer
in tulti-task learning. In International Conference on Learning Representations, 2020. URL
net http://wwwwweeeeetteeeeeeae
Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
Greg Yang. Tensor programs i:: Neural tangent kernel for any architecture. arXiv preprint orXiv:2006.14548, 2020a.
Greg Yang. Tensor programs ii:: Neural matrix laws. arXiv preprint arXiv:2009.10663 2020b.
Greg Yang and Edward Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Greg Yang and Etai Littin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In International Conference Machine Learning, pp. 11762-11772. PMLR, 2021.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466. 2022.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, San- jiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Ad- vances in Neural Information Processing Systems, volume 33, pp. 15383-15393, Curran As- sociates, Inc., 2020. URL htt Tttp:///SSaBbD0/2/////6066622223
neurii httpp..ccppppee httpsccaaaaap///2///111e
.pdf
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes parameterized deep relu networks, 2018. URL
org/ abs/ 1811. 08888
A EXPERIMENTAL DETAILS
A.1 DATASETS AND PROMPTS
Table 2 shows the set downstream tasks, which are adapted from Gao et al. (2021). We consider single sentence classification datasets (SST-2 (Socher al. 2013). MR (Pang & Lee. 2005). CR (Hu
14

---

Under review as a conference paper at ICLR 2023
Dataset
C
#Train
#Test
Type
Prompt
SST-2
6,920
872
sentiment
<5,> was [MASK]
greatt terrible)
MR
8,662
1,000
sentiment
<S,> It was [MASK]
greett terrible)
CR
3,175
500
sentiment
<S1> was [MASK]
grrat, terrihle)
MPQA
8,606
1,000
opinion polarity
<S,> was [MASK]
great, terrible)
Subj
8,000
1,000
subjectivity
<Si> This is [MASK]
subjective, objective)
TREC
5,452
500
question cls.
[MASK] : <S1>>
Description, Expression, Entity,
Human, Location, Number)
MNLI
392,702
1,000
NLI
<S,> ? [MASK] <Sp>
Yes, Maybe, No} Yes, Maybe, No}
SNLI
549,367
1,000
NLI
<S,>? ? [MASK] <S²>
QNLI
104,743
1,000
<S,> ? [MASK] <S2>
Yes, No
2,490
277
NLI
<S,>? [MASK] <5²>
Yes. No
MRPC 3,668
408
paraphrase
<S,> [MASK] <S2>
Yes. No
QQP
363,846
1,000
paraphrase
<5,> [MASK] <52>
Yes, No}
Table 2: The statistics and prompts of the datasets we used in our experiments. The choices of prompts are from Gao et al. (2021) which include template, and set of label words that are used to fill in the [MASK] token. <S1> > and <52> refer to the first and the second (if any) input sentence.
Liu 2004). MPQA
(Wiebe et al. 2005). Subj (Pang & Lee 2004) and TREC (Voorhees & Tice
2000)). and sentence pair datasets (MNLI (Williams et al. 2018) SNLI (Bowman et al. 20155).
QNL (Rajpurkar et al.
2016p. RTE (Dagan et al. 2005 Bar Haim et al. 2006; Giampiccolo et al.
2007 Bentivogli et al.
2009), MRPC (Dolan & Brockett 2005) and QQH Our datasets represent
6/8 datasets of the GLUE benchmark Wang et al. (2019) (SST-2, MNLI, QNLI, RTE, MRPC, QQP).
In contrast Gao et al. (2021), we make two modifications the test sets. First, we split CR into 500 test examples and 3, 175 training examples to ensure enough training examples for our 512-shot experiments and secondly, we limit the test sizes to 1, 1,000 examples speed up kernel evaluations.
To generate k-shot few-shot datasets, the original training data is used to randomly sample K examples per label for training and another, separate k examples per label for the validation set. Unless otherwise stated, we usually run experiments over 5 seeds of few-shot data sets. We directly use the 'manual' prompt templates and label words proposed by Gao et al. (002)) which are reproduced in T
A.2 COMPUTING THE KERNEL
We adapt the functorch implementation of Novak et al. (2022) to compute the eNTK for a large model, using a mix of backward-moo auto-differentiation for computing the jacobians and forward-moot mode auto-differentiation for computing jacobian- vector products. Note that (((ignGD) cannot be computed using jacobian vector products and thus requires significantly more memory and run-time in practice.
A.3 SOLVING THE KERNEL.
In the standard NTK setting, the initial output of the model [(; 0o) contains no information about solving the task, because Og is random initiaization. However, in the prompted FT setting, we expect the pre-trained model to be able to solve the downstream task well even before any fine-tuning occurs (see Table 4. So, we add the pre-trained model's output the output from the kernel. Furthermore, we run grid search over scaling the labels in order to take advantage of any pre-existing knowledge the model has about the downstream task. In particular, the kernel regression is based on the &g distance to the ground truth one-hot vector, but the pre-trained model outputs the logits which will be used for cross-entropy loss. Scaling the one-hot vector by So helps align its scaling with the logits. Our hyperparameter grid for fo can be found in Table where oc corresponds to not using the pre-trained model logits when solving the kernel.
Solving Multi-Class Tasks There are several options for how to solve C-way classification tasks (C > 2). We perform the most general one, which scales with C Each logit is treated as an independent output of the network, essentially scaling the size N of the original dataset by a factor of
https:///www hwtp://wwwwww.../o///////////aa. quora.oo com/ wttp///uuooaaaaaa//////o
15

---

Under review as a conference paper at ICLR 2023
C. With CN examples, the kernel now has shape CN CN. The labels also scaled up treat the multi-class problem as many binary classification problems. Solving the multi-class task this way allows the kernel regression model to view relationships between different logits.
Symmetric Kernel Given symmetric kernel X E RN**,, we solve the kernel regression problem.
In particular, use the representer theorem to write that the empirical risk minimizer of the loss be expressed as a linear combination of the kernel features computed on the train set.
N
N
h h-(+) = arg min NEHK N
Σ ((h(zz)) 2=1
+
hT(+))
Σ a.Ki (., x4))
for a given loss function L. The symmetric SignGD and SGD kernels train a via gradient descent to minimize a regularized logistic loss the downstream task. We search over grid of regularization strengths chosen proportional to [|C]llop. see Table For a test input æ, the kernel outputs the prediction h(x) E. (x,
We write how to solve the kernel regression problem with an asymmetric kernel, developed in He et al. (022b), here. Consider the augmented linear system:
Asymmetric Kernel
1/2
H
o
[]]
1/
N
1/7
where H1, (x,) (o (x(x))y, with &, and di as the two different feature maps and y, the label for the ith example. Define w" and v² as
w Σ (edd)
Solving this system yields two discriminant functions:
fx(x) = () f((x) = K(X,,)(a* OY)
where ((;;, (z;),
We can thus create one discriminant function as cff(2) + (1-c) f((r) where [0, 1] is some hyperparameter. When o, Øt. we see that f. ft and we reduce to the standard kernel problem (though with repeated equations). Note that He He et al. (2022b), this system is only meaningful in terms of stationary points when training and / using the least squares loss.
We now leverage some specific knowledge about the NTK setting. In particular, we know that we should only use f. as the predictor in order to correctly represent new test input in the kernel analog for SignGD.
Hyperparameters and Implementation We follow Gao et al. (021) in using the few-shot valida- tion set to search over hyperparameters and finding the best hyperparameter per few-shot dataset. We use value ranges given by Gao et al.) (2021) and Hu al. (2021), and search over wider range of values for SGD. Table shows the hyperparameter grids for fine-tuning and the kernel method.
Gao et al. )021) train for 1000 steps in the 16-shot setting, and validate the performance every 100 steps to take the best checkpoints. As we consider varying values of k. we use the formula training for 32kC steps and validating every 4kC steps, where C is the number of classes in the dataset. This gives a comparable number of training and validation steps for binary tasks in the 16-shot setting.
B ADDITIONAL EXPERIMENTAL RESULTS
C KERNEL BEHAVIOR AND THE PARAMETRIZATION
As mentioned in Section 7 there are two behaviors of training neural networks - kernel behavior and feature learning behavior. Kernel behavior provides tractable tool to study the training of
16

---

Under review as a conference paper at ICLR 2023
Experiment
Hyperparameters
Values
SGD FT
Batch size Learning rate
(2,4,8) x {le-4, 5e
11.1e,,,50 50,, le-2]
SGD-LoRA
Batch size
{4, 16) x
Learning rate
(1c 4, 1c- 3, 1el11-2) 2] x 168) (2, x
(CLoRA, (LoRA)
Adam FT
Batch size
Learning rate
(1e-5,2e-55 5e-
Adam- FT
Batch size
(4,16)
Learning rate
{1e-5. 5, 5, [(8, 16)3
(FLoRA, (LLoRA)
KisGD) K(SignDD)
Kemel regularization fo scaling
(0. 0.001, 0.01, 0.1, 1) x f10, 100, 1000, 10000, oo)
Signo)))
Kemel regularization fo scaling
(0, 0.001, 0.01, 0.1, 1) X
(10, 100, 1000, 10000, oo) x
Kemel
{0.01, 1, 10) x {1}
Kernel
Table 3: The hyperparameter grids used in our experiments.
SST-2
MR
CR
s-ott
Lin.
FT
Lin.
FT
Lin.
FT
0
79.0
71.9
86.2
16
87.5 (1.3)
(1.2)
(1.8
(6.3
(o.6)
93.0
64
88.6(0.4)
(9.3
85.0
86.0
94.0 94.0
93.7 (0.8)
512
(0.2)
90.7 (1.2(
(6.3
88.6(0.6)
93.7 (0.8)
MQPA
Subj
TREC
k-shot
Lin.
FT
Lin.
FT
Lin.
FT
0
68.2
54.6
27.4
16
75.6 75.6(3.1)
82.8(2.2)
82.9
87 (7.4(2.1)
30.4(7.2)
79.6(6.1)
64
75.6(2.3)
(0.0
(14.0
92.7 (0.6)
31.2
92.6(1.3)
512
72.1 (2:2)
87.7
(3.8)
94.8
35.4
97.1 (0.3)
MNLI
SNLI
QNLI FT
s-ohot
Lin.
FT
Lin.
FT
Lin.
0
48.1
49.8 (7.2
51.2 57.5
16
(3.6(6.4)
S6.8
64.6(1.1)
63.1 (3.5)
64
55. (1.8)
67.9(1.0)
56.9
76.9
60.4
74.2
512
55.1 (5.1)
77.9 77.9(0.8)
55.3 (6.1)
83.5 (0.3)
67.7 67.7(2.0) (2.0)
82.6(1.)
RTE
MRPC
QQP
shot
Lin.
FT
Lin.
FT
Lin.
FT
0
53.1
41.7
42.7
16
55.4
57.6/6.3)
57.7
68.9(2.4)
57.5
61.7(6.))
64
59.6(2.9)
(7.3
64.2
73.8
61.7 61.7(0.4) (0.4)
72.7 (1.8)
512
55.5
75.4(3.0)
57.9
84.8
66.4(1.5)
78.7
Table 4: Accuracies of pre-trained model (0-shot), linearized model (Lin., Definition 3.1 and fine-tuned model (FT). Tasks that exhibit the Linearization property of kernel behavior (Definition 3. during fine-tuning will show that Lin. performance recovers substantial amount of the gain in performance achieved by performing fine-tuning, Accuracies are averaged across 5 fine-tuning seeds for each value of i and measured on the test set. This table corresponds the bar chart Figure
neural networks, but it is not believed to be the full answer to neural networks in practice. In particular, kernel behavior implies the feature of the neural networks remains unchanged in the overparameterized setting, which is not true in practical pre-training of large models. In contrast,
17

---

Under review as a conference paper at ICLR 2023
k-shot
SST-2
MR
CR
MPQA
Subj
trec
16
(.44)
(.409(o.14) (0.14)
0.419 (.416(0.05)
0.414
0.429 0.601(o.17)
2.023 (0.8588
64
(0.0()
(0.22())22)
(.416
1.588(0.2295
k-shot
MNLI
SNLI
QNLI
RTE
MRPC
QQP
16
0.666
0.658 (0.07)
(0482(0.10)
0.587
0.632
0.458
64
0.859 (0.16)
0.649 (0.03) (.582 0.573
0.857(0.us)
0.559(0.10)
Table 5: Average blement-wise relative distance of K)(GGD) computed the pre-trained and best fine-tuned model. smaller value indicates a higher likelihood that the Fixed Features property of kernel behavior (Definition 3.1) holds when performing fine-tuning. Distances are averaged across seeds for each value of k and measured on the LM-BFF test set (Gao et al. 2021).
SST-2
MR
QNLI
QQP
k-shot
Method
Standard
Prompt
Standard
Prompt
Standard
Prompt
Standard
Prompt
16
SGD-FT
77.6(4.3)
(..)
64.8(5.2)
(3.2
51.60.8)
62.1
57.0
62.1
Adam-FT
78.1 (4.2)
88.3
69.0
(1.3
56.7(3.6)
(3.6
58.5
61.8(4.5)
64
SGD-FT
85.6(1.)
(9.7
83.4
85.6(..)
65.8
72.8(2.2)
64.5(3.7)
69.2
Adam-FT
86.1(1.2)
89.3
83.9
86.0
71.5(4.5)
(3.2
65.0
69.8(0.6)
512
SGD-FT
91.0
91.8:0.4)
(8.7
89.0
81.6(1.5)
82.5().5)
75.8(0.6)
76.0
Adam-FT
(1.4(0.7)
90.7(1.2)
88.4(0.8)
88. 88.6(0.6)
82.2
82.6(1.2)
76.1
75.7
Table 6: Fine-tuning performance in the standard fine-tuning setting, where the contextual embedding of the [CLS] token is used for classification, and the prompt-based fine-tuning setting, where prompt added and the embedding for the [MASX] token used (see Section 3). This table relates
to Figure Tb comparing the SGD fine-tuning results to the more common fine-tuning with Adam.
k-shot
Method
SST-2
MR
CR
QNLI
QQP
16 SGD-FT
89.0(1.5)
83.2((24)
(3.3
(2.1)
62. (2.3)
SGI--LORA FT
(9.6
82.7 (2.0)
92.6(0.8)
57.1
59.8
KCSGD)
(8.3
84.7
(9.5
60.1
58.2
88.1(0.4)
84.9(1.4)
93.0
59.2(3.6)
56.9
SGD-FT
(9.7
85.6(1.1)
(4.3
72.8
69.2
SGD-LoRA FT
(0.0
85.7
(3.7
73.8
68.3
K(GG))
89.2(1.o)
86.4(0.0)
89.8(0.3)
67.3(1.6)
66.4(1.7)
LORA
89.2
85.6(1.6)
93.6(0.4)
66.0
63.9
Table 7: Accuracies of prompt-based SGD FT and prompt-based SGD-LORA FT, along with their
kernel analogs (SGD) and K on a subset of tasks. SGD FT and SGD-LoRA FT achieve comparable performance, and ((GGD) and LoRA also achieve comparable performance to each other. These experiments support Theorem 6.3
Yang & Hu (202T) shows how the initialization variance, multiplier, and learning rate for each parameter can move training from the kernel behavior to the feature learning behavior. They further developed the Maximal Update Parametrization (abbreviated MUP or jP) where every parameter updated maximally (in terms of scaling with width) while keeping the network stable. Yang et al. (2022) then extends yP to Transformers with Adam optimization, and showed empirically that for pre-training of large language models using μP, the optimal hyperparameters remain the same when increasing width. It allows more comprehensive hyperparameter searches on a smaller model and direct transfer of the resulting optimal hyperparameters the larger model, thus providing better performance of pre-training.
In this section, formally describe the right parametrization for kernel behavior in various fine- tuning settings. In general, we consider the overparameterized setting in which the width of the network goes infinity. Additionally, assume that when initializing weight matrix of the model,
18

---

Under review as a conference paper at ICLR 2023
Model size
SST-2
MR
CR
MPQA
Subj
QNLI
RTE
MRPC
QQP
Base (n = 768)
0.32
0.32
0.26
0.38
0.43
0.48
0.48
0.56
0.49
Large (n 1024)
0.32
0.25
0.25
0.40
0.46
0.48
0.47
0.52
0.52
Table 8: We measure X (Definition 3.2) in the prompt-based FT setting for RoBERTa-base and RoBERTa-large. A decrease in the x value when going from RoBERTa-base to RoBERTa-large indicates the task may be solvable in the infinite-wii -iith limit (Definition (.3). We find that for most tasks the eNTK can solve (Table Tb. X decreases model width grows.
each entry of the matrix is drawn from i..d. Gaussian distribution. We use Tensor Programs Yang 2020b) the framework for this setting.
C.1
PRELIMINARIES
Notations Let & E Rd be the input of the network. Let n be the hidden dimension of the network and dout be the output dimension of the network. We define the network as a function of the following form:
fC: (U") (W>), V) = - (U'), (W)),);
where & is the input, U E |nndd are the input weight matrices, W E g=*a are hidden weight matrices, V E Inxxde is the output weight matrix, and h(; (U",,, (W>) ) R" is the input of last layer (readout layer). Fl We write M as the set of weight matrices, i.e., M = [U*) (W*)) U (V). For M E M, let V TE) be the gradient of f w.r.t M at input ç.
in this section. Any non-trivial extension to dn > 1 of results below will be noted along. For any weight matrix M .,, 7M be the multiplier of M, such that M is multiplied by Yas before performing matrix multiplication. Let иaм be the learning rate of the weight m. Let o3 be the variance of entries of m at initialization, so each entry of m drawn (0,0 independently.
For simplicity of the notations,
assume din.
resultts
Because we are considering the infinite-widhh width limit, f(;: (U*),, J., {W'},, V) actually represents series of networks (E; (U*.) [WA*), V"))a>o of the same architecture, but f has a hidden dimension n. When we say model f, it includes not only the information of the architecture, but also TM, T)M AM for every weight matrix M in and the training optimizer of ʃ.
Let Mj be the weight matrix time step of training. If the network is pre-trained, we let M-1 be the weight matrix before pre-training, and Mo be the parameters right after pre-training. Let AM, = MM- MM-1 be the change of each training step. Let ft be the network at step that
ft = f(: {U/ (W/), Vi).
Let &., be the training input and target at step f, and let the loss function at step & be Le(fe(E)) C(f(EE)) v.). Let Xt= = (f((() be the derivative of the loss function.
Jnz0 O Notation For a series of scalar random variables {c")n> and function e N R. say = ((e(n)) if there exist A, B such that for sufficiently large n, Ic E [Ae(n), Be(n) almost surely. For series of vector random variables x = (x" Jnz-0. we say that x is coordinate-wise ((n"), or x = (e((n) if this series of scalar random variables f||x" / is ((e(n)). Similarly for the notation (e(r)), ((e(n), and o(e(n)). For convenience, we assume every e(n) in this section equal n for some a.
Tensor Programs We refer reader to see Section 7 of Yang & Hu 2021 for detailed explanation and full definition of Tensor Programs. Here, we provide a simple overview of Tensor Programs:
Definition C.1 (Definition 7.1 of Yang & Hu 2021)). A Tensor Program is a sequence of "= -vecorr and RR-caaass inductively generated via one of the following ways from an initial set C of random scalars, V of random R. vectors, and set W of random Ra*= matrices.
WWe are able to describe transformers (without weight tying) in the definition. The bias can be regarded as input weights assuming there is coordinate in that is always
19

---

Under review as a conference paper at ICLR 2023
MatMul
Given W R and R", we can generate Wz E R or W Z R".
Nonlin Given R x R + R. previous scalar 0 Or E R and vector x² s',...,,** R", we can generate a new vector
6(2 x² 0 01) R"
where d(-;0)..... 0r) applies coordinate-wise to each "a-slice (#a....,
Moment Given the same setup as above, we can also generate a new scalar
p Fa .1......00 0t) E IR
Yang (2019 2020a); Yang & Littwin (2021);
:; Yang al. (2022) show that Tensor Programs express the computation, SGD/Adam optimization, and the kernel of practically any architecture.
The key result of the Tensor Programs is that
represent the coordinates of any vector in the
Tensor Program with a random variable Z", and represent any scalar with a deterministic scalar Ó.
There is way define all i and Z" correspond the Tensor Program (ef. Definition 7.3 in Yang Hu (021) ) and the Master Theorem of the Tensor Program shows that > 0 when -> 0o (cf. Theorem 7.4 in Yang & Hu )).
Although it is in general hard to compute 2H and 0, it allows us to reason about the scales of vectors in the training of network.
Assumptions Related to Tensor Programs. Since are studying the infinite width limit and using Tensor Programs as our framework, there are some mild assumptions that we need in order to apply Tensor Programs and results in Yang & Hu (2021).
Assumption C.2. We assume the nework f satisfies the following
a) The forward pass of the infinite-widie limit can be written as Tensor Programs.
b) The hidden vectors have ((1) coordinates at initialization.
c) The hidden vectors have ((1) coordinates during training.
d) For any training scheme and any constant and any input y, fe(E) = O(1)
e) There exist a training scheme and some constant and input & such that fe(E) - fo(g) =
(1))
f) The activation function of tanh or o-gelu for small enough a (so it approximates RcLU), where
a-gelu(x) 1 rerf(o 22) c 2,2
+
2VT
Furthermore, we have one assumption on SignGD:
g) SignGD approximated the sign function being replaced with E-sign for small enough when updating parameters, where e-iig((r) = Φott is smoothed version of sign. We assume using different when computing the sign of M that for Vur match the maximum scale of V M..-
b). c), d) and e) in Assumption C.2 together recover the definition of nontrivial stable network in Yang & Hu (2021). b) and c) ensure that the mrcaactvatiinss in the network are not too large, that the activation function like tanh are not trivialized to always output = b) ensures that the pre-activations small at initialization, the activation function is not trivialized to its first-order Taylor expansion. d) ensures the network output is bounded. e) ensures that the network is frozen during training.
in the network
Training scheme sequence of training examples {(E,, v(E)), and loss function ((f.()).
20

---

Under review as a conference paper at ICLR 2023
assures all non-linear functions that appear in the Tensor Programs is pseudo-Lipschitz, which is required for the Master Theorem of Tensor Programs. g) also assures that e-sign is not trivialize 0 sign when V f * ((1).
and in Assumption C.2
C.2 SIGNGD KERNEL DERIVATION
Definition C.3 (Formal Definition Kernel Behavior). We say that this network training process demonstrates kernel behavior if the following properties satisfied.
1. Linearization: The change of the network can be approximated by its first order Taylor expansion, i.e.,
fr(E) fi-1(5)
AMe
lim
lim 12-+00
70+00
Xt
Xt
MEM
2. Fixed Features: The gradients at step t are approximately the same as before training, i.e.,
(M)((E) V M 0.
vM E M, lim n-000
maxe
Note that we define I Linearization with both LHS and RHS divided by so it meaningful for the case of Xt (1). We do the same thing in the following theorem.
Theorem C.4 (SignGD Kernel), If SignGD training of demonstrates kernel behavior, then under
Assumption C.2
f((E)
fo-1(5)
lim
lim n-000
Σ
-IM
12-000
Xt
Note 7/M = 7), the RHS the equation above equals to
バy< fo( s-sign fo(E)) SignA Et),
where the approximation comes from the difference between c-sign and sign.
Proof. By the update rule of SignGD, AM.
( (V suffices prove x
MM
( (ttpaa/))).aa/aaaoooeeaaaaao/aoa
when n + DO.
Since
7LM (V cesg(()) - 7M (VM (vi
7M M.(E) V (ggg +
(4)
c-sign for
(5)
7M Mf()(
Mo c-ggn (.. - s-ginn V fo())))
only need prove Equations (4 to (6) all when
Let g" = arg max.- M be the input maximum gradient scale, then by Fixed Features, have
V M Se(E) - V fo(E)ll o(1).
M
Since e-sign(æ) e-sign(y) -11-y//6.
[c-sign(V (vigg))))))iii(Vx((\(\()))) (MM(((5) - Vmfo())l2/c.
21

---

Under review as a conference paper at ICLR 2023
Combined with (o @(e) (N is the number of entries of M, this is by g) of Assumption
C.2
we have
le-sign V v. - vV
|| e-sign V fo(E"))
s IV V fo
by
|essign fo
(VM)(( ao
fo(E")l
< Vll=sign(VMM/b(((3)1////
VV fo
@(1) = o
(o(
By d) in Assumption
C.2 and consider the training scheme that sets & 8 and the loss function L
so X1 = @(1), then
J (1(5)) So(5") 1IM MV (V..sig((MMm((("))))) 0(1)
X1
And is easy to see that (V V e-sign( (i and [ has the same scale.
Now it suffices to prove Equations all O when - co. For Equation
to 6 divided nM V Val are
MM (V M (E) ersi
7M V V
V N
V - vo) o(1)
| \
Similarly, for Equation 5
JjM (V (V e-sigesssvnvv.. (-si(( e-sign( V Mo(e))))
nM follV
l--iign Vaiil =()),
[e-sign(VM:.o((=)))
and for Equation 6
7M (V Mf)(E) V (nff(EE) V mff(()) c-sign( (M
7M V M Valn
lesiggn VMf((E) Var [VMff(EE VMfolE)lla o(1).
|e-sigu(V
C.3 PROMPT-BASED FINE-TUNING
In this section, we prove that prompt-based fine-tuning exhibits kernel behavior with the assumption of Xt o(1). Prompt-based fine-tuning trained directly on the pre-trained network without substituting or adding any parameters. Without our assumption, is obvious that the behavior of fine-tuning and pre- me-training is the same from the perspective the Tensor Programs.
Theorem C.S. If the downstream task is solvable for nelwork S, that is, for any I, o(1), then under Assumption C.2 the fine-tuning of exhibits kernel behavior (Definition
Below provide proof that is heavily based Tensor Programs and the analysis in Yang & Hu (202T) For readers who are not familiar with Tensor Programs, we provide intuitive examples in the next few subsections, where focus three-layer linear network parameterized with j.P
22

---

Under review as a conference paper at ICLR 2023
Proof. We first prove the theorem under the assumption that the network is multilayer perceptron and the optimizer is SGD, which is the same setting as Yang & Hu (202I). Then we will extend to more general cases.
Consider the following L-hidden-layer perceptron:
(E) = US
and and
=()==((1())) (E) w 20 (e(), for L-...,
(()) = Vz'((S)
Following Yang & Hu (2021), we let the learning rate for every parameter equal to yn Let W = U and W/++1= V. and for / = ,,..., 1, we parametrize W as W = nw' for actual trainable parameter w', and we initialize each coordinate w' i.i.d. from (0,, of). The setting covers all parameterizations based on Lemma C.6 For convenience, we assume " = n and 0r = n -b, Without loss of generality, we further assume that X O(n (rather than Xe o(1) only).
By Theorem 3.3 of Yang & Hu (202I) stable network implies
ف min(a1+1 + 1l+1, 20 + c) + e 1 min [ai + I(1 = 1)]
min
Also by Theorem 3.8 of Yang & Hu (202I). ). for nontrivial stable network (included in Assumption C.2 if r > 0 then there exists kernel K. such that
fi+1(5) = f((E) - MxAXC(E Ec).
which is very close to our definition of kernel behavior. fact, we will prove that they are equivalent in the fine-tuning case.
Since X = (n d) for fine-tuning, it is equivalent to set the learning rate to x -e-d and replace Xt with nd Xt- Formally, stage, 2 0 (so could demonstrate feature learning or kernel behavior); at the fine-tuning stage, c increased to _ c + > c. thus, the corresponding is increased to be strictly greater than 0. Therefore, suggests kernel behavior with following caveats.
considering the following training scheme: the pre-training
Caveat 1: do pre-training affect the result? The answer is effectively NO. First of all, the scale of the update on W h², x' and f are all multiplied by n when switching from the pre-training stage (1m learning rate) to the fine-tuning stage(mm -e-d learning rate). The scales are exactly the same training from scratch with im c-d learning rate except by,+1 needs to be changed to
8 L+1 e min(brti. a + c). Note this change of b does not affect the fact that is updated to r' ~
Caveat 2: does r' formally implics our definition kernel behavior (Definition C.3 The answer is YES. We first prove Fixed Features in Definition C.3 The gradient of matrix W equal to outer product between V (gradient w.r.t. h') and x¹ Let dh, be the normalized gradient w.nt. h' at step (so dh: (1)), and z be the xf at step (af ((1) without normalization). It suffices to prove dh: dhg O(1) and x. o = o(1). The later was proved by Proposition H.27 of Yang & Hu 2021). To prove dh; dhj O(1), we let de! be the the normalized gradient w.r.t. xf at step f, and compute the scale of dh; dhi_, and dæf dri_, inductively from L to / 1. We obtain that they both has the same scale of
n min(2ar -C- 11 +o.+tte 1-min 20m) < min((.r")
the inequality is because 12 S aLT1 + and < aLTI bL+1 +
mink 2am-
Second, we prove Lincarization in Definition C We need to first make very slight modification to the Tensor Program in Yang & Hu (2021). , that is, changing the computation of f.(E) - fc-1(E) to
n (f(((( ft-1(E)). By Theorem H.32 of Yang & Hu (2021) and its definition of Σ, we can show
23

---

Under review as a conference paper at ICLR 2023
that
L+1
lim n (f((E)) - fi-1(5)) = lim 10000
mn
a-d
(Vwifi-1(E), (wr)f-1(()))
n-000
AW?
lim
V (wifr-1()) n-d
From SGD to SignGD. Since sign(xy) sign(x) sign(y), the update of matrix W can still be written as outer product of two vectors, i.e., AW; = ryn - sign(V [V & sign(>(=)) After applying sign, the scale of vector changes. If the parametrization is the same, the scales of vectors using SignGD will be different from those using SGD. This can be easily resolved by changing learning rates for each parameter, so the scaling change brought by sign is corrected. Furthermore, as mentioned Assumption C.2) we need to approximate sign by a smoothed version e-sign so the Master Theorem of Tensor Programs can still stands.
Extension to universal architecture. The theorem is correct for any network whose first forward pass can be written as Tensor Programs. Given this condition. the forward pass, backward pass and kernel of any steps can be written as Tensor Programs (Yng 2020a b). To analyse the scaling of the Tensor Program will need the following steps:
1. Extension to general computation graph. We can still inductive reason about the scale of preactivations and activations by the topological order of the computation graph; and similarly reason about the gradient by the reverse topological order.
2. Extension to weight sharing. We may use weights multiple times in forward pass. The preactivations, activations and their gradients will not be affected. Only the update weight is now of several vector outer product depending on the number of occurrence the weight.
C.4 μP FOR SGD AND SIGNGD
In the following subsections, we provide more intuitions to Theorem C.5 where kernel behavior exhibits. Although care about all type pre-trained models, we are mostly interested in models with feature learning behavior. For pre-trained models with kernel behavior, it is obvious that fine-tuning with the same settings as pre-training (corresponds Prompt-basee based FT) will lead it to the kernel behavior. Furthermore, Theorem H.17 of Yang & Hu (2021) proved that if the last layer is replaced with freshly initialized layer (corresponds to Standard FT), fine-tuning from pre-trained models with kernel behavior is the same as training the downstream task from scratch.
and some other situations
Among all the models with feature learning behavior, jaP is the most special one where each parameter itself (except the last layer) can push the model to learn feature. Therefore, we use μP example to give a proof of intuitive understanding.
The formulation of μP contains three sets of hyperparametens: initial variance of M, multiplier of M and learning rate of M for M E {U*}, U [W*}, U (V). However, even if we restrict these three hyprraarrseters to be in the form of n² μP is not unique, because there is one degree of freedom for each weight according to the following lemma.
Lemma C.6 (Lemma J.1 of Yang et al. (2022)). Consider a weight matrix M with learning rate C, initialized as M ~ (0, B²), and with a multiplier A. Then for any 7 > 0, fe(E) stays fixed for all t and & if we set
A7, B/T, C C/22 2 if training with SGD.
< Ay, B/7, + C/7 training with Adam.
Note the conclusion about Adam in Lemma C.6lalso also extends to SignGI).
24

---

Under review as a conference paper at ICLR 2023
With Lemma C.6
always set the multiplier of any weight matrix M to be which leave us only the initialization variance Js and learning rate M).- Furthermore, in terms the scale at initialization and the scale of updates, μPP for SGD and SignGD difference would be learning rate. We provide details in Table (recall M-1 is the weight M at initialization of pre-training. AM Mo M-1 is the overall change of weight in pre-training).
entirely the same. The only
Since we have different learning rate for 7JM, the kernel that we care is defined as
<(EEE") = Σ nM
MEM
where @ is identity if the algorithm is SGD, é sign if the algorithm is SignGD, And we want to prove the dynamic the network follows
fe(5) fE-1(E)
KC(E when - OC.
Xt
C.5 PROMPT-BASED FINE- TUNING: LINEAR EXAMPLE
As intuitive example, consider three-layer linear network
f(E: U, W, W. V) V WUÇ.
For simplicity, we train the network with SGD, and freeze V m 0. Then we have Vuf W VE and Vw V(UE) We assume (E. (')l > 0 for any E.
Zero step (Pre-training) We model the pre-training of f as one step of training with Xo = ((1). Then have AU -w Xo W independent from V. have W-,V V ((1/n), thus AU, @(1) matching Table 9 On the other hand, it is obvious that AWo = ((1/n) because V = 011nn) and U = @(1,, also matching T
V and AW, - VU Since W is
Then the function is now
fu(E) = v TW-111 (Wo)
7w (WV====) (U-L - V ((o,)))))))
=VTW_U-_IE TU VI
mmxollV U-1E)
+ x " 1-1--1E.
It is not difficult to that W E), mw X {) are all ((1). Unfortunately, here VWW-U-i 0 in the infinite-width limit, but if we train one more step, is easy to see that all four terms of fo is 9(1).
VV U-1E). and
First step At the first step of fintuning, we have AU = W VET and AW1 = -w (U T. Then the function is now
f1 V (Wo AW1) + AUIE.
and
fi(E) fo(E) V'T AWT + +VT V AWGAUIE.
(8)
coordinate- wise scale
M U
M W
M
M_1
9(1)
((1/Vn)
O(1nn)
AMo
O(1)
((1/n)
O(1nn)
MJ for SGD
O(n)
O(1)
(1/n
IIM for sigmGD/Adam
+(1)
O(1/n)
((1nn)
Table 9: Scales of initialization, update and learning rate for jP in pre-training.
25

---

Under review as a conference paper at ICLR 2023
Note that the of the first and second terms is exactly -x)
Plug in AW1 = 7J X1 " into the first term of eq.
8
V AWW - ((UOL)) Uos = 0(x),
because
(Uot))"ut (U-151 + AUGE)) (U-1E (U-1E + (U--11,U-1$)) - (O) - = (n)).
Plug in AU1 = X1 VEI into the second term of eq. we have
V V WoAUE mu V WoW. V 0(x)
because
V W.W. 1[ + AW)) VW AW,) -
|W_ + 7/ 1 2nw xol O(1nn)
The third term of eq. (8) equals
7UV V(U&E) W. V.
~ (G. = +(x))
Therefore, fi(E)-fo(E) K(E,E1)
X1
Second step At the second step of fine-tuning, we have AU2 = - WT Veg and AW2 -W EU) and
(2(5) (5) = V AW2UIS + V W2 V ATWAUU..
(9)
Assuming X2 and XI share the same order, then when - X0,
f2(5) - fi( V AW2U1E/Xe + V WT
X2
V/ V(UiS) U.S V V W.W.VE. V(UOE TU - WoW, W, VE - KC)
VV
tth step Same the second step by noting AU, AW, always have smaller order than AUg and AWo-
C.6 STANDARD FINE-TUNING
In standard fine-tuning, V is replaced with randomly initialized matrix V". That is, the model the step t of fine-tuning is fz(E) = f(: (U]),, (W/), h, V?). We set Vi ~ NO, o² [Txdoau): ). which has larger scale than V in pP. In this section, will prove that this standard fine-tuning has kernel behavior.
We still consider three-layer linear network
f(;: U. W. W. V WUE
where V' E Rax1 (i.e., dout = 1) and we freeze V during the fine-tuning so it is not trained. Then we have Vuf = V'E and Vwf= VW
coordinate-win wise scale
M= U
M W
=
M= V
M
O(1)
((1/vn)
O(1/n)
A AM
O(1)
O(1/n)
((1/n)
((1/vn)
A M,
((///))
((/nn/))
((1nn)
MMM for SGD
(1)
(11nn)
O(1/n)
Mas for SignGD/Adam
(1/v/n)
((1nn/n)
O(1/n)
Table 10: Scales for standard fine-tuning w.r.t.
26

---

Under review as a conference paper at ICLR 2023
First step At the first step of fintuning, have AU1 Tw xwwxie (V661) Then the function is now
- V's. and AW1
(i(5) V"T (Wo + AW
and
f1(5) - fo(E) = V'T AW V'T WoAUEE+ AW1AUE
(10)
Note the first order and the second order term is exactly -xI -XIK(
Plug in AW, TW X (U(EE) into the first term of eq. (0)
V AW = TW X1 V ('UGi,) T Uoo = ((1) Xw
since (UoE1) UoE = (n 6
Plug in AU1 -IUX W V's into the second term of eq.
10 we have (assuming |
V" WoAUis qu xu V V v's. (1))
because V' WoW, )()
It is easy to verify that the third term is O(L/Vn). Therefore, fi(E) - converges to its first-order Talyer expansion when n oc.
Second step At the second step of fine-tuning, we have AU2 W. and AW2
-I V (U) and
f2(E) fi(E) = V AW2U1E V"T WIAUE + V' AWGAUSE.
(11)
We remove all terms of o(),, then
f2(E) (E) x V AWAUIE + V WIAUS5 = 'I UnE + - W.W V'
= V'( Uog + + W. V'
- E2)
Same the second step by noting AU, AW, always have smaller order than AU and AWo.
(th step
C.7 PROMPT-BASED FINE- TUNING WITH PROJECTION
If xfx=0(1 in Prompt-based FT, we need some modification in order to push the model to kernel regime, otherwise, the network will stay in feature learning regime or it will not learn (if we decrease learning rate). In particular, we add randomized projection matrix before the last layer so the function is now equal to
(e(E) = V. WiN(e: (U)). MC ((:()))..())
(E:(U))..
where V/d & W.TVV VG, and W/ ~ NO0 o² Iaxx). Now we consider the linear example again where
f(E:U, W. v =
If we freeze both V and W" during the fine-tuning and let dout 1, then it is equivalent to the linear example Appendix C.6 with V A [ (note ||V 1/3 = (11nn).
It is clear that (Uo$1) UoC = O(n). (UoE))TUsE T o(n) would mean there is some weird correlation.
't easy to see that E/V W,W, V1] = @(1) and Var(V"T W. v') = W.]WW 1 By
Yang & Hul 2021 W,,,, 1|//n2 o
27

---

Under review as a conference paper at ICLR 2023
D SUBSPACE-BASER FINE-TUNING METHODS
We start by restating the Johnson-Lindensttruue lemma, which preserves inner products under random projection.
Lemma D.1 Johnson--Lnndenstruue
Let u, v Re such that ||u|| S and ||l|| < Choose
k log N/e,, where N is the number of datapoints. Let h Ax, where A E Rexd with each entry sampled i..d. from (0, 1) or (1, Then,
Pr[[u- h(u) h(v) 5 cxp(-
Lemma D.2 (Norm Preservation (aabssoonLindenstruus))) Let R and assume the entries of A E IR. kxn are sampled i..d. from (0, Then,
1 k
Pr -1--6h ||/11
Ar
<1+ -1-exp(-(e- -e))/4
Let u, U. IR such that |u||,1|l1 S B. Assume Ku, o) 2 ellai|llee (i.e., u and v are not orthogonal). Choose k = 20 log N/e², where N is the number of datapoints. Let h(x) VI Ar, where A E Rkkxn with each entry sampled i.i.d. from N(0, 1). Then,
Lemma D.3 (Inner Product Preservation). Let
|||||
(u, v) (M(u),(())
Pr
2 e/c s 22xxp (//4
Ku,
Proof. From the norm preservation in Lemma D.2 know that with probability at least 1 2 2exp(-(e2 k/44,
(1 <) l + w112 < [M(u [)]+++)] ~ (1 e) [| m|/
(1 e) lu - o112 [ <) e) ul
So, we can write
4h(u) h(v) = + | r)ll
2 (1 - +) - (1+ w/|
= 4u 2e([|u|| 11|l²)
2 4u 4B
This implies (u, v) (h(w), < B²c So.
(u, v) (h(u), h(v))
BºC
c/c
(uu,
cB2
The other side of the double-sided bound can be derived analogously.
We can now look at LoRA for a simple fully connected layer.
coordinate- wise scale
M = U
M Wi O(1/Vn)
M =V V
M = -
M_1
O(1)
(1/nn)
AMo
6(1)
O(1/n)
(11nn)
A A,,
(1///))
((/nu//)
MM for SGD
((1)
((1/n)
1IM for SignGD/Adam
((/Vv))
O(1/n//)
Table 11: Scales for prompt-based fine-tuning with projection layer w.r.t.
28

---

Under review as a conference paper at ICLR 2023
Lemma D.4 (LoRA SGD Kernel), Leth Wx + BAx defined in the paper, where IR", W E R , B Rm and A E R2x" with lo < n. B is initialized to and A is initialized with i.i.d mean-O Gaussian samples. SGD Training with LoRA (i.e.. fixing W and allowing A and B to be updated) yields the kernel KLLRA whereas full FT with SGD yields the kernel K::
LORA = -HHH
@ OXA AX
KCSGG) = dHAH O (XX))
where dH E NR.Nxm has dh((x;) in the ith row and X E NR has in the ith row.
Proof. We start by noting the well-known fact that dW = dh @ æ, where dh is the gradient to and is the cross product. Thus, K dHdH T o (XXT), In the LoRA setting, dA 0 and dB = dh 8 Ax. Because we are in the kernel setting, = 0 and thus, dA = 0, throughout training. So,
(i, = (dB()), dB()) = (dh(i), (lb)))Axr; Axy) where (a) denotes the gradient to B when given example a. Analogous reasoning yields (i.j) dh())
Theorem D.5 (ARA likely not far from K(SSDD), Let K( E RNNN and KISGDD E NNN be defined as in Lemma D.4 Additionally, assume that (xa(E), ax()))))) 2 cl[=(())|==(()) for some c > 0 for all pairs E,E' in the downstream dataset (i.e., no two gradients are not orthogonal). Then, for any i.j [N].
K (i,j) (isGDi(i,)) (isGDi(i.
Pr
2 c/c
Proof. Note that
K (GGD) (1,5) - K0 ((SGD) (C)S)
(dh((), dh(T))((Ax; Auy)) (1ri, xz)) (dh(i), dh(i))
((SGG)(),
(Axs, Ax,) - (x,,
(0,, 2,)
The rest of the proof follows from Lemma D.3
The statement for IntrinsieDimmnsioon FT can be derived by applying Sobnson-Lindenstrauss directly to the gradient vectors.
29


Reviews:
[{"review_id": "itkSSaeHDy", "reviewer": "Reviewer_NNgP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies behavior of NTK on fine-tuning MLM models. Supposedly, pre-trained model is no longer iid random in parameters but still it can achieve good performance on CV tasks per previous literature. In this paper, authors extended the studies to NLP models with both regular fine-tuning and prompt-based learning cases. Authors also provide a modified eNTK version when using ADAM but not SGD. Last, authors tried to propose some hypothesis on why such model on eNTK behaves so.", "summary_of_the_review": "Overall, I like the angles and the topic of this paper. But somehow I think it's a bit disappointing after reading it as it's not providing much insight. Rather, it's more of a mixed of empirical evidence and some not that useful theoretical results. I do believe we should encourage this type of work so overall I am more positive, but there are some confusions and I hope authors could explain it.\n\nOverall, I think the gist of the paper is a bit unclear. Mentioned too many points but I can't find the direct link between the claim and analysis. You listed many contributions in bullet points so I guess it's easier to go from there.\n\n- Does the eNTK method directly work for NLP tasks? In Section 5, we show that eNTK does not\nwork well for standard FT but it achieves performance comparable to prompt-based FT (Schick\n& Schutze, 2021; Gao et al., 2021) on a majority of downstream tasks that we evaluate. ¨\n\n- So I think this is just an empirical analysis. Somehow just an extension of domain from wei el. al. \nDo you agree with this? If so, can you highlight this is more of an empirical results. in addition,  can\nyou also greatly re-organize your paper to make contribution of empirical results and theoretical analysis clear?\n\n• The NTK theory was developed for gradient descent, whereas LM fine-tuning often uses\nAdam (Kingma & Ba, 2014) as a standard practice (Devlin et al., 2019). Can the above method\nbe extended to give insight into the effect of different optimization algorithms? In Section 4,\nwe derive a new asymmetric “kernel” formula (Definition 4.2) that describes the dynamics of\nshort-term training with Adam and shows that the corresponding eNTK achieves comparable\nperformance as FT using Adam (Table 1).\n\n- Here you use yet another approximation of Adam, so I don't see the purpose of it. Why does that \nmatter to analyze yet another version of eNTK  just due to a different gradient method? \nIn particular, Adam results in Table 1 doesn't really outperform SGD version eNTK musch. Especially,\nfor TREC ADAM result is much worse than eNTK one. What's the explanation? \n\n• Is the eNTK regression method simply an alternate model that also happens to have good\nperformance on the classification task, or is it actually a near-equivalent description of how\nparameters of the original model evolve during fine-tuning? Section 5.3 shows that for tasks that\nthe eNTK can solve, the fine-tuning of the pre-trained model does exhibit behavior consistent\nwith the dynamics of training kernel classifiers. See Definition 3.1.\n\n- I think this could be a real contribution. But somehow I feel not consistent as fine-tuning + eNTK \nseems not to work but here you showed its empirical performance justified the assumptions/desired\nbehavior of NTK in 3.1 ? It's kind of contradicting. \n\n• If the eNTK can solve NLP tasks, then does the eNTK give insight into phenomena such as why\nprompting can improve performance and how parameter-efficient fine-tuning methods affect\noptimization? Figure 1 demonstrates that adding a prompt is necessary for the eNTK to solve the\ntask, and in Section 6, we apply the kernel lens to provide a possible explanation for the empirical\nsuccess of subspace-based fine-tuning methods (Hu et al., 2021; Aghajanyan et al., 2021).\n\n- Sorry I don't quite get why prompting can improve performance is related to subspace-based fine-tuning methods?\nI think the connection is not very obvious to me. I was feeling strange when I read your section 6. I am not sure\nwhy you want to analyze it. And I felt the connection to other part of paper is too weak such that it's another paper.\n\nWe conclude by proposing a rigorous mechanism through which fine-tuning of complex architectures\n(e.g., Transformers (Vaswani et al., 2017)) with prompts can exhibit kernel behavior. This is done\nin context of networks whose width goes to infinity, but unlike standard infinite-width NTK theory\nit allows a non-random initialization that is the result of pretraining. This result uses the Tensor\nPrograms framework (Yang, 2019; 2020a;b; Yang & Littwin, 2021; Yang & Hu, 2021).\n\n- I actually feel this part of analysis is the most pertinent part regarding your motivations. Why don't you put it in the front? But I am not sure the correctness and framework as I didn't know Tensor Programs before.\n\n", "strength_and_weaknesses": "Strength:\n\n- An interesting observation and continuation of previous work.\n- Provide studies over various related topics with mixed empirical and theoretical analysis.\n\nWeakness:\n\nMain contribution seems to just be combining two major previous results and somehow it's not providing enough insight.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LssJwCXqcFO", "reviewer": "Reviewer_rpG6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests that the success of language model fine-tuning (or with prompt) can be explained by neural tangent kernel (NTK), a tool that describes optimization behavior of infinitely wide neural networks. To bridge NTK with LM fine-tuning, the paper tries to justify that using NTK after pre-training phase (instead of initializing from lazy regime) is reasonable, and also derived empirical NTK for adam optimizer. The paper found empirical evidence that NTK behaves similarly to the original optimization methods, and also provided a theorem that shows prompt-based fine-tuning can exhibit kernel behavior", "summary_of_the_review": "My feeling is a bit mixed for this paper. My current rating would be marginally below the threshold given that there are several issues worth clarifying. Regarding Weakness #4, I think different people may have different feeling on how important this is. I am relatively close to the median of that spectrum, but I can definitely see researchers having strong opinions from both sides. ", "strength_and_weaknesses": "Strength: \nThe paper tries to explain language model fine-tuning using NTK and cleverly uses the fact that fine-tuning takes few samples (therefore one can compute eNTK in a feasible way). \n\nWeaknesses:\nOverall, the paper is trying to bridge theory and practice. But there are a few concerns:\n1. While some of the empirical results are good, some observations are quite mixed. For example, RTE linearization does not perform well for k=512 while eNTK still performs ok -- this seems confusing to me. \n2. Section 4 is unclear to me, the paper mentioned \"we model Adam updates with SignGD\", why can you replace Adam with SignGD and is the derived kernel exactly following Adam or is just an approximation of Adam (if later is the case, why not just approximate Adam with SGD)?\n3. The theory section (Section 7): I don't see how it is related to prompt-based fine-tuning but not standard fine-tuning. The paper mentioned \"The theory focuses on the prompt-based setting since we did not find empirical evidence of kernel behavior in the standard setting\" (I hope theory can advance empirical work but maybe now the paradigm is shifted), this is fine, but which assumption in Theorem 7.3 is specific to prompt such that the final conclusion is \"prompt-based FT of f will exhibit kernel behavior\"?\n4. The bigger question is whether NTK is THE theory or just A theory. Obviously a lot of practices in LM training or FT do not match with NTK behavior (e.g., learning rate warmup), therefore assuming NTK is THE theory and develop reasoning that tries to explain FT is a bit insufficient, so I think this is a weakness of the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H5FQeoSh0PV", "reviewer": "Reviewer_Ftva", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the underlying reason behind why fine tuning pre-trained language models works well. The paper shows that in some cases, those fine tuned models can be described using neural tangent kernel. In those cases, the kernel view provide explanation of the fine-tuning success. \n", "summary_of_the_review": "The paper presents several theoretical and experimental results, that I don't feel creates a coherent bottom line. In addition, It is hard to evaluate the quality of those results, without reading the appendix (which I didn't read). Thus, I cannot recommend accepting the paper.", "strength_and_weaknesses": "Strength:\nThe paper tackle one of the most important questions in machine learning: understanding the reason behind the the performance of fine-tuned models, and in particular, when it should works.\n\nWeakness\nI found it hard to follow the paper's arguments. see section below for more details. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject