The Impact of Initialization on LoRA Finetuning Dynamics
Authors: Soufiane Hayou
Nikhil Ghosh
Bin Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Finetuning; LoRA; Large Language Models
Primary area: other
TL;DR: Initialization of the adapter weights has crucial impact on LoRA learning dynamics

Abstract: In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021). Essentially, to start from the pretrained model, one can either initialize $B$ to zero and $A$ to random, or vice-versa. In both cases, the product $BA$ is equal to zero at initialization, which makes finetuning starts from the pretrained model. These two initialization schemes are seemingly similar. They should in-principle yield the same performance and share the same optimal learning rate. We demonstrate that this is an *incorrect intuition* and that the first scheme (of initializing $B$ to zero and $A$ to random) on average in our experiments yields better performance compared to the other scheme. Our theoretical analysis shows that the reason behind this might be that the first initialization allows the use of larger learning rates (without causing output instability) compared to the second initialization, resulting in more efficient learning of the first scheme. We validate our results with extensive experiments on LLMs.

The Impact of Initialization on LoRA Finetuning Dynamics
Soufiane Hayou
Nikhil Ghosh
Bin Yu Dept of Statistics UC Berkeley binyu@berkeley.com edu
Simons Institute
Dept of Statistics
UC Berkeley hayou@berkeley.edd
UC Berkeley nikhil edu
Abstract
In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally introduced in Hu et [19]. Essentially, to start from the pretrained model as initialization for finetuning, one can either initialize B to zero and A PEFT package), or vice-versa. In both cases, the product BA is equal to zero at initialization, which makes finetuning starts from the pretrained model. These two initialization schemes are seemingly sim- ilar. They should in in-principle yield the same performance and share the same optimal learning rate. We demonstrate that this is an incorrect intuition and that the first scheme (initializing B to zero and A to random) average yields better performance compared to the other scheme. Our theoretical analysis shows that the reason behind this might be that the first initialization allows the use of larger learning rates (without causing output instability) compared to the second initial- ization, resulting in more efficient learning of the first scheme. We validate results with extensive experiments on LLMs.
to random (default initialization
Introduction
The vetrain-fineture paradigm (e.g., [7, 9I) has revolutionized deep learning, replacing task-specific models trained from scratch with finetuning of pretrained base models. These base models, trained on generic unsupervised objectives, learn powerful features that can be rapidly adapted to down- stream tasks. The most effective models are consistently the largest ones [14, 25], with state-of- the-art models reaching hundreds of billions of parameters. While many such models are openly available (e.g., Llama by Touvron et al. [38]), full finetuning remains computationally prohibitive for most practitioners. This has led to parameter-eeffcient finetuning methods, including adapters [11], prompt tuning [20], and (LA)3 [24].
Low Rank Adaptation (LoRA) [19] has emerged as a leading parameter-efficient method, training only low-rank adapter matrices added to pretrained weights, typically using Adam [3]. LoRA often matches or exceeds full-finetunin performance [35, 39], though it may underperform on complex generation tasks. While prior work has examined rank [31] and learning rate [44] hyperparame- ters, initialization schemes remain understudied. This work provides experimental and theoretical justification for choosing between seemingly equivalent initialization approaches.
In standard LoRA training, one of the two LoRA matrices is initialized with random values and the other is initialized to zero (see Section 2). Recently, in Meng et al. [48] the authors proposed an al- ternative initialization scheme to LoRA which uses the top singular vectors of the pretrained weights opposed to random initialization and showed improved training several tasks. To further im- prove LoRA training with quantization, Li et al. [34] introduced a new method called LoftQ for computing better initialization for quantized training [27]. However, to the best of our knowledge,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

there has not been any study concerning the random initialization in vanilla LoRA. Specifically, it is clear from prior work which of the two LoRA matrices should be initialized to be zero. Empirical results by Zhu et al. [50] suggested that the two initialization schemes mentioned above yield simi- lar performance, but it is not clear if the learning rate was well-tuned for each initialization scheme. Our findings suggest that these two initialization schemes lead to fundamentally different finetuning dynamics, and that one of these schemes generally yields better result compared to the other.
LoRA Variations. Beyond alter- ing the LoRA initialization scheme, there have been series of works which try to address limitations of vanilla LoRA using different varia- tions. To further reduce the num- ber of trainable parameters, LoRA- FA [42] freezes the A matrix which leads to small performance loss while reducing memory consumption by up to 1.4x. The performance of
A ~ 110.1/1
4 -0
InnjAl:
B=
Ini[B]:
1f M-M00,1/4
Pritrined Weights 2M
M/mn/A Minnter
Witth
this training scheme is also investi- gated in Zhu al. [50]. VeRA [33] freezes random weight tied adapters and learns vector scalings of the in- ternal adapter activations. LoRA-XS [43] initializes the A and B matri- ces using the SVD of the pretrained weights and trains a low-rank update of the form BRA where R is train- able r X r matrix and B, are fixed. NOLA [32] parametrizes the adapter matrices to be linear combinations of frozen random matrices and optimizes the linear coefficients of the mixtures. VB-LORA [46] shares adapter parameters using a global vector bank. In order to improve the learning ability for more chal- lenging finetuning tasks, Kalajdzievski [31] proposes scaling rule for the scalar adapter multiplier to unlock increased gains with higher adapter ranks. MoRA [45] learns high-rank updates while still preserving parameter efficiency by applying hand-designed compress and decompress opera- tions before and after a trainable adapter matrix. DoRA [47] decomposes the pretrained weight into magnitude and direction components to allow for better training dynamics.
Elaetuaiin
Move Effisient
Saabilty with subepriinl Fenture Leariing
Foarure Learning witk some inviailly
InH|A] generallty leads to better formance
Figure I: Summary of our contributions in this paper: a de- scription of the difference between the finetuning dynamics when LoRA weights A and B are initialized with Init [A] or Init[B].
Contributions. We study the impact of Initialization in LoRA through a theory of large width for neural networks. The core approach is to take the width of neural network to infinity and determine how the behavior of the limit depends on the choice of the hyperparameters, such as the learning rate and initialization. This approach allows to derive principled scaling choices for these hyperparameters such that desired properties (e.g. stable feature learning) are achieved as the network size grows (see Appendix A.2 for more details). Examples of the infinite-width limit include works on initialization (e.g. He et al. [4]), and training dynamics (e.g. [21]). Examples for the depth limit include initialization strategies [6. 10, 30), and depth scaling (see e.g. [18, 23, 28, 29, 37, 41]). A similar strategy was used to derive scaling rules for the LoRA learning rate in Hayou et al. [44] (LoRA+) that concluded that the learning rates for different LoRA matrices should be scaled differently to ensure optimal feature learning. In this work we use the same approach to provide a systematic comparison between two different random initialization schemes for vanilla LoRA finetuning (using the same learning rate for the A and B matrices). Using the notation Init(A] [A] to refer to the case where is initialized to random and B to zero (as [19]) and Init [B] for the opposite, we show that Init [A] and Init [B] lead to fundamentally different training dynamics (as shown in Figure l):
1. Init [A] allows the use of larger learning rates compared to Init [B]
2. Init [A] leads to 'internal instability" where the features Az (for some input z) are large but LoRA output BAz is small. This form of instability allows more efficient feature learning. We identify a feature learning / stability tradeoff in this case.

---

3. Init [B] does not cause any instabilities but training is suboptimal (B is undertrained).
4. Empirical results confirm the theory and show that Init [A] generally leads to better per- formance than Init [B].
2 Setup and Definitions
We consider a general neural network model of the form
Yn = Wina,
Yi(x:) Fi(W, (x)) Le E [4].
(I)
(x)r W (z).
where r E Rd is the input, L 2 1 is the network depth, (Fi)re(z) are mappings that define the layers, and Wi E Rax= are the hidden weights, where n is the network width, and Win W are input and output embedding weights." This model will represent the pretrained model that will later be finetuned on some new task.
To finetune (large) pretrained model with limited amount of computational resources, a popular resource efficient approach is to use the LoRA finetuning method defined below.
Definition 1 (Low Rank Adapters (LoRA) from [19]). To apply LoRA to a weight matrix W e 2" in the model, constrain its update in the fine-tuning process by representing the latter with a low-rank decomposition W = W + a BA. Here, only the weight matrices B E T xr. A E Rexa2 are trainable and the original pretrained weights W remain frozen. The rank r < min( [11 n2) and a E R are tunable constants.
As the width grows, 2 the network initialization scheme and the learning rate should be adapted to avoid numerical instabilities and ensure efficient learning. For instance, the variance of the initial- ization weights (in hidden layers) should scale like 1/n to prevent the pre-activations from blowing up as we increase model width n (e.g., He initialization [4]). derive proper scaling rules, prin- cipled approach consist of analyzing the statistical properties of key quantities in the model (e.g- second moment of the pre-activations) as n grows and then adjust the initialization variance, the learning rate, and the architecture to achieve desirable properties in the limit 40]. We use this approach to study the effect of initialization on the feature learning dynamics of LoRA in the infinite-width limit. For more details about the theory of scaling of neural networks, see Appendix A.2.
do [5, 10, 13,
Throughout the paper, we will be using asymptotic notation to describe the behaviour of several quantities as the width n grows. Note that the width n will be the only scaling dimension of neural network training which grows and all other scaling dimensions such as the LoRA rank number of layers I., sequence length, number of training steps, etc., will be considered as fixed. We use the following notation for the asymptotic analysis.
Notation. Given sequences Ca E R and dn E RR+, we write C, = O(d)), resp. Cn = l(dx), ),
refer Cn kdn, resp. Gn kdz, for some constant x > 0. We write On (dd,) if both Cn (d,,) and c, M(d,,) are satisfied. For vector sequences C, = (c))SEEA E R (for some k >> 0), we write C. (dd,) when of O(d,,) ) for all E [K], and same holds for other asymptotic notations. Finally, when the sequence 6, a vector of random variables, convergence is understood to be convergence in second moment (Lg norm).
2.1 Initialization of LoRA Adapters
The standard way to initialize trainable weights is to take an iid initialization of the entries Aij ~ (O o²),, By (O oz)) for some α o 2 (this includes initialization with zeros a or A
I We use the same notation from Hayou et al. [44]. TTe width in SOTA models is typically large, Le. of width 10a.
3

---

are set to 0).² Due to the additive update structure of LoRA, we want to initialize the product BA be that finetuning starts from the pretrained model [19]. This can be achieved by initializing one of the weights A and B to O. If both are initialized to 0, no learning occurs in this case since this a saddle point and the parameter gradients will remain zero. Thus, we should initialize one of the parameters A and B to be non-zero and the other to be zero. If we choose a non-zero initialization for A, then following standard initialization schemes (e.g., He Init [4], LeCun Init [1], one should set o ((n to ensure Ar does not explode for large This is justified by the Central Limit Theorem (CLT), On the other hand, if we choose a non-zero initialization for B, one should make sure that of ((r=)) (1). This leaves with two possible initialization schemes:
Init [A] 8 0, = (n (default initialization in LoRA [19]).
Init [nit[B]: 0 = 0) = (1), = 0.*
These two initialization achieve the goal of starting finetuning from the pretrained model. priori, it is unclear if there is a material difference between the two initialization schemes. Surprisingly, as we will show later in this paper, these two initialization schemes lead to fundamentally different training dynamics when model width is large.
2.2 LoRA Features
Notation. For a given LoRA layer in the network, we use Z to denote the input to that layer and ž for the output after adding the pretrained weights. More precisely, write the layer operation as Z = ZzZ + + BAZ
Our main analysis relies on careful estimation of the magnitude of several quantities involving LoRA features. Let us first give a formal definition.
Definition (LoRA Features). Given a general neural architecture and LoRA layer (Definition 1), we define LoRA features AZ ZB) as Z AZ and ZB = BZ = BAZ At Aine-tuuning step I, we use the superscript t to denote the value of LoRA features AA, BB and the subscript to denote the weights Ar, Br.
LoRA Finetuning Dynamics in the Large Width Limit
We fix the LoRA rank throughout the analysis and examine the finetuning dynamics in the limit of large width. This setup aligns well with practical scenarios where the rank is much smaller than the width (i.e., r
). Typically, for Llama models the rank r is generally of order 2 for k E .,. 6). and model width n is generally larger than 212 We will refer to a layer of the network to which LoRA is applied (see Definition 1) as a LoRA layer. For the theoretical analysis, we adopt simplified setting that facilitates a rigorous yet intuitive derivations of the results.
3.1 Simplified Setting
The following simplified setup was considered in Hayou et al. [44] to derive asymptotic results concerning the learning rates in LoRA. We use the same setup in our analysis to investigate the impact of initialization.
Finetuning Dataset. We assume that the dataset used for finetuning consists single datapoint (ar, v) and the goal is to minimize the loss calculated with the model with adjusted weights W* + B.A for all LoRA layers (here = {A, B, for LoRA layers the model)). Z' is the input to the LoRA layer, computed with data input æ. Similarly, we write dZ' to denote the gradient of the loss function with respect to the layer output features Z evaluated at data point (z, y).
Gaussianity is not important and can be replaced by any zero-ment mean distribution with mmite-variancs for our purposes.
'Here, assumed that @11) (in width), i.e. it doesn't grow with width. In general, the right scaling for Init [B] is of Or--t) "Atthough this a simplifying assumption for our analysis, the results can be extended to mini aani-batched gra- dients without affecting the conclusions. Such results will require additional assumptions be fully rigorous.

---

Single LoRA Module. Given a LoRA layer, LoRA feature updates are not only driven by the change in the A, weights, but also the changes in Z. dZ which are updated as we finetune the model (assuming there are multiple LoRA layers). To isolate the contribution of individual LoRA layers to feature learning, we assume that only a single LoRA layer is trainable and all other LoRA layers are frozen. For this LoRA layer the layer input Z is fixed and does not change with i, whereas dž changes with step (because Z' = (W* + B,) After step t, ZB is updated as follows
A AZ = B
A
A A +
AB,AZ
6
6
As discussed in Hayou et al. [44], the terms 87 02 represent 'linear' feature updates that obtain we fix one weight matrix and only train the other. The third term of represents the 'multiplicative" feature update which captures the compounded update due to updating both A and B.
3.2 Stability and Feature Learning
Hayou et al. [44] introduced the notion of stability of LoRA features as width grows. We introduce here slightly more relaxed notion of stability.
Definition 3 (Feature Stability). We say that LoRA finetuning is stable for all LoRA layers in the model, and all training steps t, we have Z, Z = O(1), as the width n goes to infinity.
Here, feature stability implies that LoRA output Zg remains bounded (in L norm) as width grows. To achieve such stability, hyperparameters (initialization, learning rate) should be scaled as n grows. We will show that the dependence of the optimal learning rate on n is highly sensitive to the choice of initialization (Init [A] or Init [B]).
Note that feature stability also requires that = 0(1) which is directly related to pretraining dynam- ics since it depends on some pretrained weights W", We assume that pretraining parameterization (how initialization and learning rate are parametrized w.r.t width) ensures this kind of stability (see Appendix A for more details)."
As discussed above, feature updates are driven by the terms (1.2..).. As n grows, these feature updates might become trivial (i.e. vanish as - oc) or unstable (i.e. grows unbounded). To avoid such scenarios, we want to ensure that A ((1). Such conditions are the main ideas behind JP [26] and Depth-uP [41], which are network parametrizations that ensure stability and feature learning in the large width and depth limits for pretraining. We recall this definition from [44].
Definition 4 (Feature Learning). We say that LoRA finetuning induces stable feature learning in the limit of large width if the dynamics are stable (Definition 3), and for all finetuning steps t, we have A f 7/+1 Z = (1).
AZD is the sum of the terms 8t's (Equation (2)). To achieve optimal feature learning, we want to ensure that 87 = (11) and oe 0(1 which means that both weight matrices A and B are efficiently updated and contribute to the update in Zp- An intuitive explanation is provided in Appendix A. This leads us to the following definition of efficient learning with LoRA.
Definition 5 (Efficient Learning with LoRA). We say that LoRA fine-tuning is efficient if it is stable (Defiition 3), and for all LoRA layers in the model, and all fine-tuning steps > 1, we have
5 = O(1) i (1,2}.
Next, we introduce the --operator, an essential tool in our analysis of the large width dynamics.
This is equivalent to having only a single LoRA layer the model since LoRA layers are initialized to zero.
'When taking the infinite width limit, can for instance assume that pretraining parameterization is pP [26]. This a technicality for the infinite-win width limit and does not have any implications on practical scenarios where the width is finite. The most important implications of this assumption is that in the pretrained network (before introducing LoRA layers), we have Z = +(1), 2 = (1), which holds for a general pupput pair (Ir, y).
5

---

3.3 Introduction to the 7-operator
In the theory of scaling, one usually tracks the asymptotic behavior of key quantities as we scale some model ingredient. For instance, if we scale the width n of a neural network, we are interested in quantifying how certain quantities in the network behave as n grows. This is standard approach for (principled) model scaling and it has so far been used to derive scaling rules for initialization [5], activation function [10], network parametrization [41], amongst other things.
With Init [A] and Init [B], initialization weights are of order O( ((n- for some 3 2 0. Assuming that the learning rate also scales polynomialy with n, it is straightforward that preactivations, gra- dients, and weight updates are all asymptotically polynomial in n. Note that this is only possible because all neural computations consists of sums of (nn)) terms, where typically a E {0, 13. For instance, when calculating the features AZ. each entry is sum of n terms, while when calculating BZA, each entry sum of terms (r fixed n goes to infinity). This is true for general neural computation that can be expressed as Tensor Programs [15].
Consequently, for some quantity in the computation graph, it is natural to track the exponent that determines the asymptotic behavior of u with respect to n. We write = =(>(e)) to capture this polynomial dependence. Elementary operations with the y-operator include:"
Zero.
When v = 0, we write >[0] = -c (as a limit of y/n when 8 + oo).
Multiplication.
Given two real-valued variables v, v', we have 7/v x r/] >u1| + vlv))
Addition. Given two real-valued variables v, v', we generally have 7/v + r'1 = max() Moll).
The only case where this is violated is when o' o'= -v. This is generally zero probability event if and v' are random variables that are not perfectly (negatively) correlated, which the case in most situations where we make use of this formula. See Appendix A.3 for discussion.
We have now introduced all required notions for the subsequent analysis. For better readability, we defer all the proofs to the appendix.
3.4 Recursive formulas
Using the y-operator, track the asymptotic behavior of the finetuning dynamics model width n grows. At finetuning step t. the weights are updated as follows
A A,-1 1111-1101 A B, = B ng's
where 94 9B are processed gradients (e.g. normalized gradients with momentum as in AdamW). We assume that the gradients are processed in way that makes their entries (1). This gen-
erally satisfied in practice (with Adam for instance) and has been considered in [40] to derive the u-parametrization for general gradient processing functions. From this, we obtain the following
recursive formulas for 7[2%] and 7 [Br), which characterizes their behavior in the large width limit.
Lemma 1 (Informal). For fixed, the asymptotic dynamics of Z'A and B, follow the recursive formula
[ max (>) 1 [ + > [B]] = max (>
(3)
The formal proof of Lemma 1 is provided in Appendix A and relies on Assumption I which fairly represents practical scenarios (see Appendix A for a detailed discussion). Lemma captures the change in asymptotic behavior of quantities 2 and B, as width grows. Naturally, the dynamics depend on the the initialization scheme which lead to completely different behaviors as we show in the next two results.
3.5 Init [A] leads to more efficient feature learning but suffers "internal" instability
Next, we provide a precise characterization of stability and feature learning when using Init [A]. Theorem (Informal). For t fixed, with Init [A] and learning rate Y,, have
'The y-operator is mapping from the set {v, s.t.v O(n²) for 9 IR U (-co)) the set R U (-o0).

---

Stability: BB= (1) if and only if I -1/2.
Feature Learning: 2 ((1) if and only if 7/]] 1122. In this case, we also have 6,,8 = (1) (efficient feature learning, Definition 5).
Moreover, "internal" instability (2') (2)) occurs when 7[n] E (-1,1/2).
With Init [n] the maximal learning rate" that does not lead to instability in ZB scales as ((n-1/9) This can be seen as an asymptotic form of the edge of stability phenomenon [17] where if we increase the learning rate beyond some level, instability occurs. Interestingly, in this case (i.e. with On 1/2) learning rate) the features are efficiently updated (Definition 5). However, this comes with caveat: the features Z'A grow (/2 which can potentially cause numerical instabilities. We call this phenomenon internal instability: only the features ZA (internal LoRA features) grows, LoRA output Zp remains (1) in this case.
The fact that ( is the maximal learning rate that does not cause instability in ZB does not mean it is the optimal learning rate. As the width n grows, this internal instability in Za will become more and more problematic. Intuitively, we expect that a trade-off appears in this case: the optimal learning rate (found by grid search) to be larger than (nn=l) but smaller than (n2313), i.e. the network will try to achieve a balance between optimal feature learning (r[v) = -1/2) and internal stability 2 = (1) (rin) 1).. We verify this empirically in the next section.
3.6 Init [B] leads to suboptimal feature learning with internal stability
In the next result, we show that the maximal learning rate allowed with Init [B] is different from that with Init [A], leading to completely different dynamics.
Theorem (Informal). For t fixed, with Init t [B], we have
Stability: Z3 (1) if and only Iml S 11
Feature Learning: AZ = ((1) if and only if = -1.
Moreover, feature learning cannot be achieved with Init t [B] for any choice of learning rate scaling 7/1] (that does not violate the stability condition). More precisely, with O(n learning rate, the limiting dynamics (when n. - oc) are the same if B was not trained and A is trained.
With Init [B], the maximal learning rate (that does not violate stability) scales as O(n (for any >> 0, a learning rate of O (1+5) leads to Z -2())).
Because of this bound the maximal learning rate, no internal instability occurs with Init [B]. In this case, feature learning is suboptimal since the weight matrix is undertrained in the large width limit (8} + 0).
Conclusions from Sections 3.5 and 3.6. The results of Theorem 1 and Theorem 2 suggest that Init [A] allows the use of larger learning rates compared Init [B], which might lead to better feature learning and hence better performance at the expense of some internal instability. Here, 'larger' learning rate should be interpreted in asymptotic terms: with Init [A] the maximal learning rate that does not cause instability satisfies 7[n] = -1/2. With Init[B], we have イ[7] -1 instead. Note that because of the constants in (nn)) learning rates (for some (3) the optimal learning rate with Init [A] is not systematically larger than Init [B] for finite width. However, as width grows, we will see that it is case.
Another important insight from this analysis is that with both initializations, the dynamics are sub- optimal in the limit: internal instability with Init [A] and undertraining of B with Init [B]. 10 We will later discuss possible solutions to this behavior.
9 "Maximal 7(1]] that does not cause instability ZB "Mor precisely, one can show that with Init [B], for tixed t. in the limit + co, B, converges to Bo. i.e. B untrained in this limit.
7

---

Width = 128
Width 8192
IntiA]]
Inil[B)
/-4.2.00 1-m 6o-02
5.3e-04, 0m-.1
t 2.4e-04, m= n=116e-03 66-033
< 3. 3.77--04, U4, q-
1.5
1.5
[Az]
1.0
1.0
[8A2]
AAz
0.5
0.0
3.00 04, 2.30-20
3.0e ge1.20-00
2.9. n-1.2:
04, 9-9.8o
1.5
1.5
1.0
0.5
0.0
0.0
50
100
0
50
100
0
50
100
0
50
100
step
step
step
step
Figure 3: Evolution of the norms the 7,, Zg features, averaged over training data. We compute the average
shown for widths [Zal def N N [2;)) (and same for ZB). where the r;'s are the training data. The dynamics are
128 and 8192, seeds, and for both Init [A] and Init [B] Train loss and the (optimal) learning rate are shown on top cach plot. observe that the magnitude of Za is significantly higher with Init [A] compared to Init [B] at large width (n 8192). Interestingly, the train loss is smaller with Init [A], as compared to Init [B]. Results with other seeds and widths are shown in Appendix B.
3.7 Toy Model
To validate our theory in controlled setting, we con- sider the following simple model:
2-6
Feuture Leuring
2-2
Yin- Wine
2-8
Yn Y. + (Wn+ + (Y)
(4)
n
2-
Your Wour
2-10
InitiAl
2-1
Cyn
where W E Baxd Wn RRx" Wout E Rixn and A E BRkn.
Init[D]
12
C2n"
24
212
We generate synthetic data from the teacher model using the following config: d 5, Fteacher N
n
20,n n and Nieet = 100 (test data size). The weight Witeacher Wttaakeer Ateacher and Bleacher are ran- domly initialized, and Wteacher 0. We train stu- dent models with d 5,r = 4, and varying widths (2k, k 13).
1000 (train data size),
Optimal Learning rate for the fine- tuning of synthetic model Equation (4) with Init [A] and Init [B] as initialization. The optimal
Figure 2:
function width Theoretical lines n- and n -1/2 shown well (constants C1, C2 are chosen provide suitable trend visualization).
are shown
model width n grows, the optimal learning rate with Init [A] becomes larger than the optimal learning rate with Init [B]. This is in agreement with the the- oretical results.
Optimal Learning Rate. We finetune model (4) on synthetic data generated from the teacher model. In Figure 2, we show the optimal learning rate when us- ing either Init [A] or Init [B] to initialize the fine- tuning, as a function of width n. For n >> (typically Init [A] is larger than the optimal learning rate with Init [B]. This is in agreement with the the- oretical results obtained in Theorem and Theorem 2 which predict asymptotic maximal learning rates (that satisfy the stability condition) of On 1/2) and O(n 1) respectively.
> 2º), the optimal learning rate with
With Init [A], we observe the stability/feature learning trade-off for large n. The optimal learning rate with Init [A] in this regime (e.g. 213 is smaller than the maximal theoretical learning
Heere the pretrained model is effectively given by Yous Whencher 6(Wteaacher z), and the finetuning dataset is simulated by injecting the LoRA weights Ateacher pteacher In this setup, student model can have larger width than the teacher model.

---

rate n 1/2 that achieves optimal feature learning (Theorem 1), Here, the model seems to balance the internal instability that occurs in the Za features with feature learning and thus favors smaller learning rates: the optimal learning rates smaller than O(n 1/22) and larger than On
Internal Instability and Feature Learning. Figure 3 shows the average magnitude of Za and ZB for Init [A] and Init [B] for widths n {128, 8192]. With Init [A], the magnitude of ZA features seem to grow with width, hence trading off internal stability for more efficient feature learning. This behavior is consistent across random seeds as shown in the figure, and as further confirmed by experiments in Appendix B. The train loss is consistently smaller with Init t [A], which can be explained by the fact that Init [A] allows more efficient feature learning at the cost of some internal instability. This flexibility cannot be achieved with Init Note also that Z3 features tends to get smaller with with Init [A] as predicted by theory: the trade-off between internal instability and feature learning implies that 17" o(n 1/2), which implies that Z/ o(1). i.e. the Zg features vanish as width grows. While this might problematic, it only becomes an issue when the width is extremely large: if the optimal learning rate scales as O P) for some (1/2, 1) (so that the learning rate is between ((n and (n balancing internal instability and efficient Therefore, if 3 ss 0.7 for instance, the vanishing rate of LoRA output feature is Zn R O( (033).3) which is slow given the order of magnitude of width in practice (for n 2 we have -0.3
feature learning). the LoRA output feature scales as Zn A, O(n
-0.3 ze 0.08).
Experiments with Language Models
Our theoretical results from earlier provides a detailed asymptotic analysis of the finetuning dy- namics when LoRA modules are initialized with Init [A] or Init [B]. The main conclusions are that Init [A] generally leads to more efficient feature learning (which can be justified by the fact that optimal learning rate is larger when using Init [A] compared to when using Init [B]). To provide evidence of this claim on real-world tasks, we use LoRA to finetune a set of language models on different benchmarks. Details about the experimental setup and more empirical re- sults are provided in Appendix B. We use LoRA+ code [44] for our experiments (available at https: /etps://gitain
thub. com com/nikhil- //gttbbb.......... behhoh--brrkelley/oooaaa..
4.1 GLUE tasks with RoBERTa
The GLUE benchmark (General Language Understanding Evaluation) consists of several language tasks that evaluate the understanding capabilities of langugage models [8]. Using LoRA, we fine- tune Roberta-large from the RoBERTa family [12] on MNLI, SST2, and QNLI tasks with varying learning rates 1) and initialization schemes (Init [A] or Init [B]). ). We use the same experimental setup of [19] for Saberta-Large to compare our results with theirs (see Appendix B for more details).
MNU
InitAA): Acc 90 09, n" 8.00-05
Int]AA Acc 95.57 -156-04
Ind[ ACC
95.09...0000 n" -.00000 [):
[Nt: Acc 89.47. 6" -1.00 45
Acc 96.44, -4.06-05
In
93.613 8.00 05
0.350
1188
((AA)
0.7
0.300
6.6-
0.100
0.3
30a0
Figure 4: Test Accuracy for LoBERTa-Lare finetuned on GLUE tasks. The results shown after vergence of finetuning with LoRA, initialized with either Init[A] or Init [B]. Models were finetuned using LoRA rank r = 8 and FP16 precision. Optimal learning rate and corresponding accuracy are shown on top of each panel for both initializations. The experimental setup is provided in Appendix B.
The results Figure are aligned with our theory: we observe that Init [A] generally leads to bet- ter performance, and the optimal learning rate with Init [A] is generally larger than with Init [B]. Models initialized with Init [A] match the performances reported in [19], while those initialized with Init [B] generally underperform that baseline. For MNLI task (the hardest one amongst the

---

three tasks), we observe a significant difference in the best test accuracy (over 3 random seeds) with 90.69 with Init [A] and 89.47 with Init [B]. We also observe that for MNLI, the optimal learn ing rate with Init [A] (n¹ 8e-5) is much larger than the optimal learning rate with Init [B] (1) = 1e-5), which aligns with our theoretical predictions. However, note that for QNLI for in- stance (an easier task), while the optimal test accuracy is significantly better with Init [A], the optimal learning rate (from the grid search) is the same for Init [A] and Init [B] There are many possible explanations for this: 1) the width is not large enough in this case to see the gap between optimal learning rates (for RoBERTa-Lage,, the width is n = 21t) 2) The constants in O(n 1) are (()
(n 1/2) are significantly different in magnitude due to dependence on finetuning task. We notice similar behaviour with LLama experiments below. A precise analysis of this observation is beyond the scope of this paper, we leave it for future work.
4.2 Llama
Teyl lano on Wilfor:
en-llan-s INITl: Ace 4-4 40+00
Ie All (ues, a mi 4.00.4
111EE: ACc
1.0-00
3.4
0.40
3.22
0.3
744
3.30
0.20
0.20
19/01
185A
9.32
3.34
Figure 5: (Left) Test perplexity (lower is better) of TinyLJama LoRA on WikiText-2 with Init[A] and Init [B]. (Center) MMLU accuracy of Llama-7b LoRA finetuned the Flan-v2 dataset. (Right) GSM8k test accuracy Llama-7b LoRA finetuned on the GSM8K dataset. More experimental details are provided in Appendix B.
To further validate our theoretical findings on more modern models and datasets, report the results of finetuning the Llama-7b model [38] on the Flan-v2 dataset [36] and the GSM8k dataset [16], and finetuning the TinyLlama model [49] on WikiText-2 using LoRA. Each trial is averaged over two seeds and the shaded region indicates one standard error. In the left panel of Figure 5 we see that when finetuning TinyLlama using LoRA the optimal learning rate using Init [A] is larger than with Init [B] and the corresponding test perplexity is lower. Similarly, for the center panel of Figure 5, when finetuning the Llama-7b model on F2an-v2, the optimal learning rates for Init [A] and Init [B] are the same (for the learning rate grid we used), but the the optimal MMLU accuracy for Init [A] is slightly higher than for Init For learning rates close to the optimal choice, the accuracy using Init [A] is generally higher than for Init [B]. An analagous result holds for the GSM8K dataset as shown in the rightmost panel of Figure 5. More details about this setting are provided in Appendix B.
5 Conclusion and Limitations
We showed that LoRA dynamics are highly sensitive to initialization. Init [A] is associated with larger optimal learning rates, compared to Init [B]. Larger learning rates typically result in better performance, as confirmed by our empirical results. Note that this is a zero-cost adjustment with LoRA finetuning: we simply recommend using Init[A] [A] instead of Init[B].
One limitation of our work is that we only define feature learning via the magnitude of feature updates in the limit of large width. In this way, our definition of feature learning is data-agnostic and therefore no conclusion about generalization can be obtained with this analysis. The constants in O()) asymptotic notation naturally depend on the data (the finetuning task) and therefore such data-agnostic approach does not allow us to infer any information about the impact of the data on the finetuning dynamics.
More importantly, our results indicate that both initialization schemes lead to suboptimal scenarios, although Init [A] allows more efficient feature learning. In both cases, instability and/or suboptimal feature learning present fundamental issues, which can potentially be mitigated by approaches such as LoRA+ [44]. Understanding the interaction of LoRA+ and related efficiency methods with the initialization scheme is important question for future work.
10

---

Acknowledgement
We thank Gradient AI for cloud credits under the Gradient AI fellowship awarded SH and thank AWS for cloud credits under an Amazon Research Grant awarded to the Yu Group. We also grate- fully acknowledge partial support from NSF grants DMSS2200997 2015341, 20241842, NSF grant 2023505 on Collaborative Research: Foundations of Data Science Institute (FODSI), the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning through awards DMS-203188 183 and 8 814639, and NSF grant MC2378 to the Institute for Artificial CyberThreat Intelligence and OperatioN (ACTION).

---

References
[1] Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. "Efficient backprop". In: Neural networks: Tricks of the trade. Springer, 2002, pp. 9-50.
[2] L. Yang, S. Hanneke, and J. Carbonell. "A theory of transfer learning with applications to active learning". In: Machine learning 90 (2013), pp. 161-189.
[3] D. P. Kingma and J. Ba. "Adam: A method for stochastic optimization". In: arXiv preprint urXiv::412 1e12.14818 (2014).
[4] K. He, X. Zhang. S. Ren, J. Sun. "Deep residual learning for image recognition". In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770- 778.
[5] S. Schoenholz, J. Gilmer, S. Ganguli, and J. Dick--ickstein. "Deep Information Propagation". In: International Conference on Learning Representations. 2017.
[6] S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep Information Propaga- tion. 2017. arXiv: 1611 01232 [stat. ML].
[7] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. "Improving language understand-
ing by generative pre-training". In: (2018).
[8] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. 2018. arXiv: 1804. 07461 [cs. CL.
[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. "BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding". In: arXiv preprint 1:10008110.44000 (2019).
[10] S. Hayou, Doucet, and J. Rousseau. "On the Impact of the Activation function on Deep Neural Networks Training". In: Proceedings of the 36th International Conference on Ma- chine Learning. Ed. by K. Chaudhuri and R. Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, Sept. 2019, pp. 2672-2680.
[11] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. "Parameter-effict" Conference on Machine Learning. PMLR. 2019, pp. 2790-2799.
transfer learning for NLP". In: International
[12] Y. Liu, M. Ott, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019, arXiv: 11692 CL).
[13] G. Yang. "Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation". In: arXiv preprint /xXiv:/902.04700 (2019).
[14) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. "Scaling laws for neural language models". In: arXiv preprint arXiv:2001.088 08366 (2020).
[15] G. Yang. "Tensor programs ii:: Neural matrix laws". In: arXiv preprint 2rXiv:2009.16683 (2020).
[16] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, al. "Training verifiers to solve math word problems". In: arXiv preprint 1rXiv:2110.1411 (2021).
[17] J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. "Gradient Descent on Neural Net- works Typically Occurs at the Edge of Stability". In: International Conference on Learning Representations. 2021.
[18] S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau. "Stable ResNet". In: Proceedings of The 24th International Conference on Artificial Intelligence and Statistics. Ed. by A. Banerjee and K. Fukumizu. Vol. 130. Proceedings of Machine Learning Research. PMLR, 13-15 Apr 2021, pp. 1324-1332.
[19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. "LoRA: Low-Rank Adaptation of Large Language Models". In: arXiv preprint arXiv:2106.0996 (2021).
[20] B. Lester, R. Al-Rfou, and N. Constant. "The power of scale for parameter-efficient prompt tuning". In: arXiv preprint arXiv:2104.08691 (2021).
12

---

[21] G. Yang and E. J. Hu. "Tensor programs iv: Feature learning in infinite-width neural net- works". In: International Conference on Machine Learning. PMLR. 2021, pp. 11727-11737. [22] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training Compute Opttmmll--pmmlll Large Language Models. 2022. arXiv: 2203, 15556 [cs.CL) CL..
[23] M. Li, M. Nica, and D. Roy. "The Neural Covariance SDE: Shaped Infinite Depth-and Width Networks Initialization". In: Advances Neural Information Processing Systems. Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. Vol. 35. Curran Asso- ciates, Inc., 2022. pp. 10795-10808.
[24] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. "Few-shot parameter-efficiee fine-tuning is better and cheaper than in-context learning". In: Advances in Neural Information Processing Systems 35 (2022), pp. 1950-1965.
[25] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph. S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. "Emergent abilities of large language models". In: arXiv preprint arXiv:2206. 276-22222007622 (2022).
[26] G. Yang. E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and Gao. "Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer". In: arXiv preprint arXiv:2203.03466 (2022).
[27] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. "QLoRA: Efficient Finetuning of Quantized LLMs". In: arXiv preprint arXiv:2305. (2023).
(28) S. Hayou, "On the infinite-depth limit of finite-width neural networks". In: Transactions on Machine Learning Research (2023). ISSN: 2835-8856.
[29] S. Hayou and G. Yang. "Width and Depth Limits Commute in Residual Networks". In: Pro- ceedings of the 40th International Conference on Machine Learning. Ed. by A. Krause, E. Brunskill, K. Cho, B. Engelhardd, S. Sabato, and Scarlett. Vol. 202. Proceedings of Ma- chine Learning Research. PMLR, 23-29 Jul 2023, pp. 12700-12723.
[30] B. He, Martens, G. Zhang, A. Botev, A. Brock, S. L. Smith, and Y. W. Tch. Deep Trans- formers without Shortcuts: Modifying Self-aftention for Faithful Signal Propagation. 2023. arXiv: 2302. 10322 .LG.
[31] D. Kalajdzievski. "A Rank Stabilization Scaling Factor for Fine- Fine-Tuning with LoRA". In: arXiv preprint arXiv:2312.0377. (2023).
[32] S. A. Koohpayegani, K. Navaneet, P. Nooralinejad, S. Kolouri, and H. Pirsiavash. "NOLA: Networks as linear combination of low rank random basis". In: arXiv preprint arXiv:2310.02556 (2023).
[33] Kopiczko, Blankevoort, and Y. M. Asano. "VeRA: Vector-based Random Matrix Adaptation". In: arXiv preprint arXiv:2310.114:: (2023).
[34] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. "Loftq: Lora-fine- uning-aware quantization for large language models".. In: arXiv preprint 0rXiv:2310.0853 (2023).
[35] H. Liu, C. Li, Y. Li, and Y. Lee. "Improved baselines with visual instruction tuning". In: arXiv preprint arXiv:2310.0374 (2023).
[36] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung. Y. Tay, D. Zhou, Q. V. Le. B. Zoph, J. Wei, et al. "The flan collection: Designing data and methods for effective instruction tuning".
In: arXiv preprint 23X1112301..3688 (2023).
[37] L. Noci, C. Li, M. B. Li. B. He, T. Hofmann, C. Maddison, and D. M. Roy. The Shaped Transformer: Attention Models in the Infinite Depth-and- Width Limit. 2023. arXiv: 2306. 17759 [stat.
[38] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan,
13

---

P. Xu. Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. "Llama 2: Open Foundation and Fine-Tuned Chat Models". In: (2023).
arXiv preprint arXiv:2307.0922.
[39] Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. "How Far Can Camels Go? Exploring the State of Instruction Tuning Open Resources", In: arXiv preprint arXiv:2306.00 04751 (2023).
[40] G. Yang and E. Littwin. "Tensor programs ivb: Adaptive optimization the infinite-width
limit". In: arXiv preprint arXiv:2308, 01814 (2023).
[41] G. Yang, D. Yu, C. Zhu, and S. Hayou. "Tensor Programs VI: Feature Learning in Infinite- Depth Neural Networks". In: arXiv preprint arXiv:2310.02244 (2023).
[42] L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li. "Lora-fa: Memory-efficient low-rank adapta-
tion for large language models fine-tuning". In: arXiv preprini arXiv:2308 03303 (2023).
[43] Balazy. M. Banaci, Aberer, and J. Tabor. "LoRA-XS: Low-Rank Adaptation with Ex- tremely Small Number of Parameters". In: arXiv preprint arXiv:2405. 17604 (2024).
[44] S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient Low Rank Adaptation of Large Models.
2024. arXiv:
2402 12354 [cs LG].
[45] T. Jiang, S. Huang, S. Luo, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, Q. Zhang, D. Wang, al. "MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning", In: arXiv preprint
arXiv:2405.12130 (2024).
[46] Y. Li, S. Han, and S. Ji. "VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks". In: arXiv preprint arXiv:240s. (2024).
[47] S.-Y. Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. "DoRA: Weight-Decomposed Low-Rank Adaptation". In: arXiv preprint arXiv:2402.0935 (2024).
[48] F. Meng, Z. Wang, and M. Zhang. "PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models". In: arXiv preprint vxXiv:2404.0294 (2024).
[49] P. Zhang. G. Zeng. T. Wang, and W. Lu. "Tinyllama: An open-source small language model".
In: arXiv preprint arXiv:2401.02333 (2024).
[50] J. Zhu, K. Greenewald, K. Nadjahi, H. S. de OcÅjriz Borde, R. B. Gabrielsson, L. Choshen, M. Ghassemi, M. Yurochkin, and J. Solomon. Asymmetry in Low-Rank Adapters of Founda- tion Models. 2024. arXiv: 2402. 16842 LG].
14

---

A Theory and Proofs
A.1 Role of A and B weight matrices
Recall the feature update decomposition
AZ Bi-1AZ
ABIZA A
AB,AZA
(5)
62
To achieve optimal feature learning, we want to ensure that 6, = (1) and 8f = (1) which means that both weight matrices A and B are efficiently updated and contribute to the update in Zz. To justify why this is a desirable property, let us analyze how changes in matrices A and affect LoRA feature ZB= BAZ-
Let (isr) denote the columns of B. We have the following decomposition of ZB:
ZB ((A)),
where (AZ), is the 7t coordinate of AZ. This decomposition suggests that the direction of Zp is a weighted sum of the columns of B, and A modulates the weights. With this, we can also write
8 = E. (4A2))
87 - (4-
where (B, a) refers to the columns of B at time step t. Having both 87 and 87 of order O(1) means that both A and B are "sufficiently" updated to induce a change weights (AZ); and directions B. If one of the matrices A, B is not efficiently updated, we might end up with suboptimal finetuning, leading to either non updated directions B direction weights (A-- For instance, assuming that the model is initialized with Init [B], and that B is not efficiently updated, the direction of Z will be mostly determined by the vector (sub)space of dimension generated by the columns of B at initialization.
This intuition was discussed in details in [44].
A.2 Scaling of Neural Networks
Scaling refers to the process of increasing the size of one of the ingredients in the model to improve performance (see e.g. [22]). This includes model capacity which can be increased via width (em- bedding dimension) or depth (number of layers) or both, compute (training data), number of training steps etc. In this paper, we interested in scaling model capacity via the width n. This is motivated by the fact that most state-of-the-art language and vision models have large width.
It is well known that the width grows, the network initialization scheme and the learning should be adapted to avoid numerical instabilities and ensure efficient learning. For instance, the initial- ization variance should scale 1/n to prevent arbitrarily large pre-activations we increase model width n (e.g. He init [4]). To derive such scaling rules, a principled approach consist of analyzing statistical properties of key quantities in the model (e.g. pre-activations) grows and then adjust the initialization, the learning rate, and the architecture itself to achieve desirable properties in the limit xc [5, 10, 13].
In this context, Yang et al. [26] introduces the Maximal Update Parameterization (or jP), set of scaling rules for the initialization scheme, the learning rate, and the network architecture that ensure stability and maximal feature learning in the infinite width limit. Stability is defined by Y/ = O(1) for all l and where the asymptotic notation 'O(.)' is with respect width n (see next paragraph for a formal definition), and feature learning is defined by AY; = @(1), where refers to the feature update after taking gradient step. μP guarantees that these two conditions any training step t. Roughly speaking, μP specifies that hidden weights should be initialized with O(n -1/2) random weights, and weight updates should of order (nn Input weights should be initialized ((1) and the weights update should be @(1) well. While the output weights should be
satisfied at
15

---

initialized ((n=)) and updated with (nn These rules ensure both stability and feature learning the anfiitt-wwithh limit, in contrast standard parameterization (exploding features the learning rate is well tuned), and kernel parameterizations (e.g. Neural Tangent Kernel parameterization where AYi = O( (112), i.e. no feature learning in the limit).
A.3
When does --Perraor fail to capture asymptotic behavior?
When non-polynomial dependencies (in terms of n) appear in neural computations, then the 7 oper- ator cannot capture asymptotic behavior of the learning dynamics. For instance, if one of the layers has embedding dimension e" or n (ng(n), polynomial exponents are no longer sufficient to capture the asymptotic dynamies. Fortunately, such cases are generally not considered in practice.
A.4 Proof of Lemma
In this section, we provide the formal proof of Lemma 1. The proof relies on the following assump- tion the processed gradient 9A- This assumption used in [44] to derive scaling rules for the optimal learning rates for A and weight matrices. Here, we use it to study the sensitivity of LoRA dynamics to initialization. We provide intuitive discussion that shows why this assumption is realistic.
Assumption 1. With the same setup of Section 3, at training step t, we have Z, dž = (1() and (
Assumption consists of two parts: that 1) Z, dž ((1) and 2) = The first condition is mainly related to pretraining paramterization which assume satisfied such conditions." The second condition is less intuitive, so let us provide an argument to justify why it is sound practice. Let us study the product g' in the simple case of Adam with no momentum, a.k.a SignSGD which given by
[aC)
ПΛ sign
0A
where the sign function is applied element-wise. At training step have
B, dZ &Z Z
0A
Let St = -BBI,dZ-> Therefore we have
9A sign @ Z)
(sign( SKZ,) 1ss5sn
However, note that also have
sign( S = sign(S;) siggnZ),,
and as a result
d = sign(S*) @ sign(Z).
Hence, we obtain
2/Z (Zign(Z) ZZisggn O(n),
where we used the fact that sign(Z) Tz (n)).
This intuition should in-principle hold for the general variant of Adam with momentum as long as the gradient processing function (a notion introduced in [2]) roughly preserves the (Zg direction. This reasoning can be made rigorous for general gradient processing function using the Tensor Pro- gram framework and taking the infinite-widrt limit where the components of ga, Z. dZ all become
There is technical intricacy on this point. While Z depends only on pretraining, the Jacobian dZ depends on finetuning. However. under the stability conditions mentioned in Definition 3, if dZ O it should remain during finetuning well.
16

---

iid. However this necessitates an intricate treatment of several quantities in the process, which we believe an unnecessary complication and does not serve the main purpose of this paper.
Lemma 1. Under Assumption I, the asymptotic behaviour of Z'A and Be follow the recursive for- mida
b|Za>| = max (7 + 1)
T|Bb] max (ax()(BB.1))))11)))
Proof.
At finetuning step t, the weights are updated as follows
A, A - B, B 19'
Using the elementary operations with the y-operator, we obtain
> = (, ²Z) max max()(ZX +
We conclude for z using Assumption 1. The formula for (BB, follows using the same techniques.
A.5
Proof of Theorem
Theorem 1. Under Assumption 1, For t fixed, with Init [A] and learning rate n, have
Stability: 7 O()) if and only vl) S 1/22.
Feature Learning: AZ @(1) if and only if Tn] 1/2. In this case, we also have 61,83 = @11) (efficient learning, Definition 5).
Moreover, "internal" instability (2)) = (2(1)) occurs when 7(n] E 1-1,
Proof. With Init [A], we have BBoo = oo and y[AoZ] = As a result, we have for all
{AA max 7() + 1)
[B:]] 7[1)
To achieve Zn = O(1), we should therefore have
>brl + max(0, ylal + <
which is equivalent to y/n] < 11/2.
This implies that the maximum learning rate that does not cause instability is learning rate causes internal instability, i.e. the feature Z, explodes with width. Why? Because, with this learning rate, we have >[A,Z] 1/2, i.e. A,Z ((n2/2) which diverges n grows. However, this growth is compensated with the fact that y[B 112, i.. B, O(n -1/2). This analysis is valid for any 7(n] E (-1, 1/2).
Such
In this case, feature learning is efficient in the sense of Definition 5: 5. = 0(1) and 57 = 0(1) To
this, recall that 5 B A which yields ||! B =>ín]
+
and ?|6ž] A A, + - >(1] + max(1(n) + 1,0) 0. So both weights contribute significantly to feature updates at the expense of benign exploding in Z = A.Z
17

---

A.6
Proof of Theorem 2
Theorem 2. Under Assumption for fixed, with Init [B] and learning rate n, we have Stability: ZB = (1) if and only if rln S -1.
Feature Learning: A = ((1) if and only if -1.
Moreover, feature learning cannot be achieved with Init [B] for any choice of learning rale scaling 7/1] (that does not violate the stability condition). More precisely, with @(n learning rate, the limiting dynamics (when - (C) are the same was not trained and A is trained.
Proof. Here, we show that maximal learning rate that does not cause instability in LoRA output features Zg ((n and no internal instability occurs in this scenario.
With Init [B], we have that > [Ba] and [(AoZ] -o0. From Equation (3), obtain that ^|4,Z = 7/1| +
[B. max(0; 7(1)).
As result, LoRA output stability is achieved if and only if
>(1] 1 max(0, (0,)()) s 0,
which is equivalent having >/n] 1.
Moreover, with v O(n-1) we have that (6,)] = + = 0 7[1] + = 0 and
r|v] + -1 As result, feature learning is not efficient in this case, and the learning dynamics are asymptotically equivalent to not training matrix B (because 53 0).
- y|0]] [,] +
B Additional Experiments
This section complements the empirical results reported in the main text. We provide the details of our experimental setup, and show the acc/loss heatmaps for several configurations.
B.1
Empirical Details
B.1.1
Toy Example
In Figure 2, we trained a simple model with LoRA layers to verify the results of the analysis in ??. Here provide the empirical details for these experiments.
Model. We consider a simple model given by
f(x) W out W in (WA + (,,=))
where Win Rad W out Ix A Rexn B E Raxr are the weights, and & is the ReLU activation function.
Dataset. Here, we used d 5, n 1000, and r model). Synthetic dataset generated by X ~ No0. Ta). = f(X). The number of training examples is Ntrain 1000, and the number of test examples Neest 100. the weights W.,, Wh, Wont. B, A are randomly sampled from Gaussian distribution with normalized variance (1/fan-in).
simulate synthetic data (the teacher
Training. We train the model with AdamW with B1 0.9 and 32 = 0.99 for a range for values of m. The weights are initialized as follows: Wi ~ (0, 1/d), , W ~ NO0 1/n), Wont NNO, 1/n) and fixed. Only the weight matrices A, B are trainable.
18

---

B.1.2
GLUE tasks with RoBERTa
For our experiments with RoBERTa models, finetuned on GLUE tasks, we use the following setup: Training Alg Details
Model
Roberta-Large
Learning Rates
{2 x 10-3, for k 0, 2. 10}
31
0.9
82
0.999
8
10-8
LR Schedule
Linear with Warmup Ratio 0.06
Weight Decay
0.0
Train Batch Size
4
Number of Epochs
10
LoRA Hyperparameters
LoRA Rank
8
LoRA a
16
LoRA Dropout
0.1
Target Modules
'query, value'
Other Hyperparameters
Sequence Length
T Turget 128
Random Seeds
Precision
FP16
GPUs.
Nvidia A10 with 24GB VRAM.
19

---

B.1.3
TinyLlama WikiText-2
For our experiments using the TinyLlama model finetuned on Wikitext-2, we use the following setup training with AdamW. W.
Training Algorithm Details
Learning Rates
x 10 5 10
1 10 2x 10 4x 7x 10 1 10 2 10
81
0.9
B2
0.999
E
1 10
LR Schedule
Linear with Warmup Ratio 0.03
Weight Decay
0.0
Train Batch Size
8
Number of Epochs
LoRA Hyperparameters
LoRA Rank
LoRA a
LoRA Dropout
0.0
Target Modules
"q "g_proj. k kproj. v v_proj. o_proj. up ppoj, down proj, gate_ projo
Other Hyperparameters
Sequence Length
1024
Random Seeds
2
Precision
BF16
GPUs.
Nvidia A10 with 24GB VRAM.
20

---

B.1.4
Llama-7b Flan-v2
For our experiments using the Llama-7b model finetuned on size 100k random subset of flan-v2, we use following setup training with AdamW
Training Algorithm Details
Learning Rates
1 10
x
10
1x
10
2
10 4 10 7 10 1 10-3
81
0.9
B2
0.999
E
1 10
LR Schedule
Linear with Warmup Ratio 0.03
Weight Decay
0.0
Train Batch Size
16
Number of Epochs
LoRA Hyperparameters
LoRA Rank
LoRA a
LoRA Dropout
0.0
Target Modules
"q_proj. k kproj. v v_proj. o_proj. up ppoj, down proj, gate_ projo
Other Hyperparameters
Sequence Length
Toorre 1536, Tiaget 512
Random Seeds
Precision
BF16
MMLU Evaluation:
We evaluate average accuracy on MMLU using 5-shot prompting.
GPUs:
Nvidia A10 with 24GB VRAM.
21

---

B.1.5
Llama-7b GSM8k
For our experiments using the Llama-7b model finetuned on the GSM8K training dataset, we use following setup training with AdamW
Training Algorithm Details
Learning Rates
1 10
x
10
1x
10 2 10 4 10 7 10 1 10-3
81
0.9
B2
0.999
E
1 10
LR Schedule
Linear with Warmup Ratio 0.03
Weight Decay
0.0
Train Batch Size
16
Number of Epochs
LoRA Hyperparameters
LoRA Rank
LoRA a
LoRA Dropout
0.0
Target Modules
"q_pro), k kproj.. v v_proj. o_proj. up ppoj, down proj, gate_ projo
Other Hyperparameters
Sequence Length
Toorre 1536, Tiaget 512
Random Seeds
Precision
BF16
GPUs:
Nvidia A10 with 24GB VRAM.
B.2
Additional Exps
22

---

Width 64
Width 256
Ini[A]
IaRiB]
[nit]B]
1-777-00 0-6.7e-02
7.4c
6.9c /p 02
5.5e
A2z
Az
[BAZ
PBA
1- 6e.04, q- 5-5.6e
z= 6.5e 04, p-4.7e
1= 4.0c 04, g= 1:6.6
1- 3.9c 04, n= 03
2
2
1
1
100
50 step
100
100
50
100
step
stop
step
Width 512
Width = 1024
Init[A]
Ini[6]
Init[A]
[nit
1e-44, q- p-1.200 1.2600
3.0e-04, n 4----33
3.Se-04, gmd Re-03
Se- 04, 2=2 0s
3
2
BA4
2
8A
0
0
2.5e-04, Q...0e-3
4.4-04, g=1.4e-03
zm 2.5e-04, ge-30e-03
St- mm2.mm-22
3
3
2
2
1
1
100
slep
step
Width = 2048
Width = 4096
IniiAA]
[nit[B]
== 3. 4t 04, q= Q3
== 3.1e-06, 04, 2.0e-03
== 4.1e-04, 2= 0..8e-03
== 4.7e 04, n= 04
Az
Azz
[AZZ
[8Az]
2-
2
1- 1-2.7604, g=1.76.03
3.76.04, 0=1.26-03
1.8c-04, p=1.7e-03
4.1e-044 n=2.8604
2-
2
2
0
50 stop
100
50 stop
100
50 stop
100
0
50 step
100
Figure Same as Figure 3 with differents widths.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction we clearly state the main contribution of the paper which is to compare the difference between the natural initializations for LoRA and to conclude using theoretical justification based on large width theory and extensive experiments that one the superior choice. In our theoretical sections we provide rigorous statements with clearly stated assumptions which provide intuition for the main claims. We provide experiments on synthetic tasks which give fine- ine-grained support for the theoretical claims and show that for several real-world LLM tasks the main claims are supported, illustrating the generality and relevance of our results. Limitations are discussed in the final section.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a comprehensive discussion of the limitations in Section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are clearly stated and referenced in the theorem statement. The main intuitions for the results are provided in the main paper and complete proofs are provided in the appendix.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: All details to fully reproduce the experiments are provided in the main pa- per combination with the Appendix. Experiments were performed on standard public datasets using standard public libraries. The algorithmic complexity of the experiments is fairly minimal involves just adjusting hyperparameters.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The validity of the experiments should already be believable without code, but we plan to fully release code soon. The main claims should already be easily reproducible using standard libraries.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This is provided in Appendix B.
7. Experiment Statistical Significance
24

---

Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We average all results multiple random seeds and plot both the average and I-sigma error bars shaded in the plots.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com-
puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the compute resource details in Appendix B.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https pentpa:/nneuee http://eeerii..eeee https///pppppcttacccccccceeeeee.
Answer: [Yes]
Justification: This paper does not meet any of the concerns for potential harms.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is concerned with abstract problem of choosing appropriate initialization in an extremely general setting and is not closely tied to any societal concerns.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: There are no such risks for this paper.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: There are no licensing concerns.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: No new assets are created.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
25

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
26


Reviews:
[{"review_id": "1Xmd6zDxVo", "reviewer": "Reviewer_2Ni1", "summary": "Finetuning large language models has become a common technique among practitioners, however due to the memory footprint of practically viable LLMs, there is a dire need of techniques that allow memory-lightweight finetuning (PEFT). Among those techniques LoRA has gained immense popularity. In LoRA a dense matrix is modified via addition of a low-rank update consisting of matrices A and B. In order for this update to initially conserve the model’s outputs while preserving the ability to learn, exactly one of the matrices needs to be initialized to zeros.\n“The Impact of Initialization on the Finetuning Dynamics in LoRA” proposes that the B matrix should be initialized with zeros. It hypothesizes that the reason for that is that this initialization (Init[A]) allows using greater learning rate without sacrificing stability.\nThe main points of the paper are supported by theoretical derivations stemming from the perspective of infinite-width networks. Furthermore, experimental evaluations serve as proof of the soundness of the theoretical approach and as practical justification of the method.", "questions": "- There are other works that try to improve LoRA’s performance (LoRA+) or are inspired by it (VeRA, AdaLoRA, etc.). Do you believe your work could be used with those techniques?\n- Do you feel that there is some simple, informal intuition suggesting why the Init[A] outperforms Init[B]?\n- AdamW seems to be among the most popular optimizers, however the paper focuses on rigorous derivation in case of SignSGD claiming that the result should extend to the typical choice of the Adam optimizer. Even though the authors claim they are using the AdamW optimizer in the experiments, they set weight decay = 0, effectively replacing AdamW with Adam. Would the method perform as well while using weight decay? Would it be justified theoretically the same way Adam is?\n- Can you justify the fixed hyperparameters’ values?\n- Can you explain the toy model setup more clearly? As I understand it now, the same model is being adapted and is used to create the dataset which does not seem to make sense.", "limitations": "- I do not believe the authors have sufficiently explained the limitations of the fixed hyperparameters in the evaluations", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Mathematical support combined with experimental results\n- Potentially significant due to how widespread LoRA adoption is and due to how easy the technique is to implement\n- The paper is quite original - even though LoRA is a popular technique and it is a fundamental question whether the A or the B matrix should be initialized with zeros, this work seems to provide the first attempt at answering it.", "weaknesses": "- There is virtually no discussion of why the training hyperparameters are set to those exact values, in particular with regard to the number of epochs and weight decay = 0. I suspect changing those HPs might mitigate the need to choose the initialization schema so carefully. Also the r parameter seems to be chosen arbitrarily.\n- The gains provided by the method seem to be small and would be better visualized if they were put in the perspective of comparison with other methods such as differing r or even, if possible, full finetuning.\n- The Toy Model setup is not explained clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We believe there are some misunderstandings and we take this oppotunity to address them. \n\n1) **Connection to other LoRA variants**: We emphasize that we do not introduce a new method in this paper; rather, we study the finetuning dynamics of the original LoRA method [1] for two natural initializations: init[A] (set A to random and B to zero) and init[B] (set B to random and A to zero). The impact of initialization on other LoRA variants (e.g., LoRA+, VeRA, AdaLoRA) is an interesting question but outside the scope of this paper, as stated in our conclusion and limitations section. While we haven't conducted a theoretical analysis of initialization's impact on advanced LoRA variants (e.g., QLoRA, VeRA, LoRA+, AdaLoRA), we can provide some intuition: For QLoRA, we expect similar results since the only difference from LoRA is the quantization step. Our analysis should remain fundamentally the same. For LoRA+, the situation is more complex as it involves choosing different learning rates for A and B. While we can't definitively state the outcomes, our preliminary results suggest that the optimal ratio in LoRA+ is affected by the choice of initialization (init[A] vs init[B]). As the reviewer might guess from this intuition, the impact of initialization on advanced LoRA methods will depend on the method itself. We will add this discussion in the revised version.\n\n2) **Intuition on why Init[A] outperforms init[B]**: Assume LoRA rank is $r$ and the embedding dimension is $n$. LoRA features are given by $BA z$ for input vector $z \\in \\mathbb{R}^n$ (LoRA input). With init[A], B is set to zero, allowing the magnitude of $Az$ to increase significantly without causing a blow-up in LoRA features $BAz$, as the small magnitude of B compensates for the growth in $Az$. We show in Thm 1 that the maximal learning rate (that does not cause features $BAz$ to explode) scales as $\\Theta(n^{-1/2})$ in width. This is not possible with init[B], where B's magnitude is always $\\Theta(1)$ due to random initialization with scaled variance, in which case the maximal learning scales as $\\Theta(n^{-1})$ (Thm 2). We will add this discussion to the revised version.\n\n3) **AdamW vs Adam (impact of weight decay)**: We ran sweeps with weight decay $wd \\in (0, 0.05, 0.1)$ (0.1 used in the original LoRA paper). This hyperparameter did not significantly impact the results for common $r$ values (8, 64), and our conclusion that Init[A] outperforms Init[B] holds for different $wd$ values. With $wd=0$, we reproduced the results of [1] within a statistically sound confidence interval (recall that in [1], the authors used wd=0.1). We expect $wd$'s impact to be more significant for large (though generally impractical) $r$ values due to overparameterization. We will add these details to the revised version.\n\n4) **Fixed hyperparameters' values**: Our results are insensitive to $wd$ choices within the tested range. We used LoRA ranks $r=8$ (RoBERTa) and $r=64$ (LLama). For train batch size, we tested 4 and 16, selecting the best (4 for RoBERTa, 16 for LLama), aligning with Hu et al. (2021). This setup maximizes useful empirical results within our compute budget. While larger sweeps and more models would be ideal, our experiments confirm our theoretical results beyond a trivial statistical threshold.\n\n5) **Toy model setup**: In statistics, the teacher-student model is a simplified framework for studying learning. It involves a \"teacher\" model generating data and a \"student\" model learning from it. Both models are of the same nature, making learning feasible. This setup allows analytical study of learning dynamics and generalization, providing insights into more complex machine learning scenarios. It's particularly useful for understanding phenomena like overfitting, learning curves, and the relationship between model complexity and sample size.\n\n[1] Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models\n\nWe hope these answers clarify any misunderstandings. We're happy to address further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a9DN3fwBIy", "reviewer": "Reviewer_Emo5", "summary": "The paper analyzes the impact of different initialization techniques for A and B matrices in LoRA adapters. Typically, either A or B matrix is initialized with zero while the other is initialized form a Gaussian distribution. This is done so that fine-tuning starts with LoRA adapters ($A \\times B = 0$) having zero effect. However, the paper argues that it matters which of A and B initialize to 0. As per the results, $B = 0$ init allows higher optimal learning rates which leads to better overall performance. However, there is still instability during training which could be mitigated with a method like LoRA+.", "questions": "None.", "limitations": "The limitations section in the paper is adequate.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It's useful to know that the default initialization scheme in LoRA happens to be the better alternative. The experiments are convincing.", "weaknesses": "Perhaps I am not the right person to review this paper, but I don't fully understand the impact of this paper. It shows that there are two ways of initializing AB matrices and the default one that we have been using until now is the better option. Additionally, it says that the default initialization is not good enough but doesn't propose anything new. Maybe the theory part is important, but I couldn't understand it fully in the limited time I had. Having said that, I still feel that the paper has something worthwhile to contribute to the community. So, I vote to accept it for now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. We provide some details that explain the scope and contributions of our paper.\n\n1) **Scope and contributions**: As correctly noted, in low-rank adaptation (LoRA), we generally aim to initialize the model such that $BA=0$. This naturally leads to two possible initialization schemes: either initialize $A$ randomly and $B$ to zero (init[A] in our paper), or vice versa (init[B]). Prior to our study, practitioners had no clear guidance on which scheme to use, often defaulting to the one implemented in their chosen package (e.g., init[A] in the PEFT package). Our paper provides the first definitive answer to this question: init[A] is generally superior to init[B] because it enables finetuning with larger learning rates, resulting in more efficient learning.\n\n2) **Issue at extreme width**: The reviewer correctly points out that both init[A] and init[B] become problematic in extreme scenarios where the width is exceptionally large (large enough that a value of order $n^{1/2}$ causes numerical instability). However, this is an extreme regime, and current state-of-the-art models have not yet reached this threshold.\n\n\nWe hope this clarifies the scope and significance of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dWINM7ew8V", "reviewer": "Reviewer_quNJ", "summary": "This paper investigates the impact of initialization schemes on the finetuning dynamics of Low Rank Adaptation (LoRA) in large language models. The authors compare two initialization approaches: Init[A], where A is initialized randomly and B to zero, versus Init[B], where B is initialized randomly and A to zero. Through theoretical analysis and empirical experiments, they demonstrate that Init[A] allows for larger learning rates and leads to more efficient feature learning compared to Init[B], albeit at the cost of some internal instability. The paper provides a rigorous theoretical framework based on large width limits and supports the findings with experiments on toy models and real-world language tasks.", "questions": "Q1: How sensitive are the results to the choice of LoRA rank? Do the theoretical predictions and empirical findings hold across different rank values?\n\nQ2: The paper focuses on vanilla LoRA. How do the authors expect these initialization schemes to interact with variants like QLoRA or LoRA+? Would the trade-offs and recommendations change?\n\nQ3: Given the internal instability observed with Init[A], are there any potential negative consequences for downstream task performance or generalization ability? How might this instability manifest in practice (and would there be a difference depending on input domain and/or other parameters of the model + problem space)?\n\nQ4: The theoretical analysis seems to assume a single LoRA layer, for simplicity. How well do the insights generalize to more realistic scenarios with multiple LoRA layers throughout a network?", "limitations": "L1: The theoretical analysis seems to rely on assumptions about the asymptotic behavior in the large width limit. The paper could benefit from a more detailed discussion of how these assumptions may or may not hold in practical scenarios with finite-width networks.\n\nL2: The empirical evaluation, while comprehensive, is limited to a subset of NLP tasks and model architectures. Expanding the evaluation to a broader range of tasks, model sizes, and architectures would strengthen the generalizability of the findings.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1: The paper addresses an important and understudied aspect of an usually handwaved-away aspect of LoRA, namely the impact of initialization schemes. This is a relevant topic given the widespread use of LoRA for efficient adaptation of large language models.\n\nS2: The theoretical analysis is rigorous and well-grounded, using asymptotic analysis in the large width limit to derive principled insights about the dynamics of LoRA finetuning under different initialization schemes.\n\nS3: The empirical results on both toy models and real-world language tasks provide strong support for the theoretical findings, demonstrating the practical relevance of the initialization choice.\n\nS4: The paper presents a clear trade-off between feature learning efficiency and internal stability, providing nuanced insights that go beyond simply recommending one initialization over the other.\n\nS5: I would like to commend the authors for putting great amount of work on both the main document as well as the appendix. Overall not only does the paper read well and seems theoretically and experimentally sound, but the manuscript includes an excellent degree of experimental details, and it's thus likely that future work will be able to cleanly build on it.", "weaknesses": "W1: While the theoretical analysis is sound, the paper could benefit from a more intuitive explanation of why Init[A] allows for larger learning rates and more efficient feature learning. This would make the insights more accessible to some practitioners.\n\nW2: The experiments focus primarily on language models and NLP tasks. This is definitely acceptable and overall a very sound choice, however the inclusion of experiments from other domains (e.g., finetuning an image classification model, or some other architecture that relates to a vision transformer) would strengthen the experimental section significantly, and increase the overall impact of the paper in the broader NeurIPS community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and constructive comments. We address their main questions below:\n\n1) **Sensitivity to LoRA rank**: Our results hold for different rank values. For LoRA rank $r$, we used two primary values: $r=8$ (for RoBERTa) and $r=64$ (for LLama). We also conducted limited experiments with $r=4$ for RoBERTa. However, we chose to allocate more compute resources to $r=8$ to increase the number of random seeds. We designed these experiments to maximize the amount of useful empirical results within our compute budget constraints. All our experiments were conducted using the LoRA+ codebase (available on GitHub) and can be easily reproduced. To replicate vanilla LoRA, one can set the lora_plus_ratio to 1 (ensuring the same learning rate for both A and B) and choose between Init[A] or Init[B] using the use_original_lora_init argument.\n\n\n2) **Interaction with LoRA variants**: We appreciate this interesting question. While we haven't conducted a theoretical analysis of initialization's impact on advanced LoRA variants (e.g., QLoRA and LoRA+), we can provide some intuition: For QLoRA, we expect similar results since the only difference from LoRA is the quantization step. Our analysis should remain fundamentally the same.\nFor LoRA+, the situation is more complex as it involves choosing different learning rates for $A$ and $B$. While we can't definitively state the outcomes, our preliminary results suggest that the optimal ratio in LoRA+ is affected by the choice of initialization (init[A] vs init[B]).\n\n\n3) **Instability of Init[A] in practice**: As specified in the paper, with Init[A], LoRA \"internal\" features ($Az$) grow at most as $\\Theta(n^{1/2})$ with respect to width (embedding dim). This growth, although potentially problematic, is slow (in $n$) and should only become an issue for extremely large widths. In practical settings (e.g., $n \\approx 10^3$ for LLaMA 3.1 405B), this is not significantly problematic. The constant in the $\\Theta(.)$ growth term depends on the model and downstream task, which affects this growth term.\n\n4) **Generalization to multiple layers**: Multiple layer analysis presents challenges. With two or more layers, changes in LoRA features $Z_B = BAz$ are not only due to updates of $A$ and $B$ but also changes in $z$ via previous LoRA layers. This requires a more refined, layerwise definition of efficient feature learning. While the fundamental analysis should remain similar, this adds unnecessary complexity to the setup. Importantly, our empirical results with multiple LoRA layers confirm that our findings hold in these more complex scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8RDOwIb91a", "reviewer": "Reviewer_5gBr", "summary": "The paper investigates the impact on training dynamics of two initialization schemes for LoRA. For this purpose the authors investigate the asymptotic behaviour of activations and weights for LoRA adapters. The authors find that init[A] where A is intialized randomly and B is initalized with zeros leads to more efficient learning dynamics but leads to some numerical instabilities. This initialzation method is the one most commonly used for LoRA-style methods. Further, the authors found that init[B] leads to suboptimal learning dynamics but is less prone to numerical instabilities. Finally, init[A] performs better with larger learning rates than init[B], which is supported by experiments on NLU as well as NLG finetuning tasks.", "questions": "- Did the authors investigate other commonly used initialization schemes?\n- Will the authors provide code that can be used to reproduce the results in the experiments section?", "limitations": "The authors have properly addressed limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well motivated and presents theoretical analysis accompanied by empirical experiments.\nClaims are supported by theoretical analysis as well as relevant experiments. \nFurthermore experiment hyperparameters as well as hardware requirements are well documented.", "weaknesses": "**Novelty**\n\nMost of the theoretical Analysis of the LoRA finetuning dynamics in this paper have already been presented in [1]. \nThe authors should have made it more clear how their work differs from the Analysis done in [1], especially since most formulas and notation is identical. \nFurther, both of the methods evaluated in this paper have also been investigated in [1].\nThe paper would greatly benefit from a clear explanation on how the analysis in this work differs from results published in [1].\nFinally, it is not clear if the contribution of individual LoRA layers to feature learning can be derived from a setting where only a single LoRA layer is trainable while all others are frozen (as claimed in chapter 3.1).\n\n**Experimental results**\n\nSince the main gist of the paper is to investigate the learning dynamics of LoRA, it would be good to also investigate different initialization schemes aside from init[A] and init [B] prevalent in the litearture, e.g. gaussian init [2], kaiming init [3], principal components [4], etc.\nIn [1] only different learning rates for A and B have been investigated.\n\n**Significance of results**\n\nWhile the main finding that init[A] has a better optimal performance is interesting, this initialization scheme is already the common initialization scheme used by LoRA.\nTherefore, the impact of the main findings is marginal.\n\n[1] Hayou et al., LoRA+: Efficient low rank adaptation of large models., ICML 2024\n\n[2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022\n\n[3] He et al., & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification., ICCV 2015\n\n[4] Meng et al., PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models., arXiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback, but we respectfully disagree with several points and believe there are significant misunderstandings in their assessment. We address these below:\n\n1) **Initialization schemes**: The reviewer suggests investigating \"different initialization schemes aside from init[A] and init[B],\" citing Gaussian, Kaiming, and principal component initializations. We believe this is a fundamental misunderstanding of our work. Init[A] and Init[B] in our setup already use Kaiming (Gaussian) initialization. The distinction lies in whether A or B is set to zero to ensure $BA=0$ at the start. More generally, our method covers (as stated in Footnote 2 in page 3) general random distributions (not only Gaussian). \n\n2) **Connection to prior work**: The reviewer states, \"In [1] only different learning rates for A and B have been investigated.\" This is precisely our point - [1] focused on learning rates, while our work addresses the unexplored impact of initialization schemes in vanilla LoRA. While our paper uses similar tools to those used in [1], to study the learning dynamics in the large width limit, our results are *orthogonal* to those of [1]. In this paper, we study the impact of the initialization scheme in **vanilla** LoRA as introduced in [2] (same learning rate for A and B), while in [1], the authors study the impact of setting different learning rates for A and B. Moreover, note that the tools used in our paper and also in [1] are not new and are based on the theory of infinite-width networks. For instance, the same machinery was used in the Tensor Programs series (see e.g. [10]). We decided to use the $\\gamma$ notation introduced in [1] because it simplifies the message for practitioners. However, as we explained, our results are orthogonal to those of [1]. We will add this discussion to the revised version of the paper.\n\n3) **Contribution**: The reviewer claims our findings have \"marginal\" impact because init[A] is already commonly used. We respectfully agree with this argument for the following reasons:\na) It ignores the scientific value of providing the first rigorous explanation for an empirical practice.\nb) It overlooks the importance of validating existing methods, which is crucial in scientific research.\nc) It fails to recognize that our work prevents potential future misapplications of init[B], which could lead to suboptimal results.\nd) it judges our paper as a 'method' paper where the main contribution is the introduction of a novel method. This is not the case. Our paper studies theoretically and empirically an existing method, and shows which initialization is better.\n\n5) **Scope**: Multiple layer analysis presents challenges. With two or more layers, changes in LoRA features $Z_B = BAz$ are not only due to updates of $A$ and $B$ but also changes in $z$ via previous LoRA layers. This requires a more refined, layerwise definition of efficient feature learning. While the fundamental analysis should remain similar, this adds unnecessary complexity to the setup. Importantly, our empirical results with multiple LoRA layers confirm that our findings hold in these more complex scenarios.\n\n[1] Hayou et al., LoRA+: Efficient low rank adaptation of large models., ICML 2024\n\n[2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022\n\n[10] Feature Learning in Infinite-Width Neural Networks (2021). Yang and Hu.\n\nWe believe our work provides valuable, novel insights into LoRA dynamics, filling a gap in the literature. We are surprised by the low score assigned to our paper, and we hope this clarification addresses the misunderstandings raised above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the impact of initialization schemes for LoRA. The reviewers appreciated, among others, the importance of the topic, theoretical analysis, and experimental contributions. The discussion that followed convinced two skeptical reviewers to increase their ratings to 5. Consequently, the reviewers unanimously recommend acceptance; hence, I recommend accepting.
Decision: Accept (poster)