Title: The Impact of Initialization on LoRA Finetuning Dynamics
Venue: NeurIPS 2024
Accepted: yes
Keywords: Finetuning; LoRA; Large Language Models
Abstract: In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021). Essentially, to start from the pretrained model, one can either initialize $B$ to zero and $A$ to random, or vice-versa. In both cases, the product $BA$ is equal to zero at initialization, which makes finetuning starts from the pretrained model. These two initialization schemes are seemingly similar. They should in-principle yield the same performance and share the same optimal learning rate. We demonstrate that this is an *incorrect intuition* and that the first scheme (of initializing $B$ to zero and $A$ to random) on average in our experiments yields better performance compared to the other scheme. Our theoretical analysis shows that the reason behind this might be that the first initialization allows the use of larger learning rates (without causing output instability) compared to the second initialization, resulting in more efficient learning of the first scheme. We validate our results with extensive experiments on LLMs.

The Impact of Initialization on LoRA Finetuning Dynamics

Abstract

In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally introduced in Hu et al. [19]. Essentially, to start from the pretrained model as initialization for finetuning, one can either initialize B to zero and A to random (default initialization in PEFT package), or vice-versa. In both cases, the product BA is equal to zero at initialization, which makes finetuning starts from the pretrained model. These two initialization schemes are seemingly similar. They should in-principle yield the same performance and share the same optimal learning rate. We demonstrate that this is an incorrect intuition and that the first scheme (initializing B to zero and A to random) on average yields better performance compared to the other scheme. Our theoretical analysis shows that the reason behind this might be that the first initialization allows the use of larger learning rates (without causing output instability) compared to the second initialization, resulting in more efficient learning of the first scheme. We validate our results with extensive experiments on LLMs.

1 Introduction

The pretrain-finetune paradigm (e.g., [7, 9]) has revolutionized deep learning, replacing task-specific models trained from scratch with finetuning of pretrained base models. These base models, trained on generic unsupervised objectives, learn powerful features that can be rapidly adapted to downstream tasks. The most effective models are consistently the largest ones [14, 25], with state-ofthe-art models reaching hundreds of billions of parameters. While many such models are openly available (e.g., Llama by Touvron et al. [38]), full finetuning remains computationally prohibitive for most practitioners. This has led to parameter-efficient finetuning methods, including adapters [11], prompt tuning [20], and ( IA ) 3 [24].

Low Rank Adaptation (LoRA) [19] has emerged as a leading parameter-efficient method, training only low-rank adapter matrices added to pretrained weights, typically using Adam [3]. LoRA often matches or exceeds full-finetuning performance [35, 39], though it may underperform on complex generation tasks. While prior work has examined rank [31] and learning rate [44] hyperparameters, initialization schemes remain understudied. This work provides experimental and theoretical justification for choosing between seemingly equivalent initialization approaches.

In standard LoRA training, one of the two LoRA matrices is initialized with random values and the other is initialized to zero (see Section 2.1). Recently, in Meng et al. [48] the authors proposed an alternative initialization scheme to LoRA which uses the top singular vectors of the pretrained weights as opposed to a random initialization and showed improved training on several tasks. To further improve LoRA training with quantization, Li et al. [34] introduced a new method called LoftQ for computing a better initialization for quantized training [27]. However, to the best of our knowledge,

there has not been any study concerning the random initialization in vanilla LoRA. Specifically, it is not clear from prior work which of the two LoRA matrices should be initialized to be zero . Empirical results by Zhu et al. [50] suggested that the two initialization schemes mentioned above yield similar performance, but it is not clear if the learning rate was well-tuned for each initialization scheme. Our findings suggest that these two initialization schemes lead to fundamentally different finetuning dynamics, and that one of these schemes generally yields better result compared to the other.

LoRA Variations. Beyond altering the LoRA initialization scheme, there have been a series of works which try to address limitations of vanilla LoRA using different variations. To further reduce the number of trainable parameters, LoRAFA [42] freezes the A matrix which leads to small performance loss while reducing memory consumption by up to 1.4 × . The performance of this training scheme is also investigated in Zhu et al. [50]. VeRA [33] freezes random weight tied adapters and learns vector scalings of the internal adapter activations. LoRA-XS [43] initializes the A and B matrices using the SVD of the pretrained weights and trains a low-rank update of the form BRA where R is a trainable r × r matrix and B , A are fixed. NOLA [32] parametrizes the adapter matrices to be linear combinations of

Pretrained

Weights

+

Init[A]:

Init[B]:

Init

Optimal

Finetuning

Dynamics

More Efficient

Feature Learning with

some instability

Stability with

suboptimal Feature

Learning

Init[A] (generally) leads to better performance

frozen random matrices and optimizes the linear coefficients of the mixtures. VB-LORA [46] shares adapter parameters using a global vector bank. In order to improve the learning ability for more challenging finetuning tasks, Kalajdzievski [31] proposes a scaling rule for the scalar adapter multiplier to unlock increased gains with higher adapter ranks. MoRA [45] learns high-rank updates while still preserving parameter efficiency by applying hand-designed compress and decompress operations before and after a trainable adapter matrix. DoRA [47] decomposes the pretrained weight into magnitude and direction components to allow for better training dynamics.

Contributions. We study the impact of Initialization in LoRA through a theory of large width for neural networks. The core approach is to take the width of a neural network to infinity and determine how the behavior of the limit depends on the choice of the hyperparameters, such as the learning rate and initialization. This approach allows to derive principled scaling choices for these hyperparameters such that desired properties (e.g. stable feature learning) are achieved as the network size grows (see Appendix A.2 for more details). Examples of the infinite-width limit include works on initialization (e.g. He et al. [4]), and training dynamics (e.g. [21]). Examples for the depth limit include initialization strategies [6, 10, 30], and depth scaling (see e.g. [18, 23, 28, 29, 37, 41]). A similar strategy was used to derive scaling rules for the LoRA learning rate in Hayou et al. [44] (LoRA + ) that concluded that the learning rates for different LoRA matrices should be scaled differently to ensure optimal feature learning. In this work we use the same approach to provide a systematic comparison between two different random initialization schemes for vanilla LoRA finetuning (using the same learning rate for the A and B matrices). Using the notation Init[A] to refer to the case where A is initialized to random and B to zero (as in [19]) and Init[B] for the opposite, we show that Init[A] and Init[B] lead to fundamentally different training dynamics (as shown in Figure 1):

Init[A] allows the use of larger learning rates compared to Init[B]

Init[A] leads to 'internal instability' where the features Az (for some input z ) are large but LoRA output BAz is small. This form of instability allows more efficient feature learning. We identify a feature learning / stability tradeoff in this case.

Init[B] does not cause any instabilities but training is suboptimal ( B is undertrained).

Empirical results confirm the theory and show that Init[A] generally leads to better performance than Init[B] .

2 Setup and Definitions

We consider a general neural network model of the form

where x ∈ R d is the input, L ≥ 1 is the network depth, ( F l ) l ∈ [ L ] are mappings that define the layers, and W l ∈ R n × n are the hidden weights, where n is the network width , and W in , W out are input and output embedding weights. 1 This model will represent the pretrained model that will later be finetuned on some new task.

To finetune a (large) pretrained model with a limited amount of computational resources, a popular resource efficient approach is to use the LoRA finetuning method defined below.

Definition 1 (Low Rank Adapters (LoRA) from [19]) . To apply LoRA to a weight matrix W ∈ R n 1 × n 2 in the model, we constrain its update in the fine-tuning process by representing the latter with a low-rank decomposition W = W ∗ + α r BA . Here, only the weight matrices B ∈ R n 1 × r , A ∈ R r × n 2 are trainable and the original pretrained weights W ∗ remain frozen. The rank r ≪ min( n 1 , n 2 ) and α ∈ R are tunable constants.

As the width n grows, 2 the network initialization scheme and the learning rate should be adapted to avoid numerical instabilities and ensure efficient learning. For instance, the variance of the initialization weights (in hidden layers) should scale like 1 /n to prevent the pre-activations from blowing up as we increase model width n (e.g., He initialization [4]). To derive proper scaling rules, a principled approach consist of analyzing the statistical properties of key quantities in the model (e.g. second moment of the pre-activations) as n grows and then adjust the initialization variance, the learning rate, and the architecture to achieve desirable properties in the limit n → ∞ [5, 10, 13, 40]. We use this approach to study the effect of initialization on the feature learning dynamics of LoRA in the infinite-width limit. For more details about the theory of scaling of neural networks, see Appendix A.2.

Throughout the paper, we will be using asymptotic notation to describe the behaviour of several quantities as the width n grows. Note that the width n will be the only scaling dimension of neural network training which grows and all other scaling dimensions such as the LoRA rank r , number of layers L , sequence length, number of training steps, etc., will be considered as fixed. We use the following notation for the asymptotic analysis.

Notation. Given sequences c n ∈ R and d n ∈ R + , we write c n = O ( d n ) , resp. c n = Ω( d n ) , to refer to c n < κd n , resp. c n > κd n , for some constant κ > 0 . We write c n = Θ( d n ) if both c n = O ( d n ) and c n = Ω( d n ) are satisfied. For vector sequences c n = ( c i n ) 1 ≤ i ≤ k ∈ R k (for some k > 0 ), we write c n = O ( d n ) when c i n = O ( d i n ) for all i ∈ [ k ] , and same holds for other asymptotic notations. Finally, when the sequence c n is a vector of random variables, convergence is understood to be convergence in second moment ( L 2 norm).

2.1 Initialization of LoRA Adapters

The standard way to initialize trainable weights is to take an iid initialization of the entries A ij ∼ N (0 , σ 2 A ) , B ij ∼ N (0 , σ 2 B ) for some σ A , σ B ≥ 0 (this includes initialization with zeros if σ B or σ A

are set to 0 ). 3 . Due to the additive update structure of LoRA, we want to initialize the product BA to be 0 so that finetuning starts from the pretrained model [19]. This can be achieved by initializing one of the weights A and B to 0 . If both are initialized to 0 , no learning occurs in this case since this is a saddle point and the parameter gradients will remain zero. Thus, we should initialize one of the parameters A and B to be non-zero and the other to be zero. If we choose a non-zero initialization for A , then following standard initialization schemes (e.g., He Init [4], LeCun Init [1]), one should set σ 2 A = Θ( n -1 ) to ensure Ax does not explode for large n . This is justified by the Central Limit Theorem (CLT). On the other hand, if we choose a non-zero initialization for B , one should make sure that σ 2 b = Θ( r -1 ) = Θ(1) . This leaves us with two possible initialization schemes:

Init[A] : σ 2 B = 0 , σ 2 A = Θ( n -1 ) (default initialization in LoRA [19]).

Init[B] : σ 2 B = Θ( r -1 ) = Θ(1) , σ 2 A = 0 . 4

These two initialization achieve the goal of starting finetuning from the pretrained model. A priori, it is unclear if there is a material difference between the two initialization schemes. Surprisingly, as we will show later in this paper, these two initialization schemes lead to fundamentally different training dynamics when model width is large.

2.2 LoRA Features

Notation. For a given LoRA layer in the network, we use Z to denote the input to that layer and ¯ Z for the output after adding the pretrained weights. More precisely, we can write the layer operation as ¯ Z = W ∗ Z + α r BA Z.

Our main analysis relies on a careful estimation of the magnitude of several quantities involving LoRA features . Let us first give a formal definition.

Definition 2 (LoRA Features) . Given a general neural architecture and a LoRA layer (Definition 1), we define LoRA features ( Z A , Z B ) as Z A = A Z, and Z B = BZ A = BA Z. At fine-tuning step t , we use the superscript t to denote the value of LoRA features Z t A , Z t B , and the subscript t to denote the weights A t , B t .

3 LoRA Finetuning Dynamics in the Large Width Limit

We fix the LoRA rank r throughout the analysis and examine the finetuning dynamics in the limit of large width. This setup aligns well with practical scenarios where the rank is much smaller than the width (i.e., r ≪ n ). Typically, for Llama models the rank r is generally of order 2 k for k ∈ { 2 , . . . , 6 } , and model width n is generally larger than 2 12 . We will refer to a layer of the network to which LoRA is applied (see Definition 1) as a LoRA layer . For the theoretical analysis, we adopt a simplified setting that facilitates a rigorous yet intuitive derivations of the results.

3.1 Simplified Setting

The following simplified setup was considered in Hayou et al. [44] to derive asymptotic results concerning the learning rates in LoRA. We use the same setup in our analysis to investigate the impact of initialization.

Finetuning Dataset. We assume that the dataset used for finetuning consists of a single datapoint ( x, y ) , 5 and the goal is to minimize the loss calculated with the model with adjusted weights W ∗ + BA for all LoRA layers (here θ = { A,B, for all LoRA layers in the model } ). Z t is the input to the LoRA layer, computed with data input x . Similarly, we write d ¯ Z t to denote the gradient of the loss function with respect to the layer output features ¯ Z evaluated at data point ( x, y ) .

Single LoRA Module. Given a LoRA layer, LoRA feature updates are not only driven by the change in the A,B weights, but also the changes in Z , d ¯ Z which are updated as we finetune the model (assuming there are multiple LoRA layers). To isolate the contribution of individual LoRA layers to feature learning, we assume that only a single LoRA layer is trainable and all other LoRA layers are frozen. 6 For this LoRA layer the layer input Z is fixed and does not change with t , whereas d ¯ Z changes with step t (because ¯ Z t = ( W ∗ + α r B t A t ) Z). After step t , Z B is updated as follows

As discussed in Hayou et al. [44], the terms δ 1 t , δ 2 t represent 'linear' feature updates that we obtain if we fix one weight matrix and only train the other. The third term δ 3 t represents the 'multiplicative' feature update which captures the compounded update due to updating both A and B .

3.2 Stability and Feature Learning

Hayou et al. [44] introduced the notion of stability of LoRA features as width grows. We introduce here a slightly more relaxed notion of stability.

Definition 3 (Feature Stability) . We say that LoRA finetuning is stable if for all LoRA layers in the model, and all training steps t , we have Z , Z B = O (1) , as the width n goes to infinity.

Here, feature stability implies that LoRA output Z B remains bounded (in L 2 norm) as width grows. To achieve such stability, hyperparameters (initialization, learning rate) should be scaled as n grows. We will show that the dependence of the optimal learning rate on n is highly sensitive to the choice of initialization ( Init[A] or Init[B] ).

Note that feature stability also requires that Z = O (1) which is directly related to pretraining dynamics since it depends on some pretrained weights W ∗ . We assume that pretraining parameterization (how initialization and learning rate are parametrized w.r.t width) ensures this kind of stability (see Appendix A for more details). 7

As discussed above, feature updates are driven by the terms ( δ i t ) i ∈{ 1 , 2 , 3 , } . As n grows, these feature updates might become trivial (i.e. vanish as n →∞ ) or unstable (i.e. grows unbounded). To avoid such scenarios, we want to ensure that ∆ Z B = Θ(1) . Such conditions are the main ideas behind µ P [26] and Depthµ P [41], which are network parametrizations that ensure stability and feature learning in the large width and depth limits for pretraining. We recall this definition from [44].

Definition 4 (Feature Learning) . We say that LoRA finetuning induces stable feature learning in the limit of large width if the dynamics are stable (Definition 3), and for all finetuning steps t , we have ∆ Z t B def = Z t +1 B -Z t B = Θ(1) .

∆ Z B is the sum of the terms δ i t 's (Equation (2)). To achieve optimal feature learning, we want to ensure that δ 1 t = Θ(1) and δ 2 t = Θ(1) which means that both weight matrices A and B are efficiently updated and contribute to the update in Z B . An intuitive explanation is provided in Appendix A.1. This leads us to the following definition of efficient learning with LoRA.

Definition 5 (Efficient Learning with LoRA) . We say that LoRA fine-tuning is efficient if it is stable (Definition 3), and for all LoRA layers in the model, and all fine-tuning steps t > 1 , we have

Next, we introduce the γ -operator, an essential tool in our analysis of the large width dynamics.

3.3 Introduction to the γ -operator

In the theory of scaling, one usually tracks the asymptotic behavior of key quantities as we scale some model ingredient. For instance, if we scale the width n of a neural network, we are interested in quantifying how certain quantities in the network behave as n grows. This is a standard approach for (principled) model scaling and it has so far been used to derive scaling rules for initialization [5], activation function [10], network parametrization [41], amongst other things.

With Init[A] and Init[B] , initialization weights are of order Θ( n -β ) for some β ≥ 0 . Assuming that the learning rate also scales polynomialy with n , it is straightforward that preactivations, gradients, and weight updates are all asymptotically polynomial in n . Note that this is only possible because all neural computations consists of sums of Θ( n α ) terms, where typically α ∈ { 0 , 1 } . For instance, when calculating the features A Z, each entry is a sum of n terms, while when calculating BZ A , each entry is a sum of r terms ( r fixed as n goes to infinity). This is true for general neural computation that can be expressed as Tensor Programs [15].

Consequently, for some quantity v in the computation graph, it is natural to track the exponent that determines the asymptotic behavior of v with respect to n . We write v = Θ( γ [ v ]) to capture this polynomial dependence. Elementary operations with the γ -operator include: 8

Multiplication. Given two real-valued variables v, v ′ , we have γ [ v × v ′ ] = γ [ v ] + γ [ v ′ ] .

Addition. Given two real-valued variables v, v ′ , we generally have γ [ v + v ′ ] = max( γ [ v ] , γ [ v ′ ]) . The only case where this is violated is when v ′ = -v . This is generally a zero probability event if v and v ′ are random variables that are not perfectly (negatively) correlated, which is the case in most situations where we make use of this formula. See Appendix A.3 for discussion.

We have now introduced all required notions for the subsequent analysis. For better readability, we defer all the proofs to the appendix.

3.4 Recursive formulas

Using the γ -operator, we can track the asymptotic behavior of the finetuning dynamics as model width n grows. At finetuning step t , the weights are updated as follows

where g A , g B are processed gradients (e.g. normalized gradients with momentum as in AdamW). We assume that the gradients are processed in a way that makes their entries Θ(1) . This is generally satisfied in practice (with Adam for instance) and has been considered in [40] to derive the µ -parametrization for general gradient processing functions. From this, we obtain the following recursive formulas for γ [ Z t A ] and γ [ B t ] , which characterizes their behavior in the large width limit.

Lemma 1 (Informal) . For t fixed, the asymptotic dynamics of Z t A and B t follow the recursive formula

The formal proof of Lemma 1 is provided in Appendix A and relies on Assumption 1 which fairly represents practical scenarios (see Appendix A for a detailed discussion). Lemma 1 captures the change in asymptotic behavior of quantities Z t A and B t as width grows. Naturally, the dynamics depend on the the initialization scheme which lead to completely different behaviors as we show in the next two results.

3.5 Init[A] leads to more efficient feature learning but suffers 'internal' instability

Next, we provide a precise characterization of stability and feature learning when using Init[A] .

Theorem 1 (Informal) . For t fixed, with Init[A] and learning rate η , we have

Stability: Z t B = O (1) if and only if γ [ η ] ≤ -1 / 2 .

Feature Learning: ∆ Z t B = Θ(1) if and only if γ [ η ] = -1 / 2 . In this case, we also have δ 1 t , δ 2 t = Θ(1) (efficient feature learning, Definition 5).

Moreover, 'internal' instability ( Z t A = Ω(1) ) occurs when γ [ η ] ∈ ( -1 , 1 / 2] .

With Init[A] , the maximal learning rate 9 that does not lead to instability in Z B scales as Θ( n -1 / 2 ) . This can be seen as an asymptotic form of the edge of stability phenomenon [17] where if we increase the learning rate beyond some level, instability occurs. Interestingly, in this case (i.e. with Θ( n -1 / 2 ) learning rate) the features are efficiently updated (Definition 5). However, this comes with caveat: the features Z t A grow as Θ( n 1 / 2 ) which can potentially cause numerical instabilities. We call this phenomenon internal instability : only the features Z A (internal LoRA features) grows, LoRA output Z B remains Θ(1) in this case.

The fact that Θ( n -1 / 2 ) is the maximal learning rate that does not cause instability in Z B does not mean it is the optimal learning rate. As the width n grows, this internal instability in Z A will become more and more problematic. Intuitively, we expect that a trade-off appears in this case: the optimal learning rate (found by grid search) to be larger than Θ( n -1 ) but smaller than Θ( n -1 / 2 ) , i.e. the network will try to achieve a balance between optimal feature learning ( γ [ η ] = -1 / 2 ) and internal stability Z t A = Θ(1) ( γ [ η ] = -1 ). We verify this empirically in the next section.

3.6 Init[B] leads to suboptimal feature learning with internal stability

In the next result, we show that the maximal learning rate allowed with Init[B] is different from that with Init[A] , leading to completely different dynamics.

Theorem 2 (Informal) . For t fixed, with Init[B] , we have

Stability: Z t B = O (1) if and only if γ [ η ] ≤ -1 .

Feature Learning: ∆ Z t B = Θ(1) if and only if γ [ η ] = -1 .

Moreover, efficient feature learning cannot be achieved with Init[B] for any choice of learning rate scaling γ [ η ] (that does not violate the stability condition). More precisely, with Θ( n -1 ) learning rate, the limiting dynamics (when n →∞ ) are the same if B was not trained and A is trained.

With Init[B] , the maximal learning rate (that does not violate stability) scales as Θ( n -1 ) (for any ϵ > 0 , a learning rate of Θ( n -1+ ϵ ) leads to Z B = Ω(1) ).

Because of this bound on the maximal learning rate, no internal instability occurs with Init[B] . In this case, feature learning is suboptimal since the B weight matrix is undertrained in the large width limit ( δ 2 t → 0 ).

Conclusions from Sections 3.5 and 3.6. The results of Theorem 1 and Theorem 2 suggest that Init[A] allows the use of larger learning rates compared to Init[B] , which might lead to better feature learning and hence better performance at the expense of some internal instability. Here, 'larger' learning rate should be interpreted in asymptotic terms: with Init[A] the maximal learning rate that does not cause instability satisfies γ [ η ] = -1 / 2 . With Init[B] , we have γ [ η ] = -1 instead. Note that because of the constants in Θ( n β ) learning rates (for some β ) , the optimal learning rate with Init[A] is not systematically larger than Init[B] for finite width . However, as width grows, we will see that it is case.

Another important insight from this analysis is that with both initializations, the dynamics are suboptimal in the limit: internal instability with Init[A] and undertraining of B with Init[B] . 10 We will later discuss possible solutions to this behavior.

0.0

0.5

1.0

1.5

Init[A]

---------------------------------

=

4.2e-04,

=1.6e-02

0.0

0.5

1.0

1.5

Init[B]

---------------------------------

=

5.3e-04,

=8.1e-03

|

Az

|

|

BAz

|

0

50

100

step

0.0

0.5

1.0

1.5

=

3.0e-04,

=2.3e-02

0

50

100

step

0.0

0.5

1.0

1.5

=

3.0e-04,

=1.2e-02

0

2

4

6

Init[A]

---------------------------------

=

2.4e-04,

=1.4e-03

0

2

4

6

seed-1

Init[B]

---------------------------------

=

3.7e-04,

=4.8e-04

|

Az

|

|

BAz

|

0

50

100

step

0

2

4

=

2.9e-04,

=1.2e-03

0

50

100

step

0

2

4

=

3.6e-04,

=9.8e-04

Width = 8192

3.7 Toy Model

To validate our theory in a controlled setting, we consider the following simple model:

where W in ∈ R n × d , W h ∈ R n × n , W out ∈ R 1 × n , and B,A ⊤ ∈ R r × n .

We generate synthetic data from the teacher model using the following config: d = 5 , r teacher = 20 , n = 1000 , N = 1000 (train data size), and N test = 100 (test data size). The weight W teacher in , W teacher out , A teacher , and B teacher are randomly initialized, and W teacher h = 0 . 11 We train student models with d = 5 , r = 4 , and varying widths n ∈ { 2 k , k = 7 , . . . , 13 } . 12

Optimal Learning Rate. We finetune model (4) on synthetic data generated from the teacher model. In Figure 2, we show the optimal learning rate when using either Init[A] or Init[B] to initialize the fine-

2 8

2 9

2 10

2 11

2 12

2 13

n

2

12

2

11

2

10

2

9

2

8

2

7

2

6

*

Stability <-> Feature Learning

Tradeoff

Init[A]

C

1

n

1/2

Init[B]

C

2

n

1

tuning, as a function of width n . For n ≫ 1 (typically n ≥ 2 9 ), the optimal learning rate with Init[A] is larger than the optimal learning rate with Init[B] . This is in agreement with the theoretical results obtained in Theorem 1 and Theorem 2 which predict asymptotic maximal learning rates (that satisfy the stability condition) of Θ( n -1 / 2 ) and Θ( n -1 ) respectively.

With Init[A] , we observe the stability/feature learning trade-off for large n . The optimal learning rate with Init[A] in this regime (e.g. n = 2 13 ) is smaller than the maximal theoretical learning

rate n -1 / 2 that achieves optimal feature learning (Theorem 1). Here, the model seems to balance the internal instability that occurs in the Z A features with feature learning and thus favors smaller learning rates: the optimal learning rates is smaller than Θ( n -1 / 2 ) and larger than Θ( n -1 ) .

Internal Instability and Feature Learning. Figure 3 shows the average magnitude of Z A and Z B for Init[A] and Init[B] for widths n ∈ { 128 , 8192 } . With Init[A] , the magnitude of Z A features seem to grow with width, hence trading off internal stability for more efficient feature learning. This behavior is consistent across random seeds as shown in the figure, and as further confirmed by experiments in Appendix B. The train loss is consistently smaller with Init[A] , which can be explained by the fact that Init[A] allows more efficient feature learning at the cost of some internal instability. This flexibility cannot be achieved with Init[B] . Note also that Z B features tends to get smaller with n with Init[A] as predicted by theory: the trade-off between internal instability and feature learning implies that η ∗ = o ( n -1 / 2 ) , which implies that Z t B = o (1) , i.e. the Z B features vanish as width grows. While this might problematic, it only becomes an issue when the width is extremely large: if the optimal learning rate scales as Θ( n -β ) for some β ∈ (1 / 2 , 1) (so that the learning rate is between Θ( n -1 ) and Θ( n -1 / 2 ) , balancing internal instability and efficient feature learning), the LoRA output feature scales as Z B = B t A t Z = Θ( n -β +1 ) . Therefore, if β ≈ 0 . 7 for instance, the vanishing rate of LoRA output feature is Z B ≈ Θ( n -0 . 3 ) which is slow given the order of magnitude of width in practice (for n = 2 12 , we have n -0 . 3 ≈ 0 . 08 ).

4 Experiments with Language Models

Our theoretical results from earlier provides a detailed asymptotic analysis of the finetuning dynamics when LoRA modules are initialized with Init[A] or Init[B] . The main conclusions are that Init[A] generally leads to more efficient feature learning (which can be justified by the fact that optimal learning rate is larger when using Init[A] compared to when using Init[B] ). To provide evidence of this claim on real-world tasks, we use LoRA to finetune a set of language models on different benchmarks. Details about the experimental setup and more empirical results are provided in Appendix B. We use LoRA + code [44] for our experiments (available at https://github.com/nikhil-ghosh-berkeley/loraplus ).

4.1 GLUE tasks with RoBERTa

The GLUE benchmark (General Language Understanding Evaluation) consists of several language tasks that evaluate the understanding capabilities of langugage models [8]. Using LoRA, we finetune Roberta-large from the RoBERTa family [12] on MNLI, SST2, and QNLI tasks with varying learning rates η and initialization schemes ( Init[A] or Init[B] ). We use the same experimental setup of [19] for Roberta-Large to compare our results with theirs (see Appendix B for more details).

10

5

10

4

10

3

0.4

0.5

0.6

0.7

0.8

0.9

Test Acc

MNLI

---------------------------------

Init[A]:

Acc

=90.69,

* =8.0e-05

Init[B]:

Acc

=89.47,

* =1.0e-05

Init[B]

Init[A]

10

5

10

4

10

3

0.5

0.6

0.7

0.8

0.9

Test Acc

SST2

---------------------------------

Init[A]:

Acc

=96.67,

* =1.6e-04

Init[B]:

Acc

=96.44,

* =4.0e-05

Init[B]

Init[A]

10

5

10

4

10

3

0.800

0.825

0.850

0.875

0.900

0.925

0.950

Test Acc

QNLI

---------------------------------

Init[A]:

Acc

=95.09,

* =8.0e-05

Init[B]:

Acc

=93.61,

* =8.0e-05

Init[B]

Init[A]

The results in Figure 4 are aligned with our theory: we observe that Init[A] generally leads to better performance, and the optimal learning rate with Init[A] is generally larger than with Init[B] . Models initialized with Init[A] match the performances reported in [19], while those initialized with Init[B] generally underperform that baseline. For MNLI task (the hardest one amongst the

three tasks), we observe a significant difference in the best test accuracy (over 3 random seeds) with 90 . 69 with Init[A] and 89 . 47 with Init[B] . We also observe that for MNLI, the optimal learning rate with Init[A] ( η ∗ = 8 e-5) is much larger than the optimal learning rate with Init[B] ( η ∗ = 1 e-5), which aligns with our theoretical predictions. However, note that for QNLI for instance (an easier task), while the optimal test accuracy is significantly better with Init[A] , the optimal learning rate (from the grid search) is the same for Init[A] and Init[B] . There are many possible explanations for this: 1) the width is not large enough in this case to see the gap between optimal learning rates (for RoBERTa-Large, the width is n = 2 10 ) 2) The constants in Θ( n -1 ) are Θ( n -1 / 2 ) are significantly different in magnitude due to dependence on finetuning task. We notice similar behaviour with LLama experiments below. A precise analysis of this observation is beyond the scope of this paper, we leave it for future work.

4.2 Llama

10

5

10

4

10

3

7.2

7.4

7.6

7.8

8.0

8.2

8.4

Test Perplexity

TinyLlama on WikiText-2

---------------------------------------------------

init[A]: PPL = 7.089,

= 7.0e-04

init[B]: PPL = 7.151,

= 4.0e-04

init[B]

init[A]

10

4

10

3

0.26

0.28

0.30

0.32

0.34

0.36

0.38

0.40

MMLU Accuracy

---------------------------------------------------

init[A]: Acc = 0.410,

= 4.0e-04

init[B]: Acc = 0.406,

= 4.0e-04

init[B]

init[A]

10

5

10

4

10

3

0.12

0.14

0.16

0.18

0.20

0.22

0.24

0.26

Test Accuracy

Llama-7b on GSM8k

---------------------------------------------------

init[A]: Acc = 0.260,

= 1.0e-03

init[B]: Acc = 0.239,

= 1.0e-03

init[B]

init[A]

To further validate our theoretical findings on more modern models and datasets, we report the results of finetuning the Llama-7b model [38] on the Flan-v2 dataset [36] and the GSM8k dataset [16], and finetuning the TinyLlama model [49] on WikiText-2 using LoRA. Each trial is averaged over two seeds and the shaded region indicates one standard error. In the left panel of Figure 5 we see that when finetuning TinyLlama using LoRA the optimal learning rate using Init[A] is larger than with Init[B] and the corresponding test perplexity is lower. Similarly, for the center panel of Figure 5, when finetuning the Llama-7b model on Flan-v2, the optimal learning rates for Init[A] and Init[B] are the same (for the learning rate grid we used), but the the optimal MMLU accuracy for Init[A] is slightly higher than for Init[B] . For learning rates close to the optimal choice, the accuracy using Init[A] is generally higher than for Init[B] . An analagous result holds for the GSM8k dataset as shown in the rightmost panel of Figure 5. More details about this setting are provided in Appendix B.

5 Conclusion and Limitations

We showed that LoRA dynamics are highly sensitive to initialization. Init[A] is associated with larger optimal learning rates, compared to Init[B] . Larger learning rates typically result in better performance, as confirmed by our empirical results. Note that this is a zero-cost adjustment with LoRA finetuning: we simply recommend using Init[A] instead of Init[B] .

One limitation of our work is that we only define feature learning via the magnitude of feature updates in the limit of large width. In this way, our definition of feature learning is data-agnostic and therefore no conclusion about generalization can be obtained with this analysis. The constants in Θ( . ) asymptotic notation naturally depend on the data (the finetuning task) and therefore such data-agnostic approach does not allow us to infer any information about the impact of the data on the finetuning dynamics.

More importantly , our results indicate that both initialization schemes lead to suboptimal scenarios, although Init[A] allows more efficient feature learning. In both cases, instability and/or suboptimal feature learning present fundamental issues, which can potentially be mitigated by approaches such as LoRA + [44]. Understanding the interaction of LoRA + and related efficiency methods with the initialization scheme is an important question for future work.

References

Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. 'Efficient backprop'. In: Neural networks: Tricks of the trade . Springer, 2002, pp. 9-50.

L. Yang, S. Hanneke, and J. Carbonell. 'A theory of transfer learning with applications to active learning'. In: Machine learning 90 (2013), pp. 161-189.

D. P. Kingma and J. Ba. 'Adam: A method for stochastic optimization'. In: arXiv preprint arXiv:1412.6980 (2014).

K. He, X. Zhang, S. Ren, and J. Sun. 'Deep residual learning for image recognition'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2016, pp. 770778.

S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. 'Deep Information Propagation'. In: International Conference on Learning Representations . 2017.

S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep Information Propagation . 2017. arXiv: 1611.01232 [stat.ML] .

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. 'Improving language understanding by generative pre-training'. In: (2018).

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding . 2018. arXiv: 1804. 07461 [cs.CL] .

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding'. In: arXiv preprint arXiv:1810.04805 (2019).

S. Hayou, A. Doucet, and J. Rousseau. 'On the Impact of the Activation function on Deep Neural Networks Training'. In: Proceedings of the 36th International Conference on Machine Learning . Ed. by K. Chaudhuri and R. Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, Sept. 2019, pp. 2672-2680.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. 'Parameter-efficient transfer learning for NLP'. In: International Conference on Machine Learning . PMLR. 2019, pp. 2790-2799.

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach . 2019. arXiv: 1907.11692 [cs.CL] .

G. Yang. 'Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation'. In: arXiv preprint arXiv:1902.04760 (2019).

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. 'Scaling laws for neural language models'. In: arXiv preprint arXiv:2001.08361 (2020).

G. Yang. 'Tensor programs iii: Neural matrix laws'. In: arXiv preprint arXiv:2009.10685 (2020).

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. 'Training verifiers to solve math word problems'. In: arXiv preprint arXiv:2110.14168 (2021).

J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. 'Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability'. In: International Conference on Learning Representations . 2021.

S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau. 'Stable ResNet'. In: Proceedings of The 24th International Conference on Artificial Intelligence and Statistics . Ed. by A. Banerjee and K. Fukumizu. Vol. 130. Proceedings of Machine Learning Research. PMLR, 13-15 Apr 2021, pp. 1324-1332.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. 'LoRA: Low-Rank Adaptation of Large Language Models'. In: arXiv preprint arXiv:2106.09685 (2021).

B. Lester, R. Al-Rfou, and N. Constant. 'The power of scale for parameter-efficient prompt tuning'. In: arXiv preprint arXiv:2104.08691 (2021).

G. Yang and E. J. Hu. 'Tensor programs iv: Feature learning in infinite-width neural networks'. In: International Conference on Machine Learning . PMLR. 2021, pp. 11727-11737.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training Compute-Optimal Large Language Models . 2022. arXiv: 2203.15556 [cs.CL] .

M. Li, M. Nica, and D. Roy. 'The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization'. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 10795-10808.

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. 'Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 1950-1965.

J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. 'Emergent abilities of large language models'. In: arXiv preprint arXiv:2206.07682 (2022).

G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. 'Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer'. In: arXiv preprint arXiv:2203.03466 (2022).

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. 'QLoRA: Efficient Finetuning of Quantized LLMs'. In: arXiv preprint arXiv:2305.14314 (2023).

S. Hayou. 'On the infinite-depth limit of finite-width neural networks'. In: Transactions on Machine Learning Research (2023). ISSN: 2835-8856.

S. Hayou and G. Yang. 'Width and Depth Limits Commute in Residual Networks'. In: Proceedings of the 40th International Conference on Machine Learning . Ed. by A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett. Vol. 202. Proceedings of Machine Learning Research. PMLR, 23-29 Jul 2023, pp. 12700-12723.

B. He, J. Martens, G. Zhang, A. Botev, A. Brock, S. L. Smith, and Y. W. Teh. Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation . 2023. arXiv: 2302.10322 [cs.LG] .

D. Kalajdzievski. 'A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA'. In: arXiv preprint arXiv:2312.03732 (2023).

S. A. Koohpayegani, K. Navaneet, P. Nooralinejad, S. Kolouri, and H. Pirsiavash. 'NOLA: Networks as linear combination of low rank random basis'. In: arXiv preprint arXiv:2310.02556 (2023).

D. J. Kopiczko, T. Blankevoort, and Y. M. Asano. 'VeRA: Vector-based Random Matrix Adaptation'. In: arXiv preprint arXiv:2310.11454 (2023).

Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. 'Loftq: Lora-finetuning-aware quantization for large language models'. In: arXiv preprint arXiv:2310.08659 (2023).

H. Liu, C. Li, Y. Li, and Y. J. Lee. 'Improved baselines with visual instruction tuning'. In: arXiv preprint arXiv:2310.03744 (2023).

S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al. 'The flan collection: Designing data and methods for effective instruction tuning'. In: arXiv preprint arXiv:2301.13688 (2023).

L. Noci, C. Li, M. B. Li, B. He, T. Hofmann, C. Maddison, and D. M. Roy. The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit . 2023. arXiv: 2306. 17759 [stat.ML] .

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan,

P. Xu, Z. Yan, I. Zarov, Y . Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. 'Llama 2: Open Foundation and Fine-Tuned Chat Models'. In: arXiv preprint arXiv:2307.09288 (2023).

Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. 'How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources'. In: arXiv preprint arXiv:2306.04751 (2023).

G. Yang and E. Littwin. 'Tensor programs ivb: Adaptive optimization in the infinite-width limit'. In: arXiv preprint arXiv:2308.01814 (2023).

G. Yang, D. Yu, C. Zhu, and S. Hayou. 'Tensor Programs VI: Feature Learning in InfiniteDepth Neural Networks'. In: arXiv preprint arXiv:2310.02244 (2023).

L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li. 'Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning'. In: arXiv preprint arXiv:2308.03303 (2023).

K. Bałazy, M. Banaei, K. Aberer, and J. Tabor. 'LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters'. In: arXiv preprint arXiv:2405.17604 (2024).

S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient Low Rank Adaptation of Large Models . 2024. arXiv: 2402.12354 [cs.LG] .

T. Jiang, S. Huang, S. Luo, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, Q. Zhang, D. Wang, et al. 'MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning'. In: arXiv preprint arXiv:2405.12130 (2024).

Y. Li, S. Han, and S. Ji. 'VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks'. In: arXiv preprint arXiv:2405.15179 (2024).

S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. 'DoRA: Weight-Decomposed Low-Rank Adaptation'. In: arXiv preprint arXiv:2402.09353 (2024).

F. Meng, Z. Wang, and M. Zhang. 'PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models'. In: arXiv preprint arXiv:2404.02948 (2024).

P. Zhang, G. Zeng, T. Wang, and W. Lu. 'Tinyllama: An open-source small language model'. In: arXiv preprint arXiv:2401.02385 (2024).

J. Zhu, K. Greenewald, K. Nadjahi, H. S. de OcÃ¡riz Borde, R. B. Gabrielsson, L. Choshen, M. Ghassemi, M. Yurochkin, and J. Solomon. Asymmetry in Low-Rank Adapters of Foundation Models . 2024. arXiv: 2402.16842 [cs.LG] .

A Theory and Proofs

A.1 Role of A and B weight matrices

Recall the feature update decomposition

To achieve optimal feature learning, we want to ensure that δ 1 t = Θ(1) and δ 2 t = Θ(1) which means that both weight matrices A and B are efficiently updated and contribute to the update in Z B . To justify why this is a desirable property, let us analyze how changes in matrices A and B affect LoRA feature Z B = BA Z.

Let ( B : ,i ) 1 ≤ i ≤ r denote the columns of B . We have the following decomposition of Z B :

where ( A Z ) i is the i th coordinate of A Z. This decomposition suggests that the direction of Z B is a weighted sum of the columns of B , and A modulates the weights . With this, we can also write

where ( B : ,i ) t refers to the columns of B at time step t . Having both δ 1 t and δ 2 t of order Θ(1) means that both A and B are 'sufficiently' updated to induce a change in weights ( A Z ) i and directions B : ,i . If one of the matrices A,B is not efficiently updated, we might end up with suboptimal finetuning, leading to either non updated directions B or direction weights ( A t -1 Z ) . For instance, assuming that the model is initialized with Init[B] , and that B is not efficiently updated, the direction of Z B will be mostly determined by the vector (sub)space of dimension r generated by the columns of B at initialization.

This intuition was discussed in details in [44].

A.2 Scaling of Neural Networks

Scaling refers to the process of increasing the size of one of the ingredients in the model to improve performance (see e.g. [22]). This includes model capacity which can be increased via width (embedding dimension) or depth (number of layers) or both, compute (training data), number of training steps etc. In this paper, we are interested in scaling model capacity via the width n . This is motivated by the fact that most state-of-the-art language and vision models have large width.

It is well known that as the width n grows, the network initialization scheme and the learning should be adapted to avoid numerical instabilities and ensure efficient learning. For instance, the initialization variance should scale 1 /n to prevent arbitrarily large pre-activations as we increase model width n (e.g. He init [4]). To derive such scaling rules, a principled approach consist of analyzing statistical properties of key quantities in the model (e.g. pre-activations) as n grows and then adjust the initialization, the learning rate, and the architecture itself to achieve desirable properties in the limit n →∞ [5, 10, 13].

In this context, Yang et al. [26] introduces the Maximal Update Parameterization (or µ P), a set of scaling rules for the initialization scheme, the learning rate, and the network architecture that ensure stability and maximal feature learning in the infinite width limit. Stability is defined by Y i l = Θ(1) for all l and i where the asymptotic notation ' Θ( . ) ' is with respect to width n (see next paragraph for a formal definition), and feature learning is defined by ∆ Y l = Θ(1) , where ∆ refers to the feature update after taking a gradient step. µ P guarantees that these two conditions are satisfied at any training step t . Roughly speaking, µ P specifies that hidden weights should be initialized with Θ( n -1 / 2 ) random weights, and weight updates should be of order Θ( n -1 ) . Input weights should be initialized Θ(1) and the weights update should be Θ(1) as well. While the output weights should be

initialized Θ( n -1 ) and updated with Θ( n -1 ) . These rules ensure both stability and feature learning in the infinite-width limit, in contrast to standard parameterization (exploding features if the learning rate is well tuned), and kernel parameterizations (e.g. Neural Tangent Kernel parameterization where ∆ Y l = Θ( n -1 / 2 ) , i.e. no feature learning in the limit).

A.3 When does γ -Operator fail to capture asymptotic behavior?

When non-polynomial dependencies (in terms of n ) appear in neural computations, then the γ operator cannot capture asymptotic behavior of the learning dynamics. For instance, if one of the layers has embedding dimension e n or n × log( n ) , polynomial exponents are no longer sufficient to capture the asymptotic dynamics. Fortunately, such cases are generally not considered in practice.

A.4 Proof of Lemma 1

In this section, we provide the formal proof of Lemma 1. The proof relies on the following assumption on the processed gradient g A . This assumption was used in [44] to derive scaling rules for the optimal learning rates for A and B weight matrices. Here, we use it to study the sensitivity of LoRA dynamics to initialization. We provide an intuitive discussion that shows why this assumption is realistic.

Assumption 1. With the same setup of Section 3, at training step t , we have Z , d ¯ Z = Θ(1) and g t A Z = Θ( n ) .

Assumption 1 consists of two parts: that 1) Z , d ¯ Z = Θ(1) and 2) g t A Z = Θ( n ) . The first condition is mainly related to pretraining paramterization which we assume satisfied such conditions. 13 The second condition is less intuitive, so let us provide an argument to justify why it is sound in practice. Let us study the product g t A Z in the simple case of Adam with no momentum, a.k.a SignSGD which is given by

where the sign function is applied element-wise. At training step t , we have

Let S t = α r B ⊤ t -1 d ¯ Z t -1 . Therefore we have

However, note that we also have and as a result

where we used the fact that sign ( Z ) ⊤ Z = Θ( n ) .

This intuition should in-principle hold for the general variant of Adam with momentum as long as the gradient processing function (a notion introduced in [2]) roughly preserves the sign ( Z ) direction. This reasoning can be made rigorous for general gradient processing function using the Tensor Program framework and taking the infinite-width limit where the components of g A , Z , d ¯ Z all become

Hence, we obtain

iid. However this necessitates an intricate treatment of several quantities in the process, which we believe is an unnecessary complication and does not serve the main purpose of this paper.

Lemma 1. Under Assumption 1, the asymptotic behaviour of Z t A and B t follow the recursive formula

Proof. At finetuning step t , the weights are updated as follows

Using the elementary operations with the γ -operator, we obtain

We conclude for Z t A using Assumption 1. The formula for γ [ B t ] follows using the same techniques.

A.5 Proof of Theorem 1

Theorem 1. Under Assumption 1, For t fixed, with Init[A] and learning rate η , we have

Stability: Z t B = O (1) if and only if γ [ η ] ≤ -1 / 2 .

Feature Learning: ∆ Z t B = Θ(1) if and only if γ [ η ] = -1 / 2 . In this case, we also have δ 1 t , δ 2 t = Θ(1) (efficient feature learning, Definition 5).

Moreover, 'internal' instability ( Z t A = Ω(1) ) occurs when γ [ η ] ∈ ( -1 , 1 / 2] .

Proof. With Init[A] , we have γ [ B 0 ] = -∞ and γ [ A 0 Z ] = 0 . As a result, we have for all t

To achieve Z B = O (1) , we should therefore have

which is equivalent to γ [ η ] ≤ -1 / 2 .

This implies that the maximum learning rate that does not cause instability is Θ( n -1 / 2 ) . Such learning rate causes internal instability, i.e. the feature Z A explodes with width. Why? Because, with this learning rate, we have γ [ A t Z ] = 1 / 2 , i.e. A t Z = Θ( n 1 / 2 ) which diverges as n grows. However, this growth is compensated with the fact that γ [ B t ] = -1 / 2 , i.e. B t = Θ( n -1 / 2 ) . This analysis is valid for any γ [ η ] ∈ ( -1 , 1 / 2] .

In this case, feature learning is efficient in the sense of Definition 5: δ 1 t = Θ(1) and δ 2 t = Θ(1) . To see this, recall that δ 1 t = B t -1 ∆ Z 1 A which yields γ [ δ 1 t ] = γ [ B t -1 ] + γ [∆ Z t A ] = γ [ η ] + γ [ η ] + 1 = 0 and γ [ δ 2 t ] = γ [∆ B t ] + γ [ Z t -1 A ] = γ [ η ] + max( γ [ η ] + 1 , 0) = 0 . So both weights contribute significantly to feature updates at the expense of benign exploding in Z t A = A t Z.

A.6 Proof of Theorem 2

Theorem 2. Under Assumption 1, for t fixed, with Init[B] and learning rate η , we have

Stability: Z t B = O (1) if and only if γ [ η ] ≤ -1 .

Feature Learning: ∆ Z t B = Θ(1) if and only if γ [ η ] = -1 .

Moreover, efficient feature learning cannot be achieved with Init[B] for any choice of learning rate scaling γ [ η ] (that does not violate the stability condition). More precisely, with Θ( n -1 ) learning rate, the limiting dynamics (when n →∞ ) are the same if B was not trained and A is trained.

Proof. Here, we show that maximal learning rate that does not cause instability in LoRA output features Z B is Θ( n -1 ) and no internal instability occurs in this scenario.

With Init[B] , we have that γ [ B 0 ] = 0 and γ [ A 0 Z ] = -∞ . From Equation (3), we obtain that

As a result, LoRA output stability is achieved if and only if

which is equivalent to having γ [ η ] ≤ -1 .

Moreover, with η = Θ( n -1 ) we have that γ [ δ 1 t ] = γ [ B t -1 ] + γ [∆ Z t A ] = 0 + γ [ η ] + 1 = 0 and γ [ δ 2 t ] = γ [∆ B t ] + γ [ Z t -1 A ] = γ [ η ] + 0 = -1 . As a result, feature learning is not efficient in this case, and the learning dynamics are asymptotically equivalent to not training matrix B (because δ 2 t → 0 ).

B Additional Experiments

This section complements the empirical results reported in the main text. We provide the details of our experimental setup, and show the acc/loss heatmaps for several configurations.

B.1 Empirical Details

B.1.1 Toy Example

In Figure 2, we trained a simple model with LoRA layers to verify the results of the analysis in ?? . Here we provide the empirical details for these experiments.

Model. We consider a simple model given by

where W in ∈ R n × d , W out ∈ R 1 × n , A ∈ R r × n , B ∈ R n × r are the weights, and ϕ is the ReLU activation function.

Dataset. Here, we used d = 5 , n = 1000 , and r = 20 to simulate synthetic data (the teacher model). Synthetic dataset generated by X ∼ N (0 , I d ) , Y = f ( X ) . The number of training examples is N train = 1000 , and the number of test examples is N test = 100 . the weights W in , W h , W out , B, A are randomly sampled from a Gaussian distribution with normalized variance (1/fan-in).

Training. We train the model with AdamW with β 1 = 0 . 9 and β 2 = 0 . 99 for a range for values of η . The weights are initialized as follows: W in ∼ N (0 , 1 /d ) , W h ∼ N (0 , 1 /n ) , W out ∼ N (0 , 1 /n ) and fixed. Only the weight matrices A,B are trainable.

B.1.2 GLUE tasks with RoBERTa

For our experiments with RoBERTa models, finetuned on GLUE tasks, we use the following setup:

Training Alg Details

LoRA Hyperparameters

Other Hyperparameters

GPUs.

Nvidia A10 with 24GB VRAM.

B.1.3 TinyLlama WikiText-2

For our experiments using the TinyLlama model finetuned on Wikitext-2, we use the following setup training with AdamW.

Training Algorithm Details

LoRA Hyperparameters

Other Hyperparameters

B.1.4 Llama-7b Flan-v2

For our experiments using the Llama-7b model finetuned on a size 100k random subset of flan-v2, we use following setup training with AdamW

Training Algorithm Details

LoRA Hyperparameters

Other Hyperparameters

MMLUEvaluation:

We evaluate average accuracy on MMLU using 5-shot prompting.

GPUs:

Nvidia A10 with 24GB VRAM.

B.1.5 Llama-7b GSM8k

For our experiments using the Llama-7b model finetuned on the GSM8k training dataset, we use following setup training with AdamW

Training Algorithm Details

LoRA Hyperparameters

Other Hyperparameters

GPUs:

Nvidia A10 with 24GB VRAM.

B.2 Additional Exps

0

1

2

Init[A]

---------------------------------

=

7.7e-04,

=6.7e-02

0

1

2

Init[B]

---------------------------------

=

7.4e-04,

=6.7e-02

|

Az

|

|

BAz

|

0

50

100

step

0

1

2

=

5.6e-04,

=5.6e-02

0

50

100

step

0

1

2

=

6.5e-04,

=4.7e-02

0

1

2

Init[A]

---------------------------------

=

4.9e-04,

=1.4e-02

0

1

2

Init[B]

---------------------------------

=

5.5e-04,

=9.7e-03

|

Az

|

|

BAz

|

0

50

100

step

0

1

2

=

4.0e-04,

=1.6e-02

0

50

100

step

0

1

2

=

3.9e-04,

=6.8e-03

0

1

2

3

Init[A]

---------------------------------

=

2.1e-04,

=1.2e-02

0

1

2

3

Init[B]

---------------------------------

=

3.0e-04,

=3.4e-03

|

Az

|

|

BAz

|

0

50

100

step

0

1

2

3

=

2.5e-04,

=6.8e-03

0

50

100

step

0

1

2

3

=

4.4e-04,

=1.4e-03

Width = 512

0

1

2

3

Init[A]

---------------------------------

=

3.5e-04,

=4.8e-03

0

1

2

3

Init[B]

---------------------------------

=

4.5e-04,

=2.0e-03

|

Az

|

|

BAz

|

0

50

100

step

0

1

2

3

=

2.5e-04,

=4.0e-03

0

50

100

step

0

1

2

3

=

3.9e-04,

=2.8e-03

Width = 1024

0

2

4

6

Init[A]

---------------------------------

=

3.4e-04,

=4.0e-03

0

2

4

6

Init[B]

---------------------------------

=

3.1e-04,

=2.0e-03

|

Az

|

|

BAz

|

0

50

100

step

0

2

4

=

2.7e-04,

=1.7e-03

0

50

100

step

0

2

4

=

3.7e-04,

=1.2e-03

0

2

4

6

Init[A]

---------------------------------

=

4.1e-04,

=2.8e-03

0

2

4

6

Init[B]

---------------------------------

=

4.7e-04,

=8.2e-04

|

Az

|

|

BAz

|

0

50

100

step

0

1

2

3

=

1.8e-04,

=1.7e-03

0

50

100

step

0

1

2

3

=

4.1e-04,

=2.8e-04

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction we clearly state the main contribution of the paper which is to compare the difference between the two natural initializations for LoRA and to conclude using theoretical justification based on large width theory and extensive experiments that one is the superior choice. In our theoretical sections we provide rigorous statements with clearly stated assumptions which provide intuition for the main claims. We provide experiments on synthetic tasks which give fine-grained support for the theoretical claims and show that for several real-world LLM tasks the main claims are supported, illustrating the generality and relevance of our results. Limitations are discussed in the final section.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a comprehensive discussion of the limitations in Section 5.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are clearly stated and referenced in the theorem statement. The main intuitions for the results are provided in the main paper and complete proofs are provided in the appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All details to fully reproduce the experiments are provided in the main paper in combination with the Appendix. Experiments were performed on standard public datasets using standard public libraries. The algorithmic complexity of the experiments is fairly minimal as it involves just adjusting hyperparameters.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The validity of the experiments should already be believable without code, but we plan to fully release code soon. The main claims should already be easily reproducible using standard libraries.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This is provided in Appendix B.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We average all results over multiple random seeds and plot both the average and 1-sigma error bars shaded in the plots.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the compute resource details in Appendix B.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does not meet any of the concerns for potential harms.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is concerned with an abstract problem of choosing an appropriate initialization in an extremely general setting and is not closely tied to any societal concerns.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no such risks for this paper.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There are no licensing concerns.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are created.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the impact on training dynamics of two initialization schemes for LoRA. For this purpose the authors investigate the asymptotic behaviour of activations and weights for LoRA adapters. The authors find that init[A] where A is intialized randomly and B is initalized with zeros leads to more efficient learning dynamics but leads to some numerical instabilities. This initialzation method is the one most commonly used for LoRA-style methods. Further, the authors found that init[B] leads to suboptimal learning dynamics but is less prone to numerical instabilities. Finally, init[A] performs better with larger learning rates than init[B], which is supported by experiments on NLU as well as NLG finetuning tasks.\n\nSTRENGTHS:\nThe paper is well motivated and presents theoretical analysis accompanied by empirical experiments.\nClaims are supported by theoretical analysis as well as relevant experiments. \nFurthermore experiment hyperparameters as well as hardware requirements are well documented.\n\nWEAKNESSES:\n**Novelty**\n\nMost of the theoretical Analysis of the LoRA finetuning dynamics in this paper have already been presented in [1]. \nThe authors should have made it more clear how their work differs from the Analysis done in [1], especially since most formulas and notation is identical. \nFurther, both of the methods evaluated in this paper have also been investigated in [1].\nThe paper would greatly benefit from a clear explanation on how the analysis in this work differs from results published in [1].\nFinally, it is not clear if the contribution of individual LoRA layers to feature learning can be derived from a setting where only a single LoRA layer is trainable while all others are frozen (as claimed in chapter 3.1).\n\n**Experimental results**\n\nSince the main gist of the paper is to investigate the learning dynamics of LoRA, it would be good to also investigate different initialization schemes aside from init[A] and init [B] prevalent in the litearture, e.g. gaussian init [2], kaiming init [3], principal components [4], etc.\nIn [1] only different learning rates for A and B have been investigated.\n\n**Significance of results**\n\nWhile the main finding that init[A] has a better optimal performance is interesting, this initialization scheme is already the common initialization scheme used by LoRA.\nTherefore, the impact of the main findings is marginal.\n\n[1] Hayou et al., LoRA+: Efficient low rank adaptation of large models., ICML 2024\n\n[2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022\n\n[3] He et al., & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification., ICCV 2015\n\n[4] Meng et al., PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models., arXiv 2024\n\nQUESTIONS:\n- Did the authors investigate other commonly used initialization schemes?\n- Will the authors provide code that can be used to reproduce the results in the experiments section?\n\nLIMITATIONS:\nThe authors have properly addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the impact of initialization schemes on the finetuning dynamics of Low Rank Adaptation (LoRA) in large language models. The authors compare two initialization approaches: Init[A], where A is initialized randomly and B to zero, versus Init[B], where B is initialized randomly and A to zero. Through theoretical analysis and empirical experiments, they demonstrate that Init[A] allows for larger learning rates and leads to more efficient feature learning compared to Init[B], albeit at the cost of some internal instability. The paper provides a rigorous theoretical framework based on large width limits and supports the findings with experiments on toy models and real-world language tasks.\n\nSTRENGTHS:\nS1: The paper addresses an important and understudied aspect of an usually handwaved-away aspect of LoRA, namely the impact of initialization schemes. This is a relevant topic given the widespread use of LoRA for efficient adaptation of large language models.\n\nS2: The theoretical analysis is rigorous and well-grounded, using asymptotic analysis in the large width limit to derive principled insights about the dynamics of LoRA finetuning under different initialization schemes.\n\nS3: The empirical results on both toy models and real-world language tasks provide strong support for the theoretical findings, demonstrating the practical relevance of the initialization choice.\n\nS4: The paper presents a clear trade-off between feature learning efficiency and internal stability, providing nuanced insights that go beyond simply recommending one initialization over the other.\n\nS5: I would like to commend the authors for putting great amount of work on both the main document as well as the appendix. Overall not only does the paper read well and seems theoretically and experimentally sound, but the manuscript includes an excellent degree of experimental details, and it's thus likely that future work will be able to cleanly build on it.\n\nWEAKNESSES:\nW1: While the theoretical analysis is sound, the paper could benefit from a more intuitive explanation of why Init[A] allows for larger learning rates and more efficient feature learning. This would make the insights more accessible to some practitioners.\n\nW2: The experiments focus primarily on language models and NLP tasks. This is definitely acceptable and overall a very sound choice, however the inclusion of experiments from other domains (e.g., finetuning an image classification model, or some other architecture that relates to a vision transformer) would strengthen the experimental section significantly, and increase the overall impact of the paper in the broader NeurIPS community.\n\nQUESTIONS:\nQ1: How sensitive are the results to the choice of LoRA rank? Do the theoretical predictions and empirical findings hold across different rank values?\n\nQ2: The paper focuses on vanilla LoRA. How do the authors expect these initialization schemes to interact with variants like QLoRA or LoRA+? Would the trade-offs and recommendations change?\n\nQ3: Given the internal instability observed with Init[A], are there any potential negative consequences for downstream task performance or generalization ability? How might this instability manifest in practice (and would there be a difference depending on input domain and/or other parameters of the model + problem space)?\n\nQ4: The theoretical analysis seems to assume a single LoRA layer, for simplicity. How well do the insights generalize to more realistic scenarios with multiple LoRA layers throughout a network?\n\nLIMITATIONS:\nL1: The theoretical analysis seems to rely on assumptions about the asymptotic behavior in the large width limit. The paper could benefit from a more detailed discussion of how these assumptions may or may not hold in practical scenarios with finite-width networks.\n\nL2: The empirical evaluation, while comprehensive, is limited to a subset of NLP tasks and model architectures. Expanding the evaluation to a broader range of tasks, model sizes, and architectures would strengthen the generalizability of the findings.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper analyzes the impact of different initialization techniques for A and B matrices in LoRA adapters. Typically, either A or B matrix is initialized with zero while the other is initialized form a Gaussian distribution. This is done so that fine-tuning starts with LoRA adapters ($A \times B = 0$) having zero effect. However, the paper argues that it matters which of A and B initialize to 0. As per the results, $B = 0$ init allows higher optimal learning rates which leads to better overall performance. However, there is still instability during training which could be mitigated with a method like LoRA+.

STRENGTHS:
It's useful to know that the default initialization scheme in LoRA happens to be the better alternative. The experiments are convincing.

WEAKNESSES:
Perhaps I am not the right person to review this paper, but I don't fully understand the impact of this paper. It shows that there are two ways of initializing AB matrices and the default one that we have been using until now is the better option. Additionally, it says that the default initialization is not good enough but doesn't propose anything new. Maybe the theory part is important, but I couldn't understand it fully in the limited time I had. Having said that, I still feel that the paper has something worthwhile to contribute to the community. So, I vote to accept it for now.

QUESTIONS:
None.

LIMITATIONS:
The limitations section in the paper is adequate.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFinetuning large language models has become a common technique among practitioners, however due to the memory footprint of practically viable LLMs, there is a dire need of techniques that allow memory-lightweight finetuning (PEFT). Among those techniques LoRA has gained immense popularity. In LoRA a dense matrix is modified via addition of a low-rank update consisting of matrices A and B. In order for this update to initially conserve the model’s outputs while preserving the ability to learn, exactly one of the matrices needs to be initialized to zeros.\n“The Impact of Initialization on the Finetuning Dynamics in LoRA” proposes that the B matrix should be initialized with zeros. It hypothesizes that the reason for that is that this initialization (Init[A]) allows using greater learning rate without sacrificing stability.\nThe main points of the paper are supported by theoretical derivations stemming from the perspective of infinite-width networks. Furthermore, experimental evaluations serve as proof of the soundness of the theoretical approach and as practical justification of the method.\n\nSTRENGTHS:\n- Mathematical support combined with experimental results\n- Potentially significant due to how widespread LoRA adoption is and due to how easy the technique is to implement\n- The paper is quite original - even though LoRA is a popular technique and it is a fundamental question whether the A or the B matrix should be initialized with zeros, this work seems to provide the first attempt at answering it.\n\nWEAKNESSES:\n- There is virtually no discussion of why the training hyperparameters are set to those exact values, in particular with regard to the number of epochs and weight decay = 0. I suspect changing those HPs might mitigate the need to choose the initialization schema so carefully. Also the r parameter seems to be chosen arbitrarily.\n- The gains provided by the method seem to be small and would be better visualized if they were put in the perspective of comparison with other methods such as differing r or even, if possible, full finetuning.\n- The Toy Model setup is not explained clearly.\n\nQUESTIONS:\n- There are other works that try to improve LoRA’s performance (LoRA+) or are inspired by it (VeRA, AdaLoRA, etc.). Do you believe your work could be used with those techniques?\n- Do you feel that there is some simple, informal intuition suggesting why the Init[A] outperforms Init[B]?\n- AdamW seems to be among the most popular optimizers, however the paper focuses on rigorous derivation in case of SignSGD claiming that the result should extend to the typical choice of the Adam optimizer. Even though the authors claim they are using the AdamW optimizer in the experiments, they set weight decay = 0, effectively replacing AdamW with Adam. Would the method perform as well while using weight decay? Would it be justified theoretically the same way Adam is?\n- Can you justify the fixed hyperparameters’ values?\n- Can you explain the toy model setup more clearly? As I understand it now, the same model is being adapted and is used to create the dataset which does not seem to make sense.\n\nLIMITATIONS:\n- I do not believe the authors have sufficiently explained the limitations of the fixed hyperparameters in the evaluations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}