Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Authors: Shiwei Li
Xiandi Luo
Xing Tang
Haozhao Wang
Hao Chen
weihongluo
Yuhua Li
xiuqiang He
Ruixuan Li
Venue: ICML 2025 poster (2025)
Keywords: Low-Rank Adaptation
Non-Zero Initialization
Primary area: deep_learning->large_language_models
TL;DR: This paper demonstrates that the non-zero initialization of LoRA enhances robustness to learning rates.

Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method. 
In standard LoRA layers, one of the matrices, $A$ or $B$, is initialized to zero, ensuring that fine-tuning starts from the pretrained model. However, there is no theoretical support for this practice.
In this paper, we investigate the impact of non-zero initialization on LoRA's fine-tuning dynamics from an infinite-width perspective. Our analysis reveals that, compared to zero initialization, simultaneously initializing $A$ and $B$ to non-zero values improves LoRA's robustness to suboptimal learning rates, particularly smaller ones. 
Further analysis indicates that although the non-zero initialization of $AB$ introduces random noise into the pretrained weight, it generally does not affect fine-tuning performance. In other words, fine-tuning does not need to strictly start from the pretrained model.
The validity of our findings is confirmed through extensive experiments across various models and datasets. 
The code is available at https://github.com/Leopold1423/non_zero_lora-icml25.

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tunie Tine-Tunigg Dynamics
Shiwei Li 12
Xiandi Luo
Xing Tang² Xiuqiang He
Haozhao Wang'
Hao Chen
Weihong Luo
Yuhua Li
Ruixuan Li
Abstract
B E R
Low-rank adaptation (LoRA) widely used parameter-effiient fine-tuning method. In stan- dard LoRA layers, one of the matrices, A or B, is initialized to zero, ensuring that fine-tuning starts from the pretrained model. However, there is no theoretical support for this practice. this paper, we investigate the impact of non-zero ini- tialization on LoRA's fine-tuning dynamics from indinite-width perspective. Our analysis re- veals that, compared zero initialization, simul- taneously initializing A and B to non-zero val- ues improves LoRA's robustness to suboptimal learning rates, particularly smaller ones. Further analysis indicates that although the non-zero ini- tialization of AB introduces random noise into the pretrained weight, generally does not affect fine-tuning performance. In other words, fine- tuning does not need to strictly start from the pretrained model. The validity of our findings is confirmed through extensive experiments across various models and datasets. The code available at https: thub. hemp/eeoolllllio 423/ non _en_zeer__lorg icm125.
Pretruined Weights
LoRA Structure
W E & g
gr
Initialization Strategy
(0, B
Ini[[AB]: o00.0)) BEN(0,,'))
Init[A]:
Fine-tuning Dynamics
Efficiont Learning with Stubility, but Sensitive learning rate
Efficiency and Stubility. with Robustness suboptimal learning rate
Figure I. LoRA's fine-tuning dynamics with zero initlization (i.c., Init (A]) and non-zero initialization (i.e.. Init (AB]). ). Com- pared to Init IAI, Init t AB] improves LoRA's robustness to suboptimal learning rates, generally leading to better performance.
storage-intensive, presenting significant challenges for their practical deployment. To address these issues, various parameter--effiiiet fine-tuning (PEFT) methods have been proposed to reduce the number of trainable parameters, thereby decreasing the cost of fine-tuning (Xu et al., 2023).
Among these PEFT methods, low-rank adaptation (LoRA) (Hu al., 2022) has gained considerable attention due its efficiency and minimal impact inference latency. LoRA approximates the update of the pretrained weight matrix W using smaller low-rank matrices, A and B, as illustrated in Figure 1. The forward pass of LoRA is formulated as Y = (W + ()) where r is the LoRA rank and a is a tunable scaling factor. During fine-tuning, W remains fixed, while only matrices A and B are updated, usually via the Adam optimizer (Kingma & Ba, 2015).
1. Introduction
Large language models (LLMs), such as GPT-4 (Ope- nAI, 2024) and DeepSeek-V3 (DeepSeck-AI, 2024), have achieved unprecedented performance across a broad range of tasks. These State-of-the-art models typically contain hundreds of billions of parameters. Consequently, fully fine-tuning these models is computationally expensive and
'Equal contribution. 'Huazhong University of Science and Technology. Wuhan. China "Shenzhen Technology University, Shenzhen, China FIT, Tencent, Shenzhen, China. Correspon- dence Xing Tang <xing.tang tangtttngee @hotmail.comm>> Haozhao Wang waag @ hust.edu.en... Ruixuan Li herti@hhst.cdd.cn.......
In standard LoRA layers, either A or B is initialized to zero, ensuring that fine-tuning starts from the pretrained model. Building on this premise, Hayou et al. (2024b) explores the difference between applying Kaiming initialization (He et al., 2016) to and B. However, there no theoretical justification for the zero initialization of LoRA, which raises
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
the question: Is it optimal to initialize A or B to zero? To answer this question, we explore the impact of initial- ization on LoRA's fine-tuning dynamics. Let Init [A] denote the case where A is randomly initialized and B is set to 7ero, while Tnit t [AB] denotes the case where both A and B are initialized to non-zero values, as illustrated in Figure 1. Our analysis demonstrates that, compared to Init [A]. Init [AB] improves LoRA's robustness to sub- optimal learning rates, particularly smaller ones. Notably, learning rate decay is commonly applied during fine-tuning, making smaller learning rates more prevalent.
parametrizations (Bordelon et al., 2024; Yang. 2020) etc. It typically involves analyzing the statistical properties of key quantities, such as pre-activation values, by extending the network's s width or depth to infinity. Subsequently, this analysis is leveraged to refine hyperparameters, ensuring favorable convergence properties under these extreme condi- tions. For instance, Kaiming initialization (He et al., 2016) suggests that the variance of the weights in hidden layers should be scaled by 1/n to prevent excessively large pre- activations as the model width increases. More details on the scaling theory are provided in Appendix A.
However, non-zero initialization introduces noise to the pre- trained model before fine-tuning, which raises another ques- tion: Is it necessary for fine-tuning to strictly start from the pretrained model? Note that pretrained weights are usually suboptimal for downstream tasks, suggesting that the pretrained model itself contains inherent noise. Thus, we conjecture that the noise introduced by non-zero initial- ization can also be corrected through fine-tuning, without additional optimization cost. Extensive verification shows that, with appropriate initialization variance, the noise in- troduced by non-zero initialization does not degrade the fine-tuning accuracy. Notably, the range of suitable initial- ization variances is quite broad, encompassing that used in Kaiming initialization. In conclusion, fine fine-tuning does not have to strictly start from the pretrained model.
In this paper, we apply the scaling theory to investigate the fine-tuning dynamics of LoRA from the perspective of infinite width. Specifically, we explore how to set the learning rate and initialization strategy for LoRA to achieve desired performance. In our analysis, we assume that only the model width n increases, while other factors, such as the LoRA rank, number of layers, sequence length, and num- ber of training steps, remain constant. To characterize the asymptotic behavior of the analyzed variables as n increases, we further introduce the following notations, which have also been utilized in previous analysis of LoRA, partieularly in the context of learning rates (Hayou al., 2024a) and zero-initialization strategies (Hayou et al., 2024b).
Notations. Let C. E R and dn E R be sequences. If there exists a constant K >> such that G, < kdn, we write en = (dn)). If c, > kd2, we write cn = (dd)). If both Cn = (dn)) and c. = [d(,,) hold simultaneously, we write Cn ((d)). For a vector sequence C. = (c,) E R (k: > 0), we write G, (cd,) if and only if c/ = (d,,) for all i E [k]. This similarly applies to θ and Ω. Ad- ditionally, when the sequence On is a vector of random variables, convergence is understood to be convergence of its & norm. To track the exponent of the asymptotic be- haviour of some quantity v. we further introduce the 7 operator, defined as e @(n For example, 7/n] 1. >11] = 0, and >10 = - For two real-valued vari- ables u and v, we have 7/0 x u] -Tu]] + 7[1], and 7[ + u] (ax() provided that v & -u. More details about the 7-operator are provided in Appendix A.2.
The main contributions of this paper can be summarized as two novel insights that challenge the traditional inertial constraints of LoRA fine-tuning:
1. You do not need to initialize LoRA's matrices A or B to zero. Compared to zero initialization, non-zero ini- tialization enhances LoRA's robustness to the learning rate and generally achieves better fine-tuning accuracy.
2. You do not need to start fine-tuning strictly from a pretrained model. Although non-zero initialization alters the starting point of LoRA fine-tuning, it does not impact the fine-tuning performance, as long as an appropriate initialization variance is employed.
3. Experimental results confirm the validity of our theory and findings. LoRA with non-zero initialization con- sistently achieves superior accuracy compared to zero initialization, particularly at smaller learning rates.
Next, we focus on the asymptotic behavior of the learning rate, 7(1)). as the network width n - 0o, rather than its exact value. Given 7(n]. 1) = ((n7ll) x c- 7inl, where >> 0 is constant and lower-order terms are neglected. As n + oc, the term n 7in] dominates, making >(1) the key factor determining the asymptotic behavior of 17. While the constant e is important for exact values, it doesn't 'it influence the asymptotic scaling behavior. Ignoring the influence of e, perturbations to 7(v) and n are effectively equivalent. Therefore, we analyze the impact of perturbations in 7[r]]
2. Preliminaries
2.1. Scaling Theory of Neural Networks
The scaling theory of neural networks is widely used to guide the selection of hyperparameters, such as learning rates (Hayou et al., 2024a), initialization strategies (He et al., 2016; Yang, 2019; Hayou et al., 2024b) and depth
'Note that the width n of LLMs is typically large (n > 10²). 27(0] = co is writen as limit of 7/mn²) when / > oc.
2

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
on fine-tuning dynamics, excluding e from consideration. This analysis aims to guide the selection of learning rates and initialization strategies based on asymptotic properties, rather than computing precise values that depend on c.
Additionally, a simple toy model is presented in Section 3.4 to validate our theoretical findings. Finally, we explore whether non-zero initialization affects fine-tuning accuracy by deviating from the intended purpose of zero initialization, which is to start fine-tuning from a pretrained model.
2.2. LoRA and LoRA Features
3.1. Simplified Settings and the Optimizer
Based on the previous analysis of LoRA presented in (Hayou et al., 2024a;b), we adopt the same notation and consider a general neural network of the following form:
To simplify the analysis, we adopt the setup proposed in (Hayou et al., 2024a;b). Specifically, we assume that the fine-tuning dataset comprises a single data point, (x, y). The optimization goal is to minimize the loss calculated using the pretrained model that incorporates LoRA.
(x) W Vi(X)= = Fi(W, (x)( L [T), (x) = (X)),
(1)
Moreover, we train only one LoRA layer isolate the con- tribution of individual LoRA layers to feature learning. Let Z denote the input to the LoRA layer computed from a, and let dZ' represent the gradient of the loss function, eval- uated at the data point (x, y), with respect to Z. During fine-tuning, the input Z remains constant as the step I in- creases. In contrast, dž varies with t. as Zt evolves over time. Next, we examine the behavior of ZB (Definition 2) as approaches infinity from three perspectives: stabil- ity, efficiency, and robustness. Formal definitions of these perspectives will be provided in subsequent sections.
where X E I denotes the input vector, and L 2 1 is the network depth. Let n denote the network width and F denote the nonlinear mapping function. Wi E R is the weight matrix of the (-th hidden layer. Win and Wont correspond to the weight matrices of the input and output layers, respectively. This model represents a pretrained neu- ral network that will be fine-tuned for a specific downstream task using the LoRA framework defined below.
Definition 1 (LoRA). Let W E RR x be a weight ma- trix from the pretrained model. LoRA approximates the update W using low-rank matrices, such that the value of the pretrained weight becomes W + A. Here, only B E R2 and A E Brx=2 are trainable. The rank r < min(n1, n2), and a E IR is a tunable scaling factor.
Adam (Kingma & Ba, 2015) is a commonly used optimizer for fine-tuning LLMs. Therefore, we focus on the fine- tuning dynamics of LoRA with the Adam optimizer. A similar analysis for the SGD optimizer (Bottou, 2010) is provided in Appendix C, where we observe that Adam out- performs SGD in enhancing LoRA's feature learning capa- bilities, primarily due to the gradient normalization process. The conclusions drawn in this section are partially applica- ble to SGD; further details can be found in Appendix C.
For each LoRA layer within the network, let Z denote its input and let Z denote the output after incorporating the pretrained weights, ie.. 2- = W + e/BBAZ. In this paper, we primarily investigate the statistical properties of the output of the LoRA module, i.e., BAZ. To this end, we further introduce the LoRA features as follows:
3.2. Stability and Efficiency of LoRA
Definition 2 (LoRA Features). Given a general neural ar- chitecture and LoRA (Definition 1) layer, define the LoRA features AZ ZB) Z AZ and Z BZA BZZ The superscript of A Z represents the value of the LoRA features in the t-th step of fine-tuning.
To ensure that the LoRA layers produce stable outputs, it is expected that 3B remains within a reasonable range through- out the fine-tuning process. Consequently, the concept of stability is introduced and defined as follows:
Definition 3 (Stability). LoRA fine-tuning process con- sidered stable if. for all LoRA layers and for all t >> 1,* Z (1) the width approaches infinity.
Definitions 1 and 2 are straightforward and have been used in (Hayou et al., 2024a:b). Next, we focus on the training dynamics of the final output Z as n approaches infinity.
Stability indicates that as the width increases, the output of LoRA, ZB, remains bounded and does not diminish, demonstrating LoRA's effective and positive contribution to fine-tuning outcomes. For the sake of analytical clarity,
3. Fine- Jine-Tuning Dynamics of LoRA with Adam
In this section, we first introduce two critical statistics of LoRA: stability and efficiency, as discussed in Section 3.2, along with their respective solution sets for the learning rate and initialization. Next, we investigate the sensitivity of these solutions to the learning rate in Section 3.3 and ex- plain how non-zero initialization improves their robustness.
"This simplification is made for analytical purposes only; the analysis can be extended to mini-batched gradients without chang- ing the conclusions. However, additional assumptions are required to ensure the rigor of such an extension.
"The case where t 1 is excluded hecause, in standard LoRA, B is typically initialized to 0, resulting in Z/z = O.
3

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
we do not consider LoRA's internal stability (i.e., Zạ) in this section. Instead, a detailed analysis and discussion of internal stability are provided in Appendix B.2.
above equations in terms of (Au, Bo) and (MA (B) as:
(aax(:(Bo))7/nn)) + [[ma] 0,
max max()[Ao),n//nl) + +1= O, max(:(Ao)) + max( r(mal) + = O,
In addition to the stability of ZB, we are more concerned about whether Z is effectively updated during fine-tuning. At the 1-th step, the update to ZB is given by:
(4)
which further leads to the following conditions:
AZ BI-IAZA A + ABA
Tital + >[ms] -11, [>[4a] nA, > [Bo)] 1B-
(5)
82
82
By default, a uniform learning rate is used for the weight matrices A and B, i.e. 7A = 7/B. Based on this condition, the solution to the above equations
As discussed in (Hayou et al., 2024a;b), the terms i7 and 57 denote 'linear' updates, reflecting the effects of training A and B independently. The third term, 88, represents "multiplicative' update, capturing the combined effect of optimizing both A and B simultaneously. Ideally, n approaches infinity, AZB should remain ((1) to ensure effective updates to ZB during fine-tuning. A similar con- dition was proposed in (Yang, 2020) for pretraining and later adopted by (Hayou et al., 2024a;b) as a metric for effi- cient feature learning in LoRA. According to (Hayou et al., 2024a;b), it is essential for both 6. and 62 to be (11) to ensure that the updates of A and B contribute positively to BB. 5 Based on the above analysis, the efficiency of LoRA's feature learning is defined as follows:
Tnmal = >[1a] =1/22 [/Ao] - 11/2, A/Bo| S 11/2,
(6)
where 7(Ao) and [Bo] cannot both be equal to -oo simul- tancously, as this would imply that both A and B are zero, preventing optimization via gradient descent algorithms.
In standard LoRA, A is initialized using Kaiming initial- ization, while is initialized to zero This initialization strategy is denoted as Init [A]. In contrast, when B is initialized using Kaiming initialization and A is initialized to zero, the strategy is referred to as Init [B]. Let 0% and o% represent the variances of Au and Bo respectively. For Init [A], 0 = 1/~ and oß = 0,, which results in >|Ao] = -1 and > BB] = -0, thereby satisfying (6) However, for Init [B] 0² = and oB = 1/r, result- ing in >[Ao] = -o and > [BB) = O, which fails to meet the desired conditions." 7 Consequently, we conclude that LoRA achieves stable and efficient feature learning when using Adam with Init [A], but not with Init [B]. This conclusion aligns with the findings of (Hayou et al., 2024b).
Definition& 4 (Efficiency). A LoRA fine-tuning process is considered efficient if it is stable ( (Definition 3) and, for all LoRA layers and for all t >> 1, _ = (1) and 62 = e().
In contrast to stability, efficiency emphasizes the effective updating of Z.. We now analyze the effects of initialization and learning rate both stability and efficiency. At the t-th step of fine-tuning, the LoRA weights are updated as A, = A, - 749A = Ao - NA Ei=09 9' and B, = B "/ = Bo BB Ei=0 Db- where (MA 7B) are the learning rates, (Ag, Bo) are the initialization values, and (ga, 9n) represent the normalized gradients generated by the Adam optimizer. Note that both 9 and gg are of order (1) due to the gradient normalization process. By substituting AZ' = - 7Z and A = - 7BgB into E2,,)), and applying the 7ooperaoor, the conditions for LoRA to achieve stability and efficiency can be expressed as:
Similarly, when using the SGD optimizer, solutions to en- sure stability and efficiency for LoRA can be derived as:
> rnmal = 711l -1/2, n/Ao] S 3/4, y/Bo| S 11/4,
where at least one of the two inequalities must hold with cquality. It is evident that neither Init [A] nor Init [B] can satisfy the desired conditions when using the SGD op- timizer. Furthermore, we highlight that Adam offers more flexible initialization options than SGD, benefit attributed to the gradient normalization process. Detailed proofs and explanations can be found in Appendix C.
[ = > 11 16 2 =0
(o) = O(1)
2-1 A
= 0
(6? (())
(3)
- - B1B,-1 A 12 = 0
(Zh=1 O(1))
Using the properties of the y-operator, we can express the
'A detailed proof of this conversion is given in Appendix B.I. "In the case where 7) = My = 7). Init [B] can only achieve stability when = -1. LoRA+ (Hayou et al., 2024a) decouples the learning rate of matrices A and B (i.c., 7[ns] -1 and >ins] = 0), therehy enabling LoRA to achieve both stability and efficiency when using Init [B].
'One might wonder why 6 is not required to be ((1). This is because as long as LoRA is stable and 6,, 6e are both (11), then 88 e(1) (i.c.. 7 = 7(na] + >[na] 1 = 0) is guaranteed,
as shown in (5)
4

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
3.3. Robustness of LoRA
unaffected, thereby ensuring consistent stability. Based on the above analysis, we derive Theorem 1, which presents the initialization strategy that optimally supports robustness (Definition 5) while maintaining stability and efficiency (Definitions 3 and 4) for LoRA fine-tuning.
The solution presented in Eq)(6) demonstrates that LoRA's stability and efficiency impose stricter constraints the learning rate, while the range of feasible initialization val- ues is comparatively broader. Note that strictly setting Y [nA] and [[na]] to -1/2 is particularly challenging in practical see- narios, as the constant in θ is generally intractable. Conse- quently, we investigate the impact of deviations in 7[v] from - on LoRA's stability and efficiency. For the case where
Theorem (Robust LoRA (Informal)). A LoRA fine-tuning process optimized with Adam exhibits optimal robustness (Definition 5) to the learning rate when Ao = Onn and Bg = On 1/2).
r/in 1/2, 7[6]] [ = [ 27(1] + 1 > 0, which inevitably causes instability and inefficiency. There- fore, learning rate decay is commonly applied during fine- tuning to prevent excessively large learning rates. In the fol- lowing, we primarily focus on the case where >(v] - -1/2.
This initialization scheme is referred to as Init [AB], where it is observed that Bo - ((n This is equivalent to treating AB as whole and then perform- ing Kaiming initialization, rather than directly performing Kaiming initialization for A or B as done in Init [A] or Init [B]. As shown in Table 1, Init t [AB] can simultane- ously achieve stability, efficiency, and robustness, whereas Init [A] and Init [B] cannot.
First, when >/va] S 7[An] S -1/2, max)) ((AA)) =
y [Ao]. In this case, >[1A] < 11/ will not affect 7602] and y [Zs]. Similarly, when 7/10] <>[B]] S -122, [[18] <
-1/2 will not affect -/8.] and r[Zn]. Conversely, when
rina] > >[Ao], y/6,] = 7|na]= +>[7n] + 1 = 27(nl + 1,
Table 1. A comparison hetween various initialization strategies. Initialization Stability
which implies that changes in 1 will result in a quadratic change in 61. The same holds for 52 and ZB. Clearly, in this
Efficiency
Robustness
case where >/VA]= > [Ao]] and 7[ma] > [Bo]. LoRA is more sensitive to fluctuations in the learning rate. Based on
Init [B]
X
X
Init [A]
X
Init [AB]
these insights, we formally define the robustness of LoRA's
stability and efficiency as follows:
However, enforcing Au and By to be exactly O(n is challenging (similar to the learning rate), as the constant within θ is typically intractable. Therefore, we do not impose a strict requirement on Ao and Bg to be of order O(n-1/3) Instead, our main focus is ensuring that Au/B, = ((1), which guarantees that A and B are initialized to non- zero values simultaneously. For simplicity, we ignore the constant in @(1), implying that the variances of Ao and Bu are equal. Compared to initializing B to zero, this at least enhances the robustness of LoRA fine-tuning with respect to m/. Finally, we formally define the non-zero initialization for LoRA as follows:
Definition 5 (Robustness). Assume a LoRA foneAine-tune fine-tuming pro- cess is stable and efficient (Definitions 3-4) with 7[1] = =7" The process is considered robust with respect to the learning rates 7[yA)] S 7 and y(mo] I >- if. for all LoRA layers and for all 1. >> 1, the following relations hold: P(7(01] 7 r(ns)) = P(y (51)), P(7 {[nA)) = (()( and P[)[Zn] [[mA]) = P(7[Za) ((ne)) = PB)
Robustness focuses on the impact of suboptimal learning rates, particularly those lower than the optimal value, the stability and efficiency (i.e., ε1 &2 and ZB). Specifi- cally, 6, reflects the effect of training A independently, and robustness seeks its independence from 7B. The same prin- ciple applies to 8z. To achieve this, both 7 [[Ao] and 7 [[oo] should be set to -122," which leads to the following rela- tionships for all learning rates satisfying 7[1a]= ك -1/2 and y ynnal - 1/22.
Definition 6 (Non Zero Initialization). A LoRA layer is non-zero initialized if the weight matrices A and B are initialized with the same variance, i.e., 0² o
3.4. Toy Model
] = 7 [Bo] + + 1 1= + 1/2, [82] = *Aoo] + >/ns] + = >\v>] + 1/2,
In this section, we verify the advantages of non-zero ini- tialization in LoRA by fine-tuning the following toy model, which is actually the general model in Eq.(1) with L 1.
<[4a] = +|Ao++ +B[B]] =
+
f(x) = W(Wo + BA))
Clearly, 81 and 62 vary linearly with changes in the learning rates 7A and 78, respectively, while > [Z 1 remains
where X E Rd is the input vector, W.. E Raax W Wout E 1000 and Wo E R"x" are the pretrained weights. A E R2xn and B E Roxr are the LoRA weights. The dimensions (d, n) are set to 784, 4096, and 32. The MNIST (Deng, 2012) and FashionMNIST (Xiao et al., 2017)
'The conditional probahility P(z| v) = P(x) indicates that x: is independent of y-
"eetting [[Ao] 7 Bo] = 1122 ensures robustness only for learning rates that satisfy r(n] S s.
5

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
datasets are used for pretraining and fine-tuning, respec- tively. During fine-tuning, only matrices A and B are train- able, while Win, Wo and W remain fixed. Adam is used as the optimizer. More details about the toy model can be found in Appendix D. Subsequently, we compare the test loss of LoRA under zero and non-zere initializations, vary- ing learning rates and initialization variances. The results, presented in Figure 2, demonstrate that the performance of LoRA with zero initialization deteriorates significantly at lower learning rates, whereas the non-zero initialization consistently yields better performance, aligning with our theoretical expectations.
A and B based on gradient information. This involves com- puting the gradient of the pretrained model over several steps of gradient descent and applying truncated SVD on the gradient to derive the initialization values for matrices A and B. Although these previous methods utilize non-zero initialization for LoRA, they do not discuss the benefits nor provide a theoretical foundation for such initialization. This study bridges these gaps and offers theoretical support for non-zero initialization techniques.
Standard LoRA employs zero initialization to ensure that fine-tuning starts from the pretrained model. However, non- zero initialized LoRA introduces noise to the pretrained model before fine-tuning begins. A straightforward solution to this problem is to subtract : Agg Bg from the pretrained weights, as proposed by Wang et al. (2024). However, this requires modifications to the pretrained model, complicat- ing its reuse. This challenge becomes particularly evident in multi-LoRA scenarios, where the pretrained model is shared across multiple LoRA modules for different tasks (Sheng et al., 2023). In such cases, it is required that the initialization of different LoRA modules remains consistent.
Test for Init[A]
0.00
Test for An)t(AB]
0.30
3e
30.3
Ze
0.85
2e:3
0.85
Leariing Rate 55
1e-3-
S1e-3
0.80
0a-4
9e-4
8o-4
0.75
8e.4
0.75
7e 4-
7e
0.70
6e-4
5a-4
0.05
Se-4
0.45
4o-4
0.60
0.60
3e-4
2e
2e-5
1n-4
Be-4
6e-3
4w-2
20-5
1o-4
Be-4
6e-3
4e-2
Initialization Size (8)
Init ization Size (8)
(Init(AB] Init[A][[nit(A]
0.5
Convergence Curves
3e-3
Learning Bate (8)
1
2e-3
0.4
0.7
MRPC 8.10
8:16
8.16
8.26
8.42
0.52
10:48
0.88 10.04
10
Be
0.3
0.6
CoLA
8.35
8.36
8.37 8.36
8.25
5.30
11.21
11.17
11.21
Test Loss
8e
9.2
6e-4
7e-4
6.1
0.5
SST2
6.68
1663 1.88 5683 6.5 9.83
8.61
0.73
0.48
9.72
5o-4
4c
0.0
0.4
QNLI
6.24
8.54
6.52
3.43
9.24
3/33
8.0
8:38
10.19
10:45
3e-4
26.5
10:4
Bo-4 66.3
4b-2
1
16
64
128
256
Initialization Size (5)
Steps
Initialization Size (B2)
Figure 2. Test loss of LoRA fine-tuned with zero initlization (Init [A]) and non-zero initialization (Init [AB]). The lower- left corner presents the improvement ratio of Init [AB] relative Init IAI, while the lower-right corner presents the conver- gence curves of nit [AB] and Ini t. [A] under specific settings.
Figure 3. Test loss of un- un-fine-tuned T5- T5-Base on different dataset with various initialization size (8²). Note that the initialization variances of matrices A and B are both set to (Bor))², where of is the variances of A when initialized with Kaiming initialization.
Above discussion raises the question of whether it is neces- sary to subtract A from the pretrained model. Note that pretrained weights are typically not optimal for down- stream tasks. Therefore, small perturbations to the pretrained weights will not sig- nificantly affect their performance on downstream tasks, nor will it substantially impact convergence during fine- tuning. To test this hypothesis, we conducted a preliminary evaluation using the T5-Base model (Raffel et al., 2020). Specifically, we compared the accuracy of the un- un-fine-tuned T5-Base on several datasets of the GLUE benchmark (Wang et al., 2019), utilizing non-zero initialized LoRA with var- ious initialization variances. As shown in Figure 3, the non-zero initialization does not impact the initial perfor- mance of the pretrained model on downstream tasks unless the initialization variance is exceedingly large. Therefore, we conclude that, with appropriate initialization variances,
3.5. Starting Point of LoRA Fine-Tuning
Previous studies have unintentionally employed the non- zero initialization for LoRA (Li et al., 2024; Meng et al., 2024; Wang et al., 2024). LoftQ (Li et al., 2024) fine-tunes a quantized pretrained model and uses quantization errors to initialize the LoRA module. Specifically, it generates the initialization values for LoRA by performing truncated Singular Value Decomposition (SVD) on the quantization errors, thereby minimizing these errors in the pretrained model. PISSA (Meng et al., 2024) suggests performing SVD on the pretrained weights and initializing A and B using the singular vectors and largest singular values. It believes that updating the principal components of the pre- trained model accelerates convergence and enhances perfor- mance. LoRA-GA (Wang et al., 2024) proposes initializing
conjecture that introducing
6

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
MRPC- Init[A]
CoLA - Init[A]
SST2 - Init[A]
QNLI Init[A]
30.4
22.36
24.12
29:49
87.68
84.56
15,38
25.81
26.52
17.45
78.15
78.35
44 30
94.94
94
30
34.13
41:46
96.340
4%.18
41.15
93.12 95.3
12.87
93:18
82.06
76.00
94.150
93.04
Learning Rate in)
30
33.00
79.17
73.53
93.960
90.41
3e-5
77.04
09.54
70.51
03.02
93.297
02.92
85
10.5-
KA. 99
89.14
41/0K
92.634
42.24
90.70
9--6
70.55
61:20
92
91.970
75.85
3--5
68.38
30.32
59.04
00:07
91.907
85.75
11.00
20-7
K4--
65.04
41.92
54.81
90.643
81.21
86.15
63.04
$0.59
90.020
62.67
60.54
49.38
89.790
59.91
16
16
Initia inrtaliiaaionnooo (B)
Initialization (B)
Inibia lizatiin Size
Initializzatin Size (B)
MRPC- [Init[AB]- Init(A])
CoLA- - (Init[AB]- Init[A])
SST2 (Init[AB] - Init[A])
QNU (Init[AB] Init[A]]
Leaming Rate (n)
1e-4-
9:35
2:30
5.00
13.54
11.40
808
2:40
B:14
2:93
2b
5.535
6.240
344
6:23
0:01
0:30
0:16
0:00
0.62
1.128
9,07
7.570
9e-5
20-5-
6--6-
14.40
8.01
7.42
9:55
5.58
4,00
0.20
4.000
3.390
2.438
0.27
0.23
0:39
0.310
8/30
0.12
3.072
2.802
3.037
6.230
2e-6
2,12
4.86
1.733
1.30
0.012
5.79
-0.230
36-7-
1e-7
8:00
1.29
0.00
2:34
0.030
-0.520
A.2
0.20
9:16
-0.970
000060
4
16
8
8 16
2
16
Initiallzation Size (B)
initializattion Size (B)
Inttaalization Size (B)
Initialization Size (B)
Figure 4. Fine-tuning accuracy of TS-Base on different datasets with various learning rates (1) and initialization sizes (3). Note that the initialization variance is set to (Box)?, where of is the variance of matrix A when initialized with Kaiming initialization. Ini t [AB]- Init [A] denotes the accuracy improvement achieved by non-zero initlization, where B is initialized with the same variance as A.
there is no need to subtract the non-zero initialized LoRA from the pretrained model. Further verification will be con- ducted in subsequent experiments.
is generally minimal compared to the performance gains achieved by non-zero initialization.
4.1. Natural Language Understanding Tasks
4. Experiments
Models and Datasets. this section, we fine-tune the T5- Base model (Raffel et al., 2020) on several datasets from the GLUE benchmark, including MRPC, CoLA, SST-2, and QNLI. The experiment is conducted based on the code provided in (Wang et al., 2024). Each experiment is run three times, and the average test accuracy is reported.
In this section, we evaluate our findings across three bench- marks. First, we evaluate natural language understanding tasks using subset of the GLUE benchmark (Wang al., 2019). Next, we evaluate natural language generation tasks using the commonsense reasoning and arithmetic reasoning benchmarks, as proposed in (Hu et al., 2023). Our primary objective is to examine the impact of zero and non-zero initialization on fine-tuning performance across different learning rates and initialization variances. Let of repre- sent the variance matrix A under Kaiming initialization (i.e.. of = 4). In our experiments, the initialization vari- ance is set to (2 where B controls the initialization size. For non-zero initialization, B is initialized with the same variance as A. By default, we subtract the non-zero initialized LoRA from the pretrained weights. The effects of omitting this subtraction process will be discussed in Section 4.3. More experimental details and results can found in Appendices and F. It is worth noting that the standard deviation for the GLUE dataset ranges from 0.01 to 0.4, while for commonsense and arithmetic reasoning tasks, it spans from 0.2 to 0.4. In our experiments, smaller learning rates tend to converge less effectively and exhibit higher standard deviations. However, this standard deviation
Results. Figure 4 presents the test accuracy of T5-Base on the GLUE benchmark. As the initialization variance in- creases, Init [A] exhibits enhanced robustness at smaller learning rates, aligning with our theoretical predictions. Notably, 3 corresponds to the Kaiming initialization (v[AA] - 1), which is smaller than the theoretically opti- mal value of [[A]] = -1. Moreover, initializing B with the same variance as A (i.e., Init [AB]) further improves performance over Init [A] at smaller learning rates, re- sulting up to a 10% accuracy improvement. Figure 5 illus- trates the convergence curves of Init [A] and Init [AB] tonvergeeee
the QNLI dataset with a learning rate of 3e-7, where Init [AB] accelerates convergence by nearly factor of two. Additionally, even when the learning rate is near op- timal, Tnit t [AB] still yields a 1% accuracy improvement. A heatmap for Init [AB] is provided in Appendix E.
7

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Commonsense Init[A]
Arithmetic Init[A]
0.85
Init[A]
36-4
68.23
75.48
Init[AB]
1e-4
0.80
2X SpeedUp
Learnngg Rote
74.64
63.40
Accuracy
1e-5
65:54
83.72
84.05
65.99
88
06
75
07.53
55.72
0.75
10.35
10.6
12.00
66
23.65
43.01
2.6.44
26.14
76
32.86
0.70
25.01
3e-7
28.91
22.78
0.25
0.5
0.25
0.5
0.65
Initialization Size (8)
Initialization Size (9)
Commonsense [Init[AB] Init(A]) Arithmetic (Init]AB) Init[A])
0.60
500
1000
Training Steps
1500
2000
2500
3000
Se4
29.81
25.38
23.32
27.45
Learning Rabe
Bes
16.83
1e-5
3-32
12.09
9.75
13.29
Figure 5. Test accuracy of T5-Base finetuned on the QNLI dataset with learning rate 1) and initlization size 16.
123
7.67
11
1e-6
18:12
7.3%
6.56
3.24
10:00
20.23
18.16
3.26
0.28
-0.2
-1.93
0.25
0.5
Initialization Size (B)
Initialization Size (B)
4.2. Natural Language Generation Tasks.
Models and Datasets. In this section, we fine-tune the Llama 3-8B model (Dubey et al., 2024) on commonsense and arithmetic reasoning benchmarks. The former contains eight subtasks, while the latter contains six subtasks. Each subtask is associated with a predefined training and test set. The experiment is conducted using the code provided in (Hu et al., 2023). Following the procedure outlined in (Hu et al., 2023), we combine the training datasets of all subtasks to create a final training dataset and evaluate each task using single test dataset. The average test accuracy is reported, and task-specific accuracies are provided in Appendix E.
Figure 6. Fine-tuning accuracy of Llama 3-8B.
racy drop duc to excessive noise. However, when suitable initialization variances are applied, there is no discernible difference between Init [AB+] and Init [AB]. Further- more, Figure demonstrates that the range of suitable ini- tialization variances is relatively wide, encompassing the variance used the Kaiming initialization. More ablation results for different models and datasets are in Appendix E,
SST2 (Init[AB+] Init[AB))
QNU (Init[AB+] Init[AB]]
Results. Figure illustrates the average test accuracy of Llama 3-8B on commonsense and arithmetic reasoning benchmarks. The results are consistent with those observed on the GLUE benchmark, where Init [AB] generally out- performs Init [A], particularly smaller learning rates. Moreover, the performance improvement observed with
Learning Rate [g)
0.00
1.49
0.02
-9.16
2.2
-5.21
Ie-5
0.04
11.40
-1.66
21.37
-207
15.30
10.6-
33.38
27.33
32003
25.33
3e-7
-30.58
10.74
Init [AB] on these two benchmarks is notably greater than that on the GLUE dataset. This is likely due to the in-
16-7
-1.17
8 10
2
8
Initialization Size (8)
Initialization Size (B)
creased complexity of these two benchmarks, which impose higher demands on the learning rate robustness.
Figure 7. Fine-tuning accuracy with different starting points. Init [AB+] denotes the variant which do not subtract non-zero initialized AB from W, and Init [AB+ J-Init [AB] indicates the accuracy gap hetween these two initialization strategics.
4.3. A Ablation on the Starting Point of Fine-Ture Fine-Tuning
By default, we subtract the non-zero initialized AB from the pretrained weights W to ensure that fine-tuning starts from the pretrained model. this section, we investigate whether this subtraction is necessary. The strategy of non- zero initialization without this subtraction process denoted as Init [AB+]. The advantage of Init [AB+] is that it avoids modifying the pretrained model, rather than improv- ing accuracy. Figure 7 illustrates the differences between Init [AB+] and Init t [A3] under varying initialization variances. When the initialization variance exceeds a certain threshold (B 2 8), Init [AB+] leads significant accu-
5. Related Works
5.1. Low-Rank Adaptation
The fine-tuning performance of LoRA is influenced by sev- eral factors, such as rank, gradient, learning rate, and initial- ization. AdaLoRA (Zhang et al., 2023) adaptively adjusts the rank across different LoRA layers to improve efficiency. Zeng Lee (2024) provide theoretical guidelines for select-
8

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
ing an appropriate rank in LoRA. BoRA (Li al., 2025) adopts block-wise mechanism in LoRA, leveraging block matrix multiplication with block-wise diagonal matrices to enhance its expressivity. LoRA-Pro (Wang et al., 2025) improves LoRA by adjusting low-rank gradients to better match full fine-tuning, reducing the performance gap. Ad- ditionally, LoRA+ (Hayou et al., 2024a) analyzes how a larger learning rate for the matrix B promotes more stable and efficient LoRA fine-tuning. A key advantage of LoRA+ is its ability to achieve internal stability, i.e., Za- (()), which cannot be achieved using a uniform learning rate for A and B. Notably, LoRA+ can also be applied to non-zero initialization to ensure internal stability, as further discussed in Appendix B.2.
2022; Wang al., 2022) introduce small trainable mod- ules into the pretrained model and update only these com- ponents, significantly reducing training costs. However, the added layers can increase inference latency. In con- trast, soft prompt-hased methods (Lester et al., 2021; Li & Liang, 2021; Razdaibiedina et al., 2023) prepend task- specific, learnable tokens to the input, leveraging the pre- trained model's existing knowledge for adaptation. Despite their efficiency, these methods also incur additional com- putational overhead and inference delays. By comparison, LoRA A supports manual integration of weight updates into the pretrained weights post fine-tuning, thereby avoiding extra inference latency. Additionally, sparse fine-tuning methods such as BitFit (Zaken et al., 2022) update only the model's bias terms, offering another lightweight alternative.
Hayou et al. (2024b) have analyzed the effect of initializa- tion on LoRA's fine-tuning dynamics, though their analysis is limited to zero initialization. In contrast, we further in- vestigate and demonstrate that non-zero initialization can enhance LoRA's robustness to suboptimal learning rates. In fact, non-zero initialization has several practical applica- tions, such as initializing the matrices A and with quan- tization errors (Li al., 2024), the main components of pretrained weights (Meng al., 2024), or the gradients of pretrained weights derived from several data samples (Wang et al., 2024). Recent work (Ponkshe et al., 2024) has highlighted the benefits of using approximate gradients for LoRA initialization. LoRA-One (Zhang et al., 2025) lever- ages spectral initialization to achieve subspace alignment and generalization guarantees at initialization, enabling effi- cient convergence from the very start.
6. Conclusion
In this paper, we revisit the fine-tuning dynamics of LoRA and identify that stable and efficient fine-tuning requires specific conditions on both the learning rate and initializa- tion. Compared to initialization, LoRA imposes stricter requirements on the learning rate. Further analysis reveals that initializing both A and B of LoRA to non-zero values (i.e., non-zero initialization) can improve its robustness to suboptimal learning rates, particularly small ones. Note that this is zero-cost adjustment of LoRA. Additionally, we examine the noise introduced by non-zero initialization for pretraining weights. Extensive experiments demonstrate that with appropriate initialization variances, the noise from non-zero initialization can be effectively corrected through fine-tuning, without negatively impacting fine-tuning per- formance. Our experiments reveal that the range of suitable initialization variances is relatively broad, encompassing commonly used methods like Kaiming initialization.
However, these studies overlook crucial distinction be- tween their approaches and standard LoRA, namely, the difference between zero and non-zero initialization. Un- like LoRA-GA and PiSSA, which were motivated by intu- ition, our approach is motivated by the theoretical analysis of LoRA's fine-tuning dynamics. From the solution set in Eqs.(5-6), we observe that stable and efficient learning imposes stricter constraints on learning rates, whereas the initialization space is more flexible. Traditional zero ini- tialization Br] = -oo) is merely an extreme case. This motivates us to reconsider the necessity of zero initialization and explore the potential benefits of non-zero initialization. Notably, our motivation and claims are not in competition with prior non-zero initialization methods, such as LoRA- GA and PiSSA. Instead, our findings offer theoretical foundation for their effectiveness and provide an explana- tion for the observed performance improvements.
A Acknowledeements
This work is supported by the National Key Re- search and Development Program of China under grant 2024YFC3307900; the National Natural Science Founda- tion of China under grants 62376103, 62302184, 62436003, and 62206102. Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Sci- ence and Technology Talent Service Project under grant 2024DIC078; and Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology.
Impact Statement
5.2. Other Parameter-Efficient Fine-Tuning Methods
This paper explores the non-zero initialization of LoRA, which has the potential to improve the performance of LoRA fine-tuning. Our work builds upon LoRA, and we assert that it does not introduce any negative social implications that would necessitate further discussion.
In addition to LoRA, two widely adopted PEFT strate- gies are adapter-based and soft prompt-based approaches. Adapter-basee based methods (Houlsby et al., 2019; He et al.,

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
References
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehle- van, C. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In International Conference on Learning Representations (ICLR), 2024.
Hu, Z.. Wang, L., Lan, Y., Xu, W., Lim, E., Bing, L., Xu, X., Poria, S., and Lee, R. K. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large lan- guage models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. (EMNLP), 2023.
Bottou, L. Large-scale machine learning with stochastic gradient descent. In International Conference on Compu- tational Statistics (COMPSTAT), 2010.
DeepSeek-AI. Deepseek-v3 technical report. CoRR, 2024.
Deng, L.. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 19(6):141-142, 2012.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., et al. Scaling laws for neural language models. CoRR, 2020.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., et al. The Ilama 3 herd of models. CoRR, 2024.
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), Interna- tional Conference on Learning Representations (ICLR), 2015.
Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 315-323, 2011.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 0045-3059) 2021.
Hayou, S., Doucet, A., and Rousseau, J. On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning (ICML), 2019.
Li, S., Luo, X., Wang. H., Tang, X., Cui, Z., Liu, D., Li, Y., xiuqiang He, and Li, R. Bora: Towards more expressive low-rank adaptation with block diversity. CoRR, 2025.
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICMI.), 2024a.
Hayou, S., Ghosh, N., and Yu, B. tion on lora finetuning dynamics. In Advances in Neurul Information Processing Systems (NeurIPS), 2024b. 20244.
The impact of initializa-
Li, X. I. and Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, (ACL/UCNLP)) 2021.
He. J., Zhou, C., Ma, X., Korr-Kirkaatrikk T., and Neubig, G. Towards a unified view of parameter-efficien trans- fer learning. In International Conference on Learning Representations (ICLR), 2022.
Li, Y. Yu, Y., Liang, C., Karampatziakis, N., He, P., Chen, W., and Zhao, T. Loftq: Lora-fine-tuning-wwee quantiza- tion for large language models. In International Confer- ence on Learning Representations (ICLR), 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ima- genet classification. In IEEE International Conference on Computer Vision, (ICCV), 2015.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations (ICLR), 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, (CVPR), 2016.
Meng, F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T.. et al. Training compute- optimall large language models. CoRR, 2022.
OpenAI. Gpt-4 technical report. CoRR, 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficien transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019.
Ponkshe, K., Singhal, R., Gorbunov, E., Tumanov, A., Horváth, S., and Vepakomma, P. Initialization using update approximation is silver bullet for extremely effi- cient low-rank fine-tuning. CoRR, 2024.
10

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:::: 2020.
Yang. G., Hu, E. Babuschkin, I,, Sidor, S., Liu, X., et al. Tensor programs V: tuning large neural networks via zero- shot hyperparameter transfer. CoRR, 2022.
Yang. G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: feature learning in infinite depth neural networks. In International Conference on Learning Representations (ICLR), 2024.
Razdaibiedina, A., Mao, Y., Khabsa, M., Lewis, M., Hou, R., Ba, J., and Almahairi, A. Residual prompt tuning: improving prompt tuning with residual reparameteriza- tion. In Findings of the Association for Computational Linguistics (ACL), 2023.
Zaken, E. B., Goldberg, Y., and Ravfogel, S. Bitfit: Sim- ple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
Schoenholz, S. S., Gilmer, J., Ganguli, S., and Sohl- Dickstein, J. Deep information propagation. In Interna- tional Conference on Learning Representations (ICLR), 2017.
Zeng, Y. and Lee, K. The expressive power low-rank adaptation. International Conference on Learning Representations (ICLR), 2024.
Sheng, Y., Cao, S., Li, D., Hooper, C.,, Lee, N., Yang. S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J. E., and Stoica, I. S-lora: Serving thousands of concurrent lora adapters. CoRR, 2023.
Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y.,
Chen, W., and Zhao, Adaptive budget allocation for parameter-efficient fine-tuning, In International Confer- ence Learning Representations (ICLR), 2023.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019.
Zhang, Y.,
Liu, F,, and Chen, Y. One-step full gradient suffices for low-rank fine-tum fine-tuning, provably and efficiently. CoRR, abs/2502.01235,, 2025.
Wang, S., Yu, L., and J. Lora-ga: Low-rank adaptation with gradient approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Wang, Y., Agarwal, S., Mukherjee, S., Liu, X., Gao, J., Awadallah, A. H., and Gao, J. Adamix: Mixture-of- adaptations for parameter-efficient model tuning. In Pro- ceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP), 2022.
Wang, Z., Liang, J., He, R., Wang, Z., and Tan, T. Lora-pro: Are low-rank adapters properly optimized? Interna- tional Conference on Learning Representations (ICLR), 2025.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. CoRR, 2017.
Xu, I.., Xie, H., Qin, S. J., Tao, X., and Wang, F. L. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. CoRR, 2023.
Yang, G. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient indepen- dence, and neural tangent kernel derivation. CoRR, 2019.
Yang, G. Tensor programs III: neural matrix laws. CoRR, 2020.

---

Beyond Zero Initialization:
Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
A. Background
A.1. Scaling Theory of Neural Networks
In neural networks, scaling refers to enhancing the model's representational capacity by increasing the scale of a specific structure within the model (Kaplan et al., 2020; Hoffmann et al., 2022). It involves analyzing and optimizing several factors, such model depth, width, training sample size, and the number of training steps. This paper focuses on scaling the model width, which holds practical significance, the most advanced L.MSs today employ extremely large widths, with hidden layer dimensions often reaching tens of thousands (OpenAI, 2024; AeepSeek-AI, 2024).
As the width increases, the synergistic adaptation of parameter initialization and training dynamics becomes increasingly critical. Previous studies have demonstrated that the variance of parameters should be scaled by 1/1n to prevent gradient instability during optimization (He et al., 2015). These scaling rules are typically derived from asymptotic analyses of key quantities (e.g., pre-activation values)) which theoretically guide adjustments to parameter initialization schemes, learning rates, and even network architectures (Hayou al., 2019; Schoenholz al., 2017; Yang, 2019).
Building this foundation, Yang et al. (2022) proposed the Maximal Update Parameterization (uP), which constructs scaling rules for initialization schemes, learning rates, and network architectures to theoretically guarantee stability and maximal feature learning in the limit of infinite network width. The key conditions for ul are as follows: 1) Stability: for all layers I, Yi = (1), where the asymptotic notation ((-) is defined in Section 2.1; 2) Feature learning: AY; = (11, where A represents the parameter update after one gradient descent step. In general, ,PP specifies the parameter weights to be randomly initialized at a scale of (n (112), with weight updates at a scale of (n ). The random initialization and update scales for input and output weights should be (1) and ((n 11), respectively. This scaling rule successfully resolves the instability issues inherent in traditional parameterization schemes, making it a key tool in the theoretical analysis and practical application of large models.
A.2. The y-operator
In neural network scaling theory, the asymptotic behavior of key quantities is typically analyzed when scaling specific model components. For example, when scaling the network width, the objective is to quantify how certain network properties evolve as the width n grows. In this context, asymptotic notations, such as e(-), are particularly useful, as defined in Section 2.1. This standard approach has been employed to derive scaling rules for various components, such as initialization (Schoenholz et al., 2017), activation functions (Hayou et al., 2019), and network parameterization (Yang et al., 2024).
Assuming that the weights are initialized as O(n 8) (8 2 0), and that the learning rate scales polynomially with respect to the width n, it follows that key quantities, such as pre-activations, gradients, and weight updates, are also asymptotically polynomial in n. This regularity arises from the aggregate nature of neural network operations. where tensor operations involve linear combinations of numerous parameters. Consequently, we introduce the 7-operator to express the polynomial behavior as e (()(e)). We now define some basic operations using the --operator.
Zero: 700 = - oo, corresponding to the limit of 7[m=] when 3 - -00.
Product: For two real-valued variables u, o' R, we have y/ov'] = >/v] + 7/07.
Addition: For two real-valued variables u, o' R and v # -v', we have 7/v o'] (7()))))))(
B. Fine-Tuunng Dynamics of LoRA with Adam
B. Detailed Proof
In this section, we present a detailed explanation of the process from Eq. (3) to Eq. (5). To facilitate the analysis, we first introduce Lemmas and 2.
Lemma 1. Let U E R and V E IR"" where is independent n. IJU = (n²)) and V = =((N), then UV = O(n
Proof of Lemma 1. By the definition of the notation θ, each element of U and V satisfies the following relations:
U. = 0(n), V, 0(=)("", [
(7)
12

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
There exist positive constants Kig si Ny and nit such that
n U,+ < nnn R.n < Vi < kyne
(8)
Multiplying U, and Vi, obtain
R +B <U.. < t Ki
(9)
Thus, the product UV satisfies the following inequality:
Σ UM < rehibnats < UV
(10)
Therefore, by the definition of the notation O we conclude that
UV = ( + 3), +(UV] - a
(11)
Note that the same reasoning and result hold for the matrix multiplication of U E Raxr and U E R Lemma 2. Let U E 12 and V R². U O(r²) and V = =(nn), then UV = O(n
Proof of Lemma 2. Similar to the proof of Lemma 1, the product UV satisfies the following inequality:
K A < UV = Σ U.V. < K
(12)
Thus, by the definition of the notation θ we conclude that
UV O(a B yUV|= = a B
(13)
Note that the same reasoning and result hold for the matrix multiplication of U E Rax= and U E gaxb
Proof of the process from E3.(3) and (5). Eq. (3) outlines the conditions for LoRA to achieve stability and efficiency, as follows:
{ 71 Bl--n 19' 1-1 Z 0
(6) @(1))
>|8? = > 1 =0 (6) = 0(1))
(14)
1/2 11 = BB =0 (Z) +(1))
Taking 87 as an example, where 7/4 E R. B Raxr 94 Brxn and Z E R expand 7(07] as follows:
>69 = 71 11 19'
(0)
< 7[nA] + 1BBi-19' 1-1 z
(6)
5 7/1a| + + + >19 -1 'z
(0)
(15)
2 7(1a] + +B 1] + 7 + + +1
=7(ma] (Bo + 718 Σ gel + 1 1 +1
(4)
2 ?[nA] + max >(m1)) + 1,
where (a) follows from the multiplication properties of the -ooprraoor (b) follows from Lemma 1, (c) follows from Lemma 2, and (d) follows from the fact that >[9,] 7[gB] = 7 = 0.
Similarly, performing the same analysis on 67 and Zn yields the following relationships:
| = max ( [b] >/m=)) + > mna 0,
|| = Ao > + + 1
(16)
2/2--11 = m)) + max(>(Bu), >[n0]) + =
13

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Furthermore, we obtain:
Y|0!| + - v(ra] + + + 0.
(17)
Substituting this equation back into 7|6]] and v|6,7| yields:
max( 7(va)) = =/na]
(18)
max(((Bu), = >/n8l-
Thus, we conclude that:
[[ma]] + +|mB]] = =-1, >[Ao] S rlaal, r[Bo] - >[nB].
(19)
B.2. Internal Stability
In this paper, we focus solely on the stability of LoRA's final output, Bn while excluding the stability of LoRA's internal output, ZĄ. Hu et al. (2022) discusses ZA and demonstrates that init [B] ensures stability for both ZA and ZB, but fails to facilitate efficient updates of Z.B. as defined in Definition In contrast, init [A] achieves stability and efficient updating of ZB. but does not ensure the stability of Zz. Here, we additionally consider the internal stability by introducing the condition A²A1 A (1), requirement in Eq. (3).
{ -1 4-1 2 = (6) ((1))
>6,7 = -1111198 1-1 A = =0
(6) 0(1))
(20)
11Z'/ = 1- =0
(Z'z = =(())
- =0
(Z =(1))
Similar to the proof in Section B.1, express the above equations in terms of (Ao. B₂) and (NA: mB)
follows:
max
+ ][ (7BB))
0,
max
max(7[4x) (>[4o)) +>>_>| + O,
max
max((1Aa)) (7[Ao),, + max(:]Bo)) ((ne)) + = 0,
max max(?(44))-7(xal) 11
Based on the derivation steps outlined in Appendix B.1, the solution set can be obtained as follows:
7lmaa -1, 7 [mna] 0, [/4b] S >(a)] = 1, T S >[ns] = 0,
(21)
where y[Ao] and > [Bo] cannot both be equal to -oo simultaneously, as this would imply that both A and B are zero, preventing optimization via gradient descent algorithms.
This is actually the solution proposed in LoRA+ (Hayou al., 2024a), which sets 7 [nA] = - 1, 7[ne] Similar the analysis in Section 3.3, setting [[Ao] -1 and [Bob] can achieve the Robust LoRA defined in Theorem 1. Thus, non-zero initialization can also improve the robustness of LoRA+ in relation to the learning rate.
C. Fine-Tuning Dynamics of LoRA with SGD
In this section, we examine the fine-tuning dynamics of LoRA using the SGD optimizer. For simplicity, we consider the model described in LoRA+ (Hayou et al., 2024a), which can be expressed as:
f(z) ba. x,
(22)
where the pretrained weights are assumed to be zero, x E T is the input vector, and b E R, a E R are the LoRA weights. Additionally, we assume that the fine-tuning dataset consists of a single data point, (a,, y), with æ = O(1). The objective minimize the loss function L(x) (f(()) - 2 The corresponding gradients are as follows:
9h b.(f((()) y)x, 9i = a x(f(x) v).
(23)
14

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Using Lemmas and 2, we have
vlo0| >6",, b1ob0 >-a'' +
(24)
Notably, the Adam optimizer guarantees that 94 - (1) and 9B (1) through gradient normalization, whereas SGD does not ensure this. Let U, = fx(x)) At step 4, the update of LoRA's output is given by:
fft = f((x) - f(-1(z)= = nab babz 11-11111111111 To
+
(25)
To ensure stability and efficiency as defined in Definitions and 4, the following conditions must hold for f 1:
7600| = -| 7|m]| + + 0 (0) = @()))
> = " 7[m]] ( = (
(11m)=+2)))) + +2
(6} = (
(26)
([=1())=((1)
Further, we have the following relation:
|oH| + 1 - 27[f(-1(z)] = >[1a]] + 7//ml + =
(27)
which is the same as the result obtained with the Adam optimizer. Using a uniform learning rate for both a and b we have
>lnal = 7/m6] =- 7/ = T 7|b--1= - 4. (t > 1).
(28)
However, the values of y[ao] and > [/oo] are unknown. Therefore, we take a1 ag + naga and bj bo + mog6. leading
r[ai] = max 7/m] >[ = (7 - + x = 1.
(29)
[661]] = max(>[60) [[m]] + +[o81} = max(>/6ol; - 2 7|a + 1} = 1
Solving this system of equations yields the following conditions:
[[ao] s (ybbll I - I+
(30)
where at least one of the inequalities must hold with equality. Notably, when both inequalities hold with equality, the conditions for robust LoRA, as defined in Theorem 1, are satisfied. Specifically, when ?[ao] = - and 7[bo] = - 4. for all
y inal= < -1/2. < -1/2, we have the following result:
{ = 7 >/naa + 1
3
7|681 = 7 [mo] + 1
(31)
(x))-1(1x) 0,
where 7607] depends only on 7/za), [62] depends only on 7/mu], and (x)] is independent of both 7[no] and > 7mm]--
D. Experimental Settings
D.1. Toy Model
Model and Dataset: The toy model used in Section 3.4 is defined as follows:
f(x) F(WW + (A)
where X e Rdd is the input data, Win E gaxd, Wo E R2x" and W... E 1 are the pretrained weights, A E grx and B E Raxr are the LoRA weights, and F denotes the ReLU activation function(Gloroo et al., 2011). The MNIST (Deng, 2012) and FashionMNIST (Xiao et al., 2017) datasets are used for pretraining and fine-tuning, with input images flattened into one-dimensional vectors. The model dimensions (d. n,r) r) are set to (784, 4096, 32).
15

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Training Algorithm: Both pretraining and fine-tuning employ the AdamW optimizer (Loshchilov & Hutter, 2019), with 81 0.9, 82 0.999, < le 8, and weight decay of O. The batch size is set to 64. During pretraining, the learning rate is set to le 3. with 2000 training steps. During fine-tuning, the learning rate is tuned among the set {3e 4. - 4, 5e - 4, 6c 4, 7e 4, 8e 4.9c - 4, 1e - 3, 2e 3. 3e 3). with Wo. Wi. Woue frozen. Each learning rate is trained for 100 steps. All experiments are conducted with 10 independent repetitions using different random seeds, and the final results are averaged over the test accuracy.
Initialization Strategy: During pretraining, Wo, W in: Wout are initialized using Kaiming initialization (He et al., 2015). During fine-tuning, A and B are initialized with a Gaussian distribution, with mean 0 and standard deviations ол and B, respectively. For Init [A], a is set to 0, and a is tuned among the set {2e - 5, 5e - 5, Se - 5, le 4, de - 4, 7e 4, 1e - 3, 3e 3, 6e 3, 9e 3, 2e - 2, 5e 2). For Init [AB]. OA and 0g are kept equal, and both standard deviations are tuned among the same set. Note that the non-zero initialized AB not aubstracted from Wo.
D.2. Experiments on Natural Language Understanding.
Model and Dataset: T5-Base (Raffel et al., 2020), with sequence length T = 128, is fine-tuned on several datasets from the GLUE benchmark (Wang et al., 2019), including MRPC, CoLA, SST-2, and QNLI.
LoRA Hyperparameters: The rank set to r 16 and 16. LoRA is applied to the "query" and "value" weights, using full precision (FP32).
Training Algorithm: AdamW (Loshchilov & Hutter, 2019) with 31 = 0.9, 32 = 0.999, e = le- 8 and weight decay
of The learning rate is tuned among the set (le- - 7, - 7, le - 6, - 6, le 5, 3e 5, le - 4, 3e 4], a warmup ratio of 0.03 employed. Number of train epochs E = 1.
Initialization Strategy: The initialization variance is set to (3 where of represent the variance of matrix A under
Kaiming initialization (i.e., of = 1). 3 is tuned among 2,,2,4, 8, 16). For non-zero initialization, B is initialized with the same variance as A. By default, we subtract the non-zero initialized LoRA from the pretrained weights.
D.3. Experiments on Natural Language Generation.
Model and Dataset: Llama 3-8B (Dubey et al., 2024), with sequence length T = 256, is fine-tuned on the common- sense reasoning and arithmetic reasoning benchmarks, as proposed in (Hu et al., 2023).
LoRA Hyperparameters: The rank is set to r 16 and 16. LoRA is applied the 'q-proj', 'k.proj', 'v-proj'. 'up-proj', and 'down.proj' weights, using full precision (FP32).
Training Algorithm: AdamW (Loshchilov & Hutter, 2019) with 31 = 0.9, 32 0.999, e le - and weight decay of The learning rate is tuned among the set {3e 7, 1e 6, 6, 1e 5, 5, 4, 3e 4), a warmup ratio of 0.03 is employed, and linear decay are employed. Number of train epochs E
Initialization Strategy: The initialization variance is set to (B0x)), where of represent the variance of matrix under Kaiming initialization. 3 tuned among (0.25, 0.5, 1, 4). For non-zero initialization, B is initialized with the same variance as A. By default, we subtract the non-zero initialized LoRA from the pretrained weights.
E. Additional Experimental Results
In Section 4, we present the accuracy heatmaps for Init [A] and Init [AB]-Init [A]. Here, we directly display the heatmaps for Init t. [AB], including for the T5-Base model, as shown in Figure 8, and for the Llama 3-8B model, as depicted in Figure 9. Additionally, we provide the accuracy gap between Init [AB+] and Init [AB] on the MRPC and CoLA datasets. As shown in Figure 10, there is no significant difference between Init [AB+] and Init [AB] under appropriate initialization variance. To verify the impact of non-zero initialization on LoRA+ (Hayou et al., 2024a), we compared the accuracy of Init [AB] and Init [A] under various settings. Figure 1I that non-zero initialization can also improve LoRA+'s robustness and accuracy.
16

---

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
MRPC Init[AB]
Init[AB]
Init[AB]
QNLI
Init[AB]
3e-4
22.05
73
82
04.07
86.
80
76
75.0
79
94:42
90.61
96
54
94.
96.720
93:20
93.23
93.26
85
92.
Learning Rate (2)
10.4
83.58
88:06
77.66
77:15
04.72
94.419
93.06
93.06
Se-5
75.30
81.28
94.210
53.04
72.12
93.618
85.77
10-5
M8
63 44
09.34
51.19
92.916
93.12
01.01
56.5
73.21
92.215
02.68
1--6
60 31
68
59.54
51.97
91.513
91.45
71.48
1e7
K022
621-8
83:28
12/0
50.73
69.47
N011
90.211
89.910
85.41
20:15
63.99
60.59
Initialization Size (B)
Initialization Size (8)
Initialization Size (8)
Initialization Size (8)
Figure 8. Fine-tuning accuracy of T5-Base the GLUE benchmark with non-zero initialization.
Commonsense Init[AB]
Arithmetic Init[AB]
3e-4
Learning Rate (R)
1c:4
09.91
74.24
71.23
3e-5
81.10
64.28
1e.5
72:38
49.06
02
G
72.38
35:83
32.02
66.68
43
40
3e-7
$7.62
24.05
0.25
0.25
Initialization Size (B)
Initialization Size (8)
Figure 9. Fine-tuning accuracy of Llama 3-8B on the commonsense and arithmetic reasoning benchmarks with non-zero initialization.
MRPC (Init[AB+] Init(AB])
CoLA (Init[AB+] Init[AB])
Learning Rate 6/2)
1o.4
11.22
0.25
8.29
3e.5
9.40
0.00
3.00
0.48
10.9
4.99
1e:5
0.74
0.25
0.24
11.60
4.00
2.08
2.20
1.4
0.09
1:22
0:08
-0.28
21.96
4.28
7.71
3e.7
-32.31
11.13
1e7
-38.36
13.42
Initialization Size (B)
Initialization Size (F)
Figure Fine-tuning accuracy with different starting points. Init ] denotes the variant which do not subtract non-zero initialized Ao Bo from W, and Init [AB+]-Init [AB] indicates the accuracy gap between these two initialization strategies.
Arithmetic Init[A] (LoRA+)
Arithmetic (Init(AB)-initAA] (LoRA+)
3e
59.10 59.17 59 59.29
59.100
59.290
0.51
0.65
1.00
0:53
1.344
Learning Rake
1.155
52/22
58.08
58.334
2
0.967
57.940
0.778
57.546
3e-6
00.00
50
08 09.02
56.928
57.200
2000
100
0.00
0:39
0.495
0.306
3e-7
56.53
52.32
50.ar
56.720
56.590
4004
1009
0.44
0.50
0.117
0.25
0.5
0.25
1
Initialization value (8)
Initialization value (B)
Figure Experimental results with LoRA+. The hyperparameter represents the value na/nA (i.e., 1/4 1),18 Any. Both Init [A] and Init [AB] use the initialization size of 3 = 1. that is. o² = of = 1/n, where n is the network width.
17


Reviews:
[{"review_id": "9J9TuV1zKn", "reviewer": "Reviewer_uk4U", "summary": "This paper considers scaling of hyperparameters for LoRA finetuning from an infinite width perspective following [1, 2]. The key difference compared to past works is that a non-zero random initialization of both the B and A adapter matrices is considered. The initialization can  optionally be subtracted from the pretrained weights to ensure the overall layer starts at the pretrained weights. The key observations are that the new initialization scheme allows \"robustness\" in addition to previous desiderata such as stability and efficiency. Informally robustness refers to a lack of sensitivity in the scaling of certain quantities to learning rate hyperparameters. The authors demonstrate on a variety of tasks the superior performance of the new scheme and improved robustness to suboptimal learning rates.\n\n[1] LoRA+: Efficient Low Rank Adaptation of Large Models - Soufiane Hayou, Nikhil Ghosh, Bin Yu\n[2] The Impact of Initialization on LoRA Finetuning Dynamics - Soufiane Hayou, Nikhil Ghosh, Bin Yu", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer uk4U:**\n\nThank you for your detailed and insightful comments. Below, we provide responses to each point individually. Additional experimental results can be found in **https://anonymous.4open.science/r/nzlora_rebuttal-7D3E**.\n\n***Q1: \"typos\"***\n\n**R1:** \nThanks again for catching these! All typos will be fixed in the revision.\n\n***Q2: \"perturbing $\\gamma[\\eta]$ or $\\eta$\"***\n\n**R2:** \nThis question is essential for understanding our analysis. To improve clarity, we restate our infinite-width analysis as follows:\n\n1. In this paper, we focus on the asymptotic behavior of the learning rate, $\\gamma[\\eta]$, as the network width $n \\to \\infty$, rather than its exact value. The $\\gamma$-operator is defined such that $\\eta = \\Theta(n^{\\gamma[\\eta]}) \\approx c \\cdot n^{\\gamma[\\eta]}$, where $c > 0$ is a constant and lower-order terms are neglected.\n2. As $n \\to \\infty$, the term $n^{\\gamma[\\eta]}$ dominates, making $\\gamma[\\eta]$ the key factor determining the asymptotic behavior of $\\eta$. While the constant $c$ is important for exact values, it doesn't influence the asymptotic scaling behavior. Ignoring the influence of $c$, perturbations to $\\gamma[\\eta]$ and $\\eta$ are effectively equivalent.\n3. Thus, we focus on how perturbations to $\\gamma[\\eta]$ affect the fine-tuning dynamics, excluding the constant $c$ in $\\Theta$. \nNote that we analyze $\\gamma[\\eta]$ to guide learning rate and initialization choices, not compute exact values (which depend on $c$).\n\nWe appreciate the opportunity to clarify our analytical framework and will explicitly incorporate these refinements in the revised paper.\n\n***Q3: \"extend beyond the top right corner\"***\n\n**R3:** \nFollowing your suggestion, we have expanded the upper right corner of the heatmap, and the updated results are presented in Fig.16 in the above link. The results show that increasing the hyperparameters further does not lead to a substantial improvement in accuracy.\n\n\n***Q4: \"insights about PiSSA\"***\n\n**R4:** \nOur analysis suggests that PiSSA is more robust to variations in the learning rate, owing to its use of non-zero initialized LoRA, which is achieved through Truncated SVD on pre-trained weights. Fig.11 in the above link shows that a significant portion of the improvement in PiSSA's accuracy can be attributed to non-zero initialization. The remaining improvement is due to the fact that the initialization values derived from the pre-trained weights are more effective than random noise. A similar trend is observed in LoRA-GA, which employs gradients for non-zero initialization. Please refer to Fig.11 for further details.\n\n***Q5: \"clarify the need of \\eta_A = \\eta_B, and the internal stability\"***\n\n**R5:** \nIn fact, the condition $\\eta_A = \\eta_B$ is not necessary for achieving maximum robustness. The solution set in Eq. (5) indicates that stable and efficient learning can be achieved as long as $\\gamma[\\eta_A] + \\gamma[\\eta_B] = -1$, $\\gamma[A_0] \\leq \\gamma[\\eta_A]$, and $\\gamma[B_0] \\leq \\gamma[\\eta_B]$. Under this condition, when both \"$\\leq$\" become \"=\", maximum robustness is attained. By default, we set $\\eta_A = \\eta_B$ since fine-tuning typically employs a uniform learning rate for all LoRA weights. However, achieving internal stability further requires $\\gamma[\\eta_A] = -1$ and $\\gamma[\\eta_B] = 0$, which are the core propositions of LoRA+. Due to space limitations, further details on internal stability can be found in **R3 from reviewer_Rpo6** or in the analysis presented in Appendix B.2.\n\n***Q6: \"LoRA+ with non-zero initialization\"***\n\n**R6:** \nWe integrated LoRA+ with non-zero initialization. As shown in Fig.12 in the above link, non-zero initialization also enhances the robustness of LoRA+ to variations in the learning rate, leading to improved model accuracy.\n\n***Q7: \"initialization variance sensitivity of Init[AB]\"***\n\n**R7:** \nLet's first explain the meaning of each initialization method:\n\nInit[A]:$A_0\\sim \\mathcal{N}(0, \\delta^2), B_0=0$,\n\nInit[AB]:$A_0\\sim \\mathcal{N}(0, \\delta^2), B_0\\sim \\mathcal{N}(0, \\delta^2)$, with \n$\\frac{\\alpha}{r}A_0B_0$ subtracted from the pre-trained weights.\n\nInit[AB+]: Same as Init[AB], but without the subtraction process.\n\nFirst, we emphasize that Init[A] itself exhibits sensitivity to variance. As shown in Eq.(6), stable and efficient learning requires the variance of $A_0$ to satisfy $\\gamma[A_0] \\leq -\\frac{1}{2}$. In Init[AB], $B_0$ uses the same variance as $A_0$ and only needs to satisfy the same condition (i.e., $\\gamma[B_0]\\leq-1/2$). Therefore, Init[AB] does not introduce additional sensitivity to initialization variance but instead enhances the robustness of LoRA to $\\eta_B$.\n\nNotably, if Init[AB+] is used, a larger initialization variance results in greater noise, which negatively impacts performance. However, this issue arises due to the absence of noise subtraction in Init[AB+], rather than a fundamental limitation of Init[AB].", "claims_and_evidence": "Yes the evidence is clear and convincing.", "methods_and_evaluation_criteria": "Yes the methods and evaluation criteria make sense.", "theoretical_claims": "Yes the proofs appear correct.", "experimental_designs_or_analyses": "Yes the experimental designs are solid.", "relation_to_broader_scientific_literature": "The paper is important for understanding the optimal setting of hyperparameters for LoRA finetuning, a popular parameter efficient finetuning method. The paper characterizes the scaling of certain quantities in terms of width and imposes various desiderata for finetuning akin to a variety of works such as [1, 2, 3]. Importantly this work goes beyond previous works by considering a non-zero initialization of LoRA. In particular they show that finetuning can be successful even when the non-zero initialization is not subtracted from the pretrained weights as long as the initialization variance is not too large, demonstrating robustness of the finetuning procedure to a noisy initialization. Furthermore, the non-zero initialization has certain advantages relative to other initializations including decreased sensitivity to learning rate hyperparameters and improved empirical performance.\n\n[1] LoRA+: Efficient Low Rank Adaptation of Large Models - Soufiane Hayou, Nikhil Ghosh, Bin Yu\n[2] The Impact of Initialization on LoRA Finetuning Dynamics - Soufiane Hayou, Nikhil Ghosh, Bin Yu\n[3] Feature Learning in Infinite-Width Neural Networks - Greg Yang, Edward J. Hu", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The strength of this paper is that it expands the practical consideration of initializations for LoRA and offers evidence for the superiority of a new initialization. Empirically this initialization appears to be non-trivially better than the standard practice and is trivial to implement.", "other_comments_or_suggestions": "typos:\nIn Section B.2 \nAppendix Hu et al. reference is incorrect. \nLast line of Eq. (20) should be Z_A^{t-1} not Z_B^{t-1}.\n\nThe presentation in Sections 3.2 and 3.3 are a bit hard to parse at first as is the definition and intent of \"robustness\". I don't think it is about perturbing \\gamma[\\eta] (which doesn't make much sense) but really perturbing \\eta and in certain scalings the perturbation has a dominant quadratic dependence on \\eta.\n\nAlso in the grid sweeps the optimum is in the top right corner. Can you extend beyond that to check that increasing the hyperparameters further does not bring improvements?", "questions_for_authors": "Can the analysis say anything useful about PiSSa?\n\nTo clarify \\eta_A = \\eta_B is needed for maximum robustness? Also in this case internal stability is not achieved?\n\nIf we use LoRA+ and non-zero initialization will we do even better? \n\nUsing Init[AB] can reduce learning rate sensitivity, but it will increase initialization variance sensitivity compared to init[A]?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zY2BFRStLV", "reviewer": "Reviewer_zK3d", "summary": "This paper investigates the impact of non-zero initialization in Low-Rank Adaptation (LoRA) fine-tuning, challenging the conventional practice of initializing one of the LoRA matrices (A or B) to zero. Through theoretical analysis and empirical validation, the authors demonstrate that simultaneously initializing A and B to non-zero values (Init[AB]) enhances LoRA’s robustness to suboptimal learning rates, particularly smaller ones, common due to learning rate decay. The study finds that while non-zero initialization introduces slight noise to the pre-trained model, it does not degrade fine-tuning performance as long as appropriate initialization variances are used. Extensive experiments across models and datasets confirm that non-zero initialization improves accuracy, stability, and convergence speed, making it practical for LoRA-based fine-tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer zK3d:**\n\nThank you for your detailed and insightful comments. Below, we provide responses to each point individually. Additional experimental results can be found in **https://anonymous.4open.science/r/nzlora_rebuttal-7D3E**.\n\n***Q1: \"accuracy's dependence on the learning rate in Tables 2,3, and accuracy differences are sometimes small\"***\n\n**R1:** \nOur analysis reveals that non-zero initialization can reduce the adverse effects of suboptimal learning rates on LoRA performance. This effect is particularly evident when the learning rate is below its optimal value. However, when the learning rate approaches its optimal value, the performance improvement from non-zero initialization becomes less significant.\n\n***Q2: \"no justification for why $\\beta\\in\\\\{1,2,4,8,16\\\\}$\"***\n\n**R2:** \nIn our experiments, we set the initialization variance of matrices $A$ and $B$ as $\\delta_A^2=\\delta_B^2=(\\beta \\delta_k)^2$, where $\\delta_k^2=1/n$ is the variance used in Kaiming initialization (the default setting of LoRA). Notably, $\\beta$ does not strictly represent variance, but rather a scaling factor applied to $\\delta_k$.\n\nOur analysis indicates that robustness improves as $\\gamma[A_0]$ and $\\gamma[B_0]$ approach −1/2. To explore this, we begin with standard Kaiming initialization ($\\beta=1$, corresponding to $\\gamma[A_0]=\\gamma[B_0]=−1$) and systematically increase the variance $(\\beta\\in\\{2,4,8,16\\})$ to study its effects. Our experimental results (Figs.4 and 6 in the original paper) confirm that, within a certain range, increasing the initialization variance enhances robustness to learning rate variations and leads to better accuracy.\n\n\n***Q3: \"limits on the variance in Init[AB]\"***\n\n**R3:** \nThe theoretical limits of initialization variance are $\\gamma[A_0] \\leq -1/2$ and $\\gamma[B_0] \\leq -1/2$. However, this condition only describes the asymptotic behavior of the initialization variance as $n \\to \\infty$, rather than providing a specific value. To further investigate this, we performed ablation experiments on the variance limits of LLaMA 3-8B and T5-base models. As shown in Fig.16, these limits vary across models or datasets (e.g. Init[AB]-Init[A] with $\\beta=4$ is generally less than 0 in the Commonsense reasoning task). However, the variance associated with Kaiming initialization (i.e., $\\beta = 1$) is generally effective, yielding near-optimal accuracy.\n\n\n***Q4: \"different ranks or scaling fators\"***\n\n**R4:** \nWe conducted ablation experiments with varying ranks and scaling factors. As shown in in Fig.14 in the above link, adjusting these hyperparameters does not affect the improvement gained through non-zero initialization.\n\n\n***Q5: \"different fine-tuning settings\"***\n\n**R5:** \nFollowing your suggestion, we conducted experiments using the an instruction tuning dataset, databricks-dolly-15k, and evaluated its performance on the MMLU task. As shown in Fig.15, non-zero initialization enhances LoRA's robustness to small learning rates in the instruction tuning task, thereby improving accuracy. Notably, LLama 3-8B exhibits limited accuracy on MMLU, and thus the improvement due to non-zero initialization is less pronounced. However, the trend is still observable.\n\n\n***Q6: \"LoRA variants\"***\n\n**R6:** \nTo address this question, we evaluated the impact of non-zero initialization on LoRA+ (using larger learning rates for matrices $B$) and HydraLoRA [1], an asymmetric LoRA variant (using one matrix $A$ with multiple matrices $B$). The results are presented in Figs.12-13 in the above link.\n1. We tested LoRA+ on GLUE and Arithmetic reasoning tasks. The results show that appropriately increasing the learning rate of $B$ can indeed improve the model accuracy. Most importantly, for the same learning rate, non-zero initialization significantly enhances the accuracy of LoRA+. \n2. We tested HydraLoRA on Arithmetic reasoning tasks. To ensure that the non-zero initialized $AB$ terms could be subtracted from the pre-trained weights, we use the same initialization for different $B$ matrices within a HydraLoRA layer. As shown in Fig.13, non-zero initialization also improves the robustness of HydraLoRA to the learning rate. \n\n[1] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, NeurIPS 2024.\n\n\n***Q7: \"standard deviations\"***\n\n**R7:** \nThe standard deviation for the GLUE dataset ranges from 0.01 to 0.4, while for commonsense and Arithmetic reasoning tasks, it spans from 0.2 to 0.4. In our experiments, smaller learning rates tend to converge less effectively and exhibit higher standard deviations. However, this effect is minimal compared to the performance gains achieved by non-zero initialization. We will include the full standard deviation results in the revised paper.\n\n***Q8: \"surrounding methods such as BitFit and Adapters\"***\n\n**R8:** \nThank you for your suggestion. We will add a discussion of relevant PEFT methods in the revised paper.", "claims_and_evidence": "Well-supported claims:\n1. Non-zero initialization improves LoRA’s robustness to suboptimal learning rates: This claim is supported by theoretical analysis and empirical proofs\n2. Fine-tuning does not need to strictly start from the pre-trained model: Experiments and theoretical evidence provided\n3. Non-zero initialization achieves superior performance compared to zero initialization, particularly at smaller learning rates: The heatmaps and performance tables demonstrate consistent improvements when using Init[AB] instead of Init[A], especially in low learning rate scenarios.\n\nClaims that need more evidence:\n1. Non-zero initialization improves performance in all cases: There is clearly a dependence on the learning rate for different tasks as can be seen in Tables 2,3\n2. What are the limits on the variance of noise that can be used in the init[AB] case", "methods_and_evaluation_criteria": "The authors test their method with Llama-3 8B and T5 Models on the GLUE and arithmetic reasoning benchmarks. It would be interesting to check how their method works for other fine-tuning settings such as instruction tuning. Also, it is not clear how their method works with varients of LoRA such as Asymmetric LoRA, LoRA+, QLoRA, etc.,", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes, I examined the soundness and validity of the experimental designs and analyses in the paper\n1. The paper evaluates natural language understanding (GLUE benchmark) and natural language generation (commonsense & arithmetic reasoning), ensuring broad applicability.\n2. The paper uses multiple model architectures: T5-Base (encoder-decoder) and Llama 3-8B (decoder-only transformer)\n3. The study systematically varies the learning rate (η) and initialization variance (β), allowing a detailed exploration of their effects.\n4. The heatmaps and accuracy tables provide clear evidence that non-zero initialization (Init[AB]) improves performance, particularly at lower learning rates.\n5. The toy model experiment provides intuitive validation that non-zero initialization reduces sensitivity to learning rate choices.\n\nThere are areas where the text can improve with additional details\n1. The reported accuracy differences (e.g., between Init[A] and Init[AB]) are sometimes small (e.g., ~1%).\n2. No confidence intervals or standard deviations are provided\n3. It’s unclear if different ranks or scaling factors would affect the relative performance of zero vs. non-zero initialization.\n4. The initialization variance (β) is tested in discrete steps (e.g., {1, 2, 4, 8, 16}), but there’s no justification for why these values were chosen.\n5. How does their method interact with version improvements of LoRA such as LoRA+, Asymmetric LoRA, etc.?", "relation_to_broader_scientific_literature": "This paper builds upon existing research in LoRA fine-tuning, neural network scaling, and weight initialization, challenging the conventional zero-initialization approach in LoRA. While prior work (Hu et al., 2022; Hayou et al., 2024a) focused on optimizing learning rates and rank selection, this study demonstrates that initializing both LoRA matrices (A and B) to non-zero values (Init[AB]) enhances robustness to suboptimal learning rates. Applying infinite-width scaling theory formalizes conditions for stable and efficient fine-tuning, extending insights from Kaiming initialization (He et al., 2015) to LoRA. Unlike recent empirical methods that use quantization errors (LoftQ), SVD (PISSA), or gradient-based initialization (LoRA-GA), this paper provides a theoretical foundation for non-zero initialization. It validates it with experiments across T5-Base, Llama 3-8B, and multiple benchmarks. These findings refine LoRA fine-tuning dynamics without additional computational cost, offering a practical and theoretically justified improvement.", "essential_references_not_discussed": "The paper focuses on LoRA and the surrounding methods while not shedding light on other PEFT methods, such as BitFit [Zaken et al., 2022] and Adapters [Houlsby et al., 2019]. Adding these papers can help the reader understand the landscape better.", "other_strengths_and_weaknesses": "See my comments above", "other_comments_or_suggestions": "See my comments above", "questions_for_authors": "See my questions in the analysis part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xf1xptiKYd", "reviewer": "Reviewer_rPo6", "summary": "This paper investigates the impact of non-zero initialization on the fine-tuning dynamics of LoRA. Traditionally, in LoRA, one of the low-rank matrices, A or B, is initialized to zero to ensure fine-tuning starts from the pretrained model. However, this practice lacks theoretical justification. The authors theoretically analyze the effects of initializing both A and B to non-zero values. Their key findings are: (1) Non-zero initialization improves robustness to suboptimal learning rates; and (2) Fine-tuning does not need to strictly start from the pretrained model. The authors validate these findings through extensive experiments across various models and datasets. These results challenge the conventional practice of zero initialization in LoRA and highlight the benefits of non-zero initialization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer rPo6:**\n\nThank you for your detailed and insightful comments. Below, we provide responses to each point individually. Additional experimental results can be found in **https://anonymous.4open.science/r/nzlora_rebuttal-7D3E**.\n\n\n***Q1: \"typos in lines 62 and 799, and Eq (19)\"***\n\n**R1:** \nThank you for your thorough review. We will correct the identified typos and carefully re-examine the entire manuscript.\n\n***Q2: \"how the definitions of stability and efficiency differ from those in previous studies\"***\n\n**A2:** \nThe only difference  between our definitions and those in previous studies is that our stability definition is slightly less restrictive. \nSpecifically, we do not consider interval stability, i.e., $Z_A=AZ=\\Theta(1)$, where $Z$ is the input of the LoRA layer. Instead, we focus on the stability of the final output of LoRA, $Z_B=BAZ=\\Theta(1)$. Eq.(5) outlines the conditions that must be met by the initialization and learning rate when interval stability is excluded. A detailed discussion on interval stability is provided in Appendix B.2. Given that other reviewers have raised concerns regarding interval stability, we summarize the key points related to this topic in **Q3**.\n\n***Q3: \"interval stability\"***\n\n**A3:** \nIn this paper, stability is defined as $Z_B=BAZ=\\Theta(1)$, where $Z$ represents the input to the LoRA layer. The condition $Z_B=\\Theta(1)$ ensures the stability of LoRA's final output, while interval stability is defined as $Z_A=AZ=\\Theta(1)$, which indicates the stability of LoRA's intermediate results. In Section 3, we present the solution set for stable and efficient learning without considering interval stability, as shown in Eq.(5) or as follows:\n\n$\\gamma[\\eta_A]+\\gamma[\\eta_B]=-1, \\gamma[A_0] \\leq \\gamma[\\eta_A]$ and $\\gamma[B_0] \\leq \\gamma[\\eta_B]$.\n\nWhen interval stability is considered (i.e., $Z_A=\\Theta(1)$), an additional constraint is imposed: $\\gamma[\\eta_A]=-1$. Consequently, the solution set of the learning rate and initialization becomes Eq.(21) in Appendix B.2:\n\n$\\gamma[A_0] \\leq \\gamma[\\eta_A]=-1$ and $\\gamma[B_0] \\leq \\gamma[\\eta_B]=0$. \n\nTwo important points should be noted here:\n1. $\\gamma[\\eta_A]=-1$ and $\\gamma[\\eta_B]=0$ are the key findings of LoRA+, which suggest using a larger learning rate for the matrix $B$ in practical applications.\n\n2. Regardless of the optimal value for $\\gamma[\\eta_A]$ and $\\gamma[\\eta_B]$, the conditions $\\gamma[A_0] \\leq \\gamma[\\eta_A]$ and $\\gamma[B_0] \\leq \\gamma[\\eta_B]$  must always be satisfied to ensure LoRA's stable and efficient learning. When both \"$\\leq$\" become \"=\", the maximum robustness to the learning rate is achieved. Therefore, non-zero initialization can also enhance LoRA+'s robustness to the learning rate, as  shown in Fig.12 in the above link.", "claims_and_evidence": "The claims made in the paper are theoretically proven and experimentally verified.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are reasonable. This paper studies the initialization problem of LoRA, a common but previously overlooked aspect of fine-tuning LLMs. The authors systematically compare different initialization methods (zero vs. non-zero) using models, datasets, and code based on published work, ensuring reliability.", "theoretical_claims": "The reviewer carefully checked the theoretical claims and corresponding proofs in the paper, including all results in Section 3 and the proofs in Appendices B and C. To the reviewer, the claims and proofs are reasonable. Although there are minor typos, for example, $\\gamma[A_0]\\leq\\eta_A$ in Eq (19) in Appendix B should be $\\gamma[A_0]\\leq\\gamma[\\eta_A]$, these do not affect the validity of the theoretical results.", "experimental_designs_or_analyses": "The reviewer checked the experimental setup, results, and analysis. As described in the paper, the authors conducted experiments on three standard benchmarks. The experimental setups were based on published work and aligned with general practices in LoRA fine-tuning. The authors primarily analyzed different initialization settings and learning rates, which is consistent with the paper's motivation. The experimental analysis is also reasonable and supports the theoretical findings.", "relation_to_broader_scientific_literature": "Previous work (Hayou et al., 2024b) discussed the difference between initializing A or B to zero but did not explore the rationale behind zero initialization. This paper fills that gap, demonstrating that both A and B can be initialized to non-zero values. These findings provide theoretical support for related LoRA variants, such as PiSSA and LoRAGA, and significantly contribute to LoRA research.", "essential_references_not_discussed": "To the best of the reviewer's knowledge, all related work on LoRA initialization has been covered.", "other_strengths_and_weaknesses": "Strengths:\nThis paper challenges the traditional LoRA initialization method, and studies the significance of non-zero initialization from the perspective of robustness to learning rate. The method is simple yet insightful.\n\nIt also fundamentally overturns the purpose of traditional zero initialization (fine-tuning from pre-trained models). Experiments show that non-zero initialization with appropriate variance does not affect fine-tuning performance, indicating that fine-tuning does not need to start strictly from a pretrained model.\n\nWeaknesses:\nA minor shortcoming is the lack of discussion on how the definitions of stability and efficiency in this paper differ from those in previous studies (e.g., LoRA+). The authors are encouraged to clarify this distinction in the appendix.", "other_comments_or_suggestions": "Typo in line 62: \"raise\" should be \"raises.\"\nIncorrect reference to Llama 3 in line 799.\nTypo in Eq (19):  $\\gamma[A_0]\\leq\\eta_A$ should be $\\gamma[A_0]\\leq\\gamma[\\eta_A]$.", "questions_for_authors": "How  do the definitions of stability and efficiency in this paper differ from those in previous studies (e.g., LoRA+)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GgmoyeYjzd", "reviewer": "Reviewer_wxon", "summary": "This paper studies how non-zero initialization improves the perforamnce of LoRA, especially the stabilitiy.\n\n- The authors define 1) the notation of stabilitity, $BAX = \\Theta(1)$ for all LoRA layers when the width is infinity, where $X$ is the input. 2) the notation of efficiency, the linear update term is $\\Theta(1)$.\n\n- Based on the above two creteria, the author derive the requirements on the random Gaussian variance of A and B as well as the step-size. When using SGD, the optimal initialization under such two creteria is neither classical LoRA initialization nor other variants of zero initialization. \n\n- The author continue to define the robustness of LoRA and derive the similar requirements on them.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer wxon:**\n\nThank you for your detailed and insightful comments. Below, we provide responses to each point individually. Additional experimental results can be found in **https://anonymous.4open.science/r/nzlora_rebuttal-7D3E**. To save space, we denote zero initialization as ***ZI*** and non-zero initialization as ***NZI***.\n\n***Q1: \"the current claim about motivation is weak\"***\n\n**R1:** \n**Unlike LoRA-GA, which was motivated by intuition, our approach is motivated by the theoretical analysis of LoRA's fine-tuning dynamics.** \nFrom the solution set in Eqs.(5-6), we observe that stable and efficient learning imposes stricter constraints on learning rates, whereas the initialization space is more flexible. Traditional *ZI* ($\\gamma[B_0]=-\\infty$) is merely an extreme case. This motivates us to reconsider the necessity of *ZI* and explore the potential benefits of *NZI*. Based on this insight, we conduct further analysis and evaluation, leading to two key findings:\n1. *NZI* can reduce the sensitivity of LoRA to suboptimal (i.e., smaller) learning rates. \n2. The purpose of traditional *ZI*, \"fine-tuning from a pre-trained model\", is not strictly necessary.\n\nNotably, our motivation and claims are not in competition with prior *NZI* methods, such as LoRA-GA and PiSSA. Instead, our findings offer a theoretical foundation for their effectiveness and provide an explanation for the observed performance improvements. Fig.11 in the above link show that the accuracy gains of LoRA-GA and PiSSA primarily stem from *NZI*. These points will be clarified in the revised version of the paper.\n\n\n***Q2: \"the experimental comparison is not sufficient\"***\n\n**R2:** \nFollowing your suggestion, we have added additional comparisons and combinations of LoRA-based methods with *NZI*. Specifically, two key aspects are considered:\n\n1. **Ablation comparison with LoRA-GA and PiSSA.** \nAs shown in Fig.11, a large portion of the accuracy gains in PiSSA and LoRA-GA can be attributed to the use of *NZI*. The remaining gains are due to the fact that initialization values derived from pre-trained weights or gradients are more effective than random noise.\n\n1. **Combination with LoRA+  and HydraLoRA.** We introduced *NZI* for both LoRA+ (using a larger learning rate for the matrix $B$) and HydraLoRA [1] (an asymmetric LoRA that uses one matrix $A$ and multiple matrices $B$). As shown in Figs.12-13, *NZI* enhances the robustness of LoRA+ and HydraLoRA to variations in learning rate and improves accuracy. The relevant settings are detailed in the figure caption.\n\n[1] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, NeurIPS 2024.\n\n***Q3: \"the derivation heavily follows with previous work\"***\n\n**R3:** \nThe derivation used in this paper, including the notation of $\\gamma$ and $\\Theta$ and the definitions of stability and efficiency, is widely employed in infinite-width analysis and can be traced back to Yang et al. (NeurIPS 2021; see line 520 of our paper). Hayou et al. used these tools to explore the effects of learning rate (ICML 2024; line 475 in our paper) and initialization (NeurIPS 2024; line 479 in our paper) on zero-initialized LoRA. \nHowever, a fundamental question remains unaddressed: Why is *ZI* necessary? In this paper, we extend these general derivations to examine the potential advantages of *NZI*.\n\nNotably, **the contribution and innovation of this paper lie not in proposing a new derivation method, but in the following three aspects**:\n1. **Motivation for *NZI*.** We provided a comprehensive solution set that ensures LoRA's stable and efficient learning. Building on this, we observe that the solution set includes both zero and *NZI*, prompting us to investigate the role of *NZI* further. This constitutes a key distinction between our work and previous studies, where the potential of pure *NZI* has often been overlooked. Our research bridges this gap and provides preliminary evidence supporting the feasibility of *NZI*.\n2. **A new metric for LoRA's fine-tuning dynamics, \"robustness\", is proposed.** We compare the fine-tuning dynamics of zero and *NZI*s and define robustness in terms of the sensitivity of these dynamics to the learning rate. The central argument of this paper is that *NZI* exhibits superior robustness compared to *ZI*. We believe that this metric is crucial for LoRA fine-tuning dynamics, offering a significant extension and enhancement to existing theoretical derivations and analyses.\n3. **Breaking inherent cognitions.** Our analysis and experiments further show that fine-tuning does not need to strictly start from a pre-trained model. This challenges the default practice in previous studies, such as LoRA, LoRA-GA, and PiSSA.\n\nThese contributions provide valuable guidance for understanding LoRA initialization and fine-tuning LLMs. They represent notable advancements and are substantial enough to be considered as independent work.", "claims_and_evidence": "This paper looks good and provides some findings beyond zero initialization. I understand the motivation of using non-zero initialization but the current claim is weak. Previous non-zero initialization work, e.g., LoRA-GA, has better motivation. For instance, LoRA-GA is to ensure that LoRA gradient updates can match full gradient updates as much as possible, which is also the spirit of using LoRA.\n\nUnder theory-guided instructions, this paper gives some non-zero initialization strategies that achieves better performance than LoRA, e.g., 2x speedup than LoRA. However, the comparison with previous work is limited: only LoRA is compared. I understand that the key idea of this work is to speedup and obtain other benefits (e.g., LoRA). Nevertheless, the experimental comparison is not sufficient.\n\nAnother significant issue is that, the derivation heavily follows with previous work, e.g., Hayou et al. When I read it at first, it can be a good journal extension but I'm not sure that it can be regarded as an independent work.", "methods_and_evaluation_criteria": "The evaluation makes sense but the comparison is limited.", "theoretical_claims": "The theoretical claim is ok in terms of the stability and efficiency.", "experimental_designs_or_analyses": "The experiments are not sufficient. Only LoRA is compared.", "relation_to_broader_scientific_literature": "this topic and the obtained findings are interesting to the machine learning community.", "essential_references_not_discussed": "The essential references are sufficient but it's true that not main LoRA-based algorithms are discussed.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received four effective reviews, and all of them are positive. Overall, the paper is of good quality and should be accepted.
Decision: Accept (poster)