LoRA vs Full Fine-tuning: An Illusion of Equivalence
Authors: Reece S Shuttleworth
Jacob Andreas
Antonio Torralba
Pratyusha Sharma
Venue: NeurIPS 2025 poster ()
Keywords: Parameter Efficient Fine-tuning (PEFT)
Low Rank Adaptation (LoRA)
LLMs
Transformers
Primary area: deep_learning

Abstract: Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, \emph{are their learned solutions really equivalent?} We study how LoRA and full-finetuning change pre-trained models by analyzing the model's weight matrices through the lens of their spectral properties. We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure: weight matrices trained with LoRA have new, high-ranking singular vectors, which we call \emph{intruder dimensions}, while those trained with full fine-tuning do not. Further, we extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension -- by causally intervening on the intruder dimensions by changing their associated singular values post-fine-tuning, we show that they cause forgetting. Moreover, scaling them down significantly improves modeling of the pre-training distribution with a minimal drop in downstream task performance. Given this, we should expect accumulating intruder dimensions to be harmful and lead to more forgetting. This will be amplified during continual learning because of sequentially fine-tuning, and we show that LoRA models do accumulate intruder dimensions here tend to perform worse in this setting, emphasizing the practicality of our findings.

LoRA vs Full Fine-tuning: An Illusion of Equivalence
Reece Shuttleworth Jacob Andreas
Antonio Torralba
Pratyusha Sharma
MIT CSAIL frshuttle, jda, torralba, pratyusha/@mit... edu
Abstract
Fine-tuning is crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with extreme reduction in trainable parameters. But, are their learned solutions really equivalent? We study how LoRA and full-finetuning change pre-trained models by analyzing the model's s weight matrices through the lens of their spectral properties. We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure: weight matrices trained with LoRA have new, high-ranking singular vectors, which call intruder dimensions, while those trained with full fine-tuning do not. Further, we extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized the intruder dimension by causally intervening on the intruder dimensions by changing their associated singular values post-fine-tuning. scaling them down significantly improves modeling of the pre- pre-training distribution with a minimal drop in downstream task performance. Given this, we should expect accumulating intruder dimensions to be harmful and lead to more forgetting. This will amplified during continual learning because of sequentially fine-tuning, and we show that LoRA models do accumulate intruder dimensions here tend to perform worse in this setting, emphasizing the practicality of our findings.
show that they cause forgetting. Moreover,
Introduction
Full Fine-tuning
LoRA, 6-6
Adapting large, pre-trained models to down- stream tasks via fine-tuning computation- and data-efficient way create domain-specifie models for a variety of tasks. The simplest ap- proach is to fine-tune all parameters of the pre- trained model on downstream task data [Devlin al. 2019 Ouyang et al. 2022] However, pre-trained models grow larger, full fine-tuning becomes increasingly challenging and expen- sive. Recently, parameter-efficient fine-tuning (PEFT) methods, especially low-rank adaptation (LoRA; Hu et al.] (021). have been shown to enable fine-tuning with only fraction of the trainable parameters. While LoRA can match full fine-tuning performance, are the solu- tions learned by the two methods similar?
mn-
101
200
300 c00
singular voctors
Wonned
singular vectors Wonned
Figure 1: LoRA and full fine-tuning update the pa- rameter space differently. Similarity matricies of pre- and post-fine-tuning singular vectors for LLaMA2-7B that characterize the spectral differences introduced dur- ing fine-tuning. Full fine-tuning retains most of the pre-training structure, while LoRA a diagonal shift, Color shows cosine similarity.
While full fine-tuning treats every parameter trainable, LoRA treats the learned update to a weight matrix as the product of two low-rank matrices Hu al. 2021 While this parameterization is empirically effective, principled explanation of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LoRA Learns Intruders
Disutations are singslar vectees dissimilar to those
b)
Companing Full vy LoKA Matni
the pre tained weight matra and are leamned during fine tuning.
Closss Sirggaa Vecker in We
a)
Coinne Similariyy io
Wo
+
B
SVD
LoRA
SYay FLll foo rodkl
0.
200
400
500
W
Wo
SYD
Singuir Vectors Wouser Ondered)
Normal Singular Vector
pre-trained weights
250
500
750
Intruder Dimensien
Wo
+
AW
SYD
Connne
full fine- fine-tuning
U
Σ
V
250
Singular Vectors in Wy (Ohdered)
Figure 2: Characterizing structural differences between solutions Iearnt by LoRA & Full Fine-tuning, We measure the changes to the SVD of the pre-trained weights made during fine-tuning. observe intruder dimensions introduced by LoRA in top ranking singular vectors but not by full fine-tuning. b) Comparing a matrix fine-tuned with full fine-tuning or LoRA. c) The intruder dimension shows near-zero absolute cosine similarity with all preitranned singular vectors, in contrast to other singular vectors of the finetuned matrix.
the mechanism by which it matches the full fine-tuning performance has remained elusive. One explanation is offered by the intrinsic dimension hypothesis (Li et al. 2018 Aghajanyan et al. [221] which posits that the update learned via fine-tuning has low intrinsic rank, suggesting that LoRA might recover an approximately equivalent solution to full fine-tuning. However, prior work has observed differences in the ability of LoRA and full fine-tuning to independently change the angle and magnitude with which a neuron transforms its input (Liu et al.) 2024). Moreover, other work has also observed that LoRA has difficulty matching the performance of full fine-tuning on difficult tasks like code generation [Biderman et al. 2022|| Zhuo et al. 2024] and long-form text generation [Ivison al. 2023). Therefore, it unclear if these findings indicate a limit LoRA's ability fit to specific downstream task, or if these methods learn inherently different solutions.
In this paper, show that full fnn-ttunig and LoRA learn different solutions with characteristic differences in their spectral properties (as shown in Fig. I for LLaMA2-7B [Touvron et al.) (023b]) and that these spectral differences are causally related to different model behaviors. We observe:
LoRA and full fine-tuning produce structurally different parameter updates, characterized by the existence of intruder dimensions in weight matrices tuned by LoRA. Intruder dimensions are singular vectors with large associated singular values that are very dissimilar to the singular vectors in the pre-trained weight matrix. In contrast, fully fine-tuned models remain spectrally similar to the pre-trained model and do not contain intruder dimensions.
2. LoRA forgets less than full fine-tuning...ut not always. We extend the findings of Biderman al. [024] that LoRA forgets less the case even when there equal fine-tuning performance between LoRA and full fine-tuning, showing that difference in fit is not simply the cause of this finding but rather is inherent to these methods. However, this not always the case: despite nearly identical fine-tuning task accuracies, we show that different selections of I LoRA alpha and learning rate lead to starkly different generalization behaviors, even leading to LoRA forgetting more than full fine-tuning. We also find that models with the best generalization for each of these hyperparameter settings have the fewest intruder dimensions.
3. Intruder dimensions cause forgetting of the pre-training distribution. Scaling down the associated singular values of high-ranking intruder dimensions leads large drop in loss on the pre- pre-training distribution (forgetting) but only minimal drop in test performance. The drop in forgetting we observe when scaling down singular vectors unique to intruder dimensions and indicates that they interfere with the pre-trained language modeling ability of these models. Given this finding, we should expect accumulating intruder dimensions to be harmful and lead to more forgetting. To amplify this accumulation and examine its effect, we fine-tune in continual learning

---

setting (sequentially fine-tuning on many tasks) and show that LoRA models do indeed tend to forget more on previously learned tasks in this setting, providing additional support for our findings.
2 Background & Related Work
Methods for fine-tuning. Pre-trained language models offer a foundation for downstream appli- cations, eliminating the need to train from scratch [Quyang et al. 2022 Devlin et al. 2019]. Full fine-tuning, in which every parameter of a pre-trained model is updated. commonly used [Devlin eet al., 2019 Liu et al. 2019]. Low Rank Adaptation (LoRA; Hu et al. (02I), which represents the update to the weights as a product of two low-rank matrices, reduces computation and memory requirements relative to full fine-tuning. Past work has shown that LoRA matches full fine-tuning performance for tasks like sequence classification (Hu al. [202T) instruction tuning (Dettmers) et al., 2023, Ghosh et al.] 2024], and chat [Dettmers et al., 2023]. Other work has shown a gap in performance harder tasks like code generation [Biderman et al.) 2024 Zhuo et al.. 2024]. We
focus our investigation on both cases to ensure our findings generalize to all use cases.
LoRA, formally. Given a pre-trained weight matrix Wo Rn*x,, full fine-tuning treats the learned matrix update as AW E Rm*" Instead, LoRA decomposes AW into a product of two matrices such that AW BA, where B Rmxr, A Brx", and where the rank is generally < min(m, n). During prediction,
Y = WinnedX (Wo + -
B is initialized to zero, and A sampled from an isotropic Gaussian. All parameters in B and are see that while full fine-tuning has mn trainable parameters per weight
trained. From this matrix, LoRA only has mr + rn. See Appendix|F]es derivation of LoRA adapter gradients.
LoRA Variants. Many variations of LoRA exist. Methods improve LoRA's performance memory- efficiency by initializing with the principal [Meng et al. 20244 or minor [Wang et al. [024] compo-
nents of the underlying weight matrix, training with quantization [Dettmers et al.) [023], adaptively
allocating different ranks (Zhang et al., [2023], or sequentially training multiple LoRAs Xia et al.
2024). Other methods propose similar but alternative architectures [Liu et al.) 2024| Kopiczko et al.
2024. Koohpayegani et al.] 2024]. Other work has also proposed low rank manipulations to the
activations instead of the weights Wu et al. 2024). Although the primary focus of our study is the original LoRA setup [Hu et al. 2021], we also study a few LoRA variants (Appendix (P). While we leave a rigorous analysis of all possible variants to future work, our preliminary experiments show that our findings generalize to several variants. Additionally, we also demonstrate the robustness of our findings across range of LoRA hyperparameter settings (Appendices B.4 N O).
Analysis of Solutions. The intrinsic dimension measure [Li et al. 2018 was used by Aghajanyan et al. [202I] argue that the fine-tuning update for a pre-trained LLM has low intrinsic rank, explaining why only a small number of trainable parameters are necessary to reach 90% of full fine-tuning performance. This finding motivated Hu et al. 202T] to hypothesize that LoRA works because solutions of low intrinsic rank exist. But to our knowledge, no past work has compared the rank (or other properties of weight matrices) between LoRA and full-fine tuning on tasks where they are matched in performance. While Liu et al. [2024) showed that LoRA has difficulty changing directional and magnitude components of neuron independently. it is unclear if this difference is due to an inability of LoRA to fit as well as full fine-tuning to the adaptation task.
Relation to Biderman et al. [024].. Recent work comparing LoRA to full fine-tuning has found that LoRA forgets less when fine-tuned on math and code [Biderman et al.) 2024] and more closely resembles the pre-trained model [Ghosh et al. 2024 We extend the findings of Biderman et al. [2024] to the case when there is equal fine-tuning performance between LoRA and full fine-tuning, showing that a difference in fit to the fine-tuning task is not simply the cause of this finding but rather is inherent to these methods.
Singular Value Decompasition. The SVD decomposes a matrix M E R such that M UΣVT. where E Rm and V E Raxx have orthonormal columns representing the singular vectors of M and Σ E mm is diagonal matrix containing the singular values of M. U and 1/2 represent rotations that matrix M performs, while Σ represents scaling along those axes. Singular vectors, ordered by singular values, reveal matrix's 's most important axes of transformation.
3

---

Structural Differences
Inspired by Sharrma
2024] finding that
the singular value decomposition (SVD, Klema and Laub 1980) can be used to selectively prune singular vectors to improve model performance, this paper adopts the SVD of neural network pa-
Full Finc-tuning
Singular Vecters in Wo
0.6
rameters as a lens for understanding the changes that fine-tuning makes to pre-trained weights. Understanding how these dimensions change can give us insight into how a particular fine- tuning method changes the pre-trained model. In particular, we study how well singular vec- tors in weight matrices fine-tuneds with LoRA or full fine-tuning map to singular vectors in the pre-trained weights (using cosine similarity).
002
Singular Vectors in Wouned
Singular Voctors
Wonedd
Figure 3: LoRA and full fine-tuning learn distinct structural solutions. LoRA introduces intruder dimen- sions (represented by outlined columns).
Visually, we observe Fig. bb) that LoRA and full fine-tuning's 's singular vectors have very different similarities to the pre-trained singular vectors: singular vectors of models fine-tuned with LoRA appear to have, on average, much lower cosine similarity to pre-trained singular vectors in comparison to full fine-tuning. Interestingly, in LoRA fine-tuned models, also observe the presence of high ranking singular vectors with very low cosine similarity to any pre-trained singular vector In Fig. Cc), we show the difference between these vectors with low cosine similarity to the pre-trained singular vectors and normal singular vectors from the fine-tuned weights. This "new" dimension can be seen in Fig. ((b) the lone red dot in the bottom left corner. We name these "new" dimensions intruder dimensions, which we define formally as follows:
Definition 3.1. singular vector Y5 from the fine-tuned weight matrix Wtaned is an intruder dimension if and only if mcxx(cos(J;; x;)) e, where is similarity threshold and x; are the singular vectors of Wo-
Examples of intruder dimensions may be seen in Fig. B Here, we plot the similarity matrix between the top 10 singular vectors (ranked by singular value) in the pre-trained and fine-tuned matrices. While full fine-tuning appears have a clear one-to-one mapping, LoRA appears to have its mapping shifted by "blank" columns (outlined in magenta): these are intruder dimen-
Algorithm 1 Finding intruder dimensions.
Require: Pre-trained weights Wo. fine-tuned
weights W,, cosine similarity threshold of fine-tuned singular vectors examine k.
1: [Vo, Eo, VJ < SVD(W))
2: [U,. E,, V + SVD (Fnned))))
sions, with low cosine similarity to every pre- trained singular vector. A zoomed out version of this plot can be seen in Fig. in which see an off diagonal shift due to intruder dimensions.
3: n_ intruders
O
4: n + of pre-trained singular vectors
5: for <- 1 to is do
6: if S n cos( Vol:), Utlil) < then
It is important to note that in the case of full fine-tuning, the singular vectors that map to pre-trained singular vector with high cosine sim- ilarity also have similar singular values. From these initial measurements, it appears that LoRA and full fine-tuning have structural differences in the changes they make to the pre-trained weights: while full fine-tuning appears make small changes the existing singular vectors and singular values, LoRA introduces new singular vectors that have a large contribution to the norm of the updated parameter matrix.
7:
n n_intruders + n_intruders
8: end if
9:
end for
10: return intruders
Our Models. We study LLaMA2-7B 'Tovrro et al. 2023b and RoBERTa-base [Lu ct al. 2019 RoBERTa-baal is pre-trained encoder-only language model and we fine-tune it on six different sequence classification tasks. See Appendix C.3 for fine-tuning details. LLaMA2-7B is a pre-trained decoder-only language model, and we study it when fine-tuned either code math. These
checkpoints are provided by Biderman et al. [024]. We also study LLaMA-7B (Touvron et al.
Recall that in high dimensions. vector can have low cosine similarity to a set of orthogonal vectors that span space; Appendix for discussion.

---

LLaMA-7b Alpaca
LLMMA2-7b MetaMathOA
LLaMA2-7b on Mugicoder
intruders
300
LeftA---o)
1-15
400
1000
200
500
0.2
04
0.5
0.
0.6
E ((hreshold)
E (threshold)
((hreshold)
(a) LLaMA-7B fine-tuned
(b) LLaMA2-7B fine-tuned on (c) LLaMA2-7B fine-tuned on
Alpaca.
MetaMathQQ.
Magicoder-Ewoo Evol Instruct. -voltnrtuutt..
1-6
1-64
full
num. intruders
MNLI
QQP
SST2
SIQA
WinoCirande
FEVER
400-
200
100
200
50
0.50 (threshold)
0,50 E (threshold)
(threshold)
E (threshold)
E (threshold)
(threshold)
(d) Number of intruder dimensions in RoBERTa models fine-tuned on different tasks.
Figure 4: LoRA intruder dimensions, whereas full fine-tuning not. Here, we set k 10 and measure the impact of e on the number of intruder dimensions measured. LoRA introduces many intruder dimensions in the top 10 ranked singular vectors, while full fine-tuning does not. Numbers are reported are the sums across the entire model.
Max Casine Similarity to Singular Vectors in Wn
Singular Valioes in Finetuned Matrix S04
Test Accuracy
Max Cosine Similarty
1.00
Singular Value
Shel
0.54
Azcurscy
0.50
5:0.5
0.90
0.25
50ec
Micc Stre
0.00
Stnce
10000
20non
30003
10000
20000
30000
10600
20000
30000
Training Step
Training
Training Step
Figure 5: Evolution of an intruder dimension across training steps. (Left) Intruder dimensions, and their rank, in LoRA fine-tuned weight matrix during fine tuning. (Middle) Their associated singular values, which shows that the singular value associated with the intruder dimension increases. (Right) Test accuracy across training steps.
2023a models fine-tuned on instruction following. See Appendix D for more details about these models. Importantly, these L.aMAA models span math, code, and chat, which are considerably harder than sequence classification tasks. This ensures wide coverage of LoRA use cases.
Our Method. To calculate the number of intruder dimensions in a specific weight matrix, we use Algorithm. [ In it, we first compute the SVD of both the pre-trained and resulting LoRA and full fine-tuned weights. Then, for each of the top highest-ranking singular vectors, we measure its maximum cosine similarity with all of the pre-trained singular vectors. If this maximum cosine similarity is less than some threshold e, we classify this singular vector as an intruder dimension. Note that both k, the number of fine-tuned singular vectors to examine, and the cosine similarity threshold, are hyperparameters; we verify the robustness of our findings for a wide range e and l values in Fig. 4 and Fig. 15 respectively. To determine the number of intruder dimensions in specific model, we run this algorithm for each weight matrix in the model and sum the total.
LoRA fine-tuned models contain high-ranking intruder dimensions while fully fine-tuned models do not. To characterize the differences fine-tuning methods, we first evaluate the differences in the total number of intruder dimensions in the top 10 highest-ranking singular vectors (k: 10). We repeat this procedure for range of c values, our cosine similarity threshold. The results are presented in Fig. For LLLM2---B, we find that models trained with LoRA contain intruder dimensions for ranks least as high as r S 256. For RoBERTa, we consistently observe intruder dimensions for rank < 16, even for low values of Interestingly, we observe that fully fine-tuned models, for all model sizes, almost never contain intruder dimensions their top 10 singular vectors, even for epsilon values of about 0,6 to 0.9. This means that full fine-tuning makes smaller changes to the same
5

---

set of high contribution pre-trained singular vectors, rather than introducing new singular vectors like LoRA. Importantly, the number of intruder dimensions appears drop as rank increases past a certain threshold, suggesting that the low-rank nature, as well as the update rule of LoRA, induces them to occur. This is underscored by the r = 2048 case of L.LAMA-7BB fine-tuned on math (Fig 4b). which does not have intruder dimensions and instead has a very similar curve to full fine-tuning. As rank increases past a threshold and LoRA begins to resemble a high rank update, intruder dimensions begin to disappear.
LoRA variants have intruder dimensions. We examine 4 other LoRA variants to ensure that our findings do not only apply to vanilla LoRA. We examine AdaLoRA (Zhang et al. 2023). LoRA+
(Hayou et al. 2024], PiSSA [Meng et al.) [2024] and VeRA [Kopiczko et al. 2024]. In all of these cases, we find intruder dimensions with similar characteristics to vanilla LoRA (see Fig. 23 This
shows our findings hold to other variants. For more discussion about these methods, see Appendix P
Intruder dimensions are distributed across both high and low singular values. We examine the extent to which intruder dimensions exist throughout the entire weight matrix and how they are distributed. To do this, we hold fixed and measure the number of intruder dimensions while varying the proportion of the fine-tuned singular vectors that we examine (Appendix D Fig. [5]. Here, can see that LoRA consistently has more intruder dimensions than full fune-tuning, regardless of what fraction of the singular values we examine. See Appendix for for more discussion.
Intruder dimensions increase in magnitude and change in direction as fine-tuning progresses. To further understand how particular intruder dimension is introduced during fine-tuning with LoRA, we measure the maximum cosine similarity between the top individual fine-tuned singular vectors and all the pre-trate me-ttrined singular vectors across many intermediate steps in the fine-tuning process, as seen in Fig. 5 (leff). In parallel, we track changes in their associated singular values as seen in Fig. 5 (middle). As is evident from the graphs, intruder dimensions appear to gradually increase their "rank" (left) as their singular value is increased (middle) while simultaneously changing in direction too as training progresses.
Additional empirical observations. 1. We find that the random seed used by LoRA to initialize its adapters plays role in the resulting structure (see Appendix O 2. We observe that the total number of intruder dimensions increases linearly with respect to the size of the fine-tuning dataset up to certain point before saturating (Appendix K). 3. We study the effective rank of these fine-tuning updates (Appendix [.. However, we find that this measure does not suffice to explain the behavioral differences observe in LoRA and full fine-tuning. Also, it is important to note that even if it had, its global nature would prevent the precise examinations we conduct in future sections, like in Fig. g
Experimental and theoretical justification for why intruder dimensions occur. It is important to note that intruder dimensions are an empirical observation of LoRA. We find that a variety of factors play a role in the introduction of intruder dimensions. In the next section, we present results that suggest that learning rate and LoRA's a contribute to intruder dimensions. In the appendix, we present findings that suggest that tuning the B matrix only leads to fewer intruder dimensions (B.3), and demonstrate how the addition of orthogonal vectors to the pre-trained weight matrix models the introduction of intruder dimensions well (B.2
4 Model Differences: Forgetting and Out-of-Distribution Generalization
LoRA forgets less. We measure the change in out of distribution performance (forgetting) induced by fine-tuning. For LLaMAA-7B, we follow Biderman al. [004 and measure forgetting as the average score on Hellaswag [Zelers et al. 2019). WinoGrannde (aakauuci et al. 2021], Arc-Challenge (Clark] al. 2018). For RoBERTa-base, we measure its "pseudo-loss", which is analogous to language modelling loss for encoder-only models, as described by /alazar et al. (2020) on sample of its pre-training dataset (as described by Liu al. (2019). Going forward, we refer to these values as "forgetting" and report them in Fig. 6 We observe that across all tasks, full fine-tuning forgets more of its pre-training language modeling ability in comparison to LoRA. Importantly, all our GoBERTa-base models fine-tune to equivalent accuracy on the downstream task (Table 2 This extends the finding that LoRA forgets less [Biderman et al., 2024 to the case where LoRA and full fine-tuning have equal fit, showing that LoRA forgetting less is not simply a function of it underfitting the fine-tuning task in comparison to full fine-tuning (like in Biderman et al. 2024) ) but rather a characteristic of LoRA itself.

---

0.55
Pre-training Pseudo Loss
Forgeting (Accuracy)
2.50
225
6.4
2.00
MeaMlat
OpatWicsMath
1.75
Magisider
1.50
Staronder
Pre- re-iTmined
LoRA
LoRA
LoRA
Full Fine-Tuning
Pre-Trained
LaRA
LoRA
LoRA
LoRA
Full Fine-Tuning
(a) L.MM22-7B.
(b) RoBeRRa-hase.
Figure 6: LoRA forgets less, with same fit fine-tuning task. For LLaMA2-7B, forgetting is measured on unrelated tasks, as described in Biderman et al. [2024). For RoBERTa, Pseudo loss on sample its pre treirrainin distribution measured as described by Salazar et al. 2020). In both, LoRA forgets less than full fine-tume mun-tuning.
lora-3e-4
lora-2e-4
lora-le-4
full-le-5
Number of Intruders
Test Accuracy
LoRA Pre-training Pscudo Loss
0.88
200
Intrudess
Accuracy
0.87
Pseudo Loss
3.5
100
3.0
0.85
0.84
Number of Epochs
Number of Epochs
Number of Epochs
Figure 7: As training progresses, models with growing amount of intruder dimensions continue to forget more, despite non-increasing test performance. We also measure a strong correlation(p 0.971. p-value < 0.001) between number of intruder dimensions and pre-training pseudo loss. Bigger learning rates lead to more intruder dimensions and forgetting.
LoRA a impacts generalization and intruder dimensions. For our experiments, we use the commonly used = 2r [Biderman et al. 2024] well as = 8 HHu et al. 202T]. For both settings of a, models obtain equivalent performance on the target task (Tables & 2 However, when a
all ranks of LoRA- even very large ones exhibit intruder dimensions (Fig. 19a). have much smaller effective rank than when a = 2r (Appendix [ and have much worse generalization (more forgetting, Fig. 20). Models trained with = 2r have fewer intruder dimensions and generalize better. This provides additional evidence highlighting the importance of using a 2r Kalajdzievski 2023 Biderman et al. 2024), particularly for higher ranks of LoRA.
An increase in intruder dimensions leads to an increase in forgetting. We do learning rate sweep for RoBERTa-base with LoRA r = 8 on MNLI to observe its impact. Across epochs, we measure the number of intruder dimensions, test accuracy, and forgetting (pre-training loss). We report the results of these models and our baseline full fine-tuning model in Fig. [ We observe that across and within training runs, as the number of intruder dimensions increase, forgetting (meaning worse generalization) also increases. Test accuracy has no such relation. Separately, we see that for large learning rates with many intruder dimensions. LoRA models forgel more than full fine-tuning. This shows that while LoRA in general does forget less than full fine-tuning, it is not guarantee.
Intruder dimensions strongly correlate with forgetting. When we measure the Spearman correla- tion between the number of intruder dimensions with forgetting in Fig. 7 we find an extremely strong fit(p 0.971, p-value < 0.001). When measuring the same for our LLaMA2-7B models across training epochs (Fig. TTp. we still find strong and still statistically significant relationship (p = 0.59, p-value 0.0006). In contrast, when measuring the correlation between intruder dimensions and test accuracy, we find no statistically significant relationship: for RoBERTa, we measure p = -0.3381 & p-value = 0.218. For LLaMA2-7B, we measure p = -0.3178 p-value- = 0.0869. See Appendix G for more information. These results suggest that intruder dimensions are clearly linked with forgetting are not necessary for performance. We examine this claim and whether this relationship causal in the next section.
7

---

100
25
2.25
2.4
2.5
80
90
20
2.20
40
80
15
2.15
2.2
60
2.10
70
0.0
0,5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
(scaling factor)
À (scaling factor)
(scaling factor)
(scaling factor)
(a) LLLMMA-7B code with r=16.
(b) LLaMA2-7B
(c) RoBERTa-hase on MNLI with r=8.
(d) RoBERTa-base
math with r=16.
QQP with r=8.
Figure 8: Scaling down intruder dimensions in fine-tuned models reduces forgetting but not performance. We scale the top intruder dimension in each matrix such that W Wo + AW +A D)uaoee Lines represent forgetting (red) and learning (green). Dotted lines represent pre-trained baselines. Axis Labels: Green; Test Accuracy (%). Red: Pre-training Loss (Forgetting).
Intruder Dimensions Cause Forgetting
Previously, we observed that the number of intruder dimensions correlates strongly with forgetting. Do intruder dimensions cause this forgetting?
Scaling the magnitude of intruder dimensions. To test if intruder dimensions cause increased forgetting, we must intervene on intruder dimensions and see the impact. We do this by finding the highest ranked (by singular value) intruder dimension in each weight matrix and scale its contribution such that the new weight matrix W Wo + AW + (A - 1)u,a,y vf where is the index of the top intruder dimension (A = 0 is removal and À = 1 is no change). We sweep X's between 0 and 1, scale the intruder dimensions, and measure test accuracy and pre-training loss. For comparison baseline, we select the neighbor of the intruder dimension to separately scale. See Fig. g for results (and Figs. 12 13 for full results.)
Scaling down intruder dimensions reduces forgetting. In Fig. g we show that when we scale down the top intruder dimension of each weight matrix, we measure a significant reduction forgetting (pre-training loss) while incurring minimal drop in test accuracy. For all examples in Fig. 8 we observe that when using À 0.7 0.9, there almost no impact fine-tuning performance, while there is large percentage drop in forgetting (See Tables 3 5
In one example for LLaMA2-7B
fine-tuned on MetaMath with LoRA
256, we observe that scaling the top intruder dimension in each matrix with À = 0.3 leads to a 0. 1.1% % drop in test accuracy and 33.3% drop in the forgetting induced by fine-tuning. In another for RoBERTa-base fine-tuned on QQP, using X = 0.7 leads to equivalent in test accuracy and a 33.2% reduction in the forgetting induced by fine-tuning. In certain scenarios, we even see test accuracy improve along with drop forgetting. we instead increase their contribution (A 1), observe more forgetting. Across the board, scaling down intruder dimensions seems to have little impact on test accuracy but a major impact on forgetting. This pattern is exclusive to intruder dimensions (Fig. 12 if we instead intervene pre-trained singular vectors that have a similar singular value to the intruder dimension (ensuring similar contributions to matrix) and scale their magnitude down, the forgetting observed in LoRA is caused by intruder dimensions interfering with the pre-trained language modeling capabilities. Moreover, the scale (singular value) of these intruder dimensions is not essential for the fine-tuning task performance. See Appendix H for further discussion.
that forgetting goes up. These results indicate that
Continual Learning Setup. To examine practical example of how accumulating intruder dimen- sions, which cause forgetting, may impact performance, we study continual learning, since requires learning and remembering across range of tasks. To do this, train RoBERTa sequentially multiple tasks and measure performance as new tasks are learned. We use the same training recipe and datasets as before but now sequentially in the following dataset order: MNLI, QQP, SST-2, SIQA, WinoGrande, FEVER. After training on certain dataset in the sequence, we merge the LoRA weights into the model and reinitialize the LoRA adapter before training the next task. After training on a specific task, we test on all tasks by, for each task, separately retraining its classification head before testing its test set. Results shown in Fig. [a
Accumulating intruder dimensions hurts LoRA models during continual learning. In Fig. 9a initially both LoRA and full fine-tuning train equal performance (MNLI), which is consistent with

---

Full
LORA,
after MNLI (#1)
after QQP [+2)
after FEVER (#6)
LoRA.F1
14
Aecuracy on MNLI
Singular Vectors In
0.8
Accuracy
0.7-
0.6
Singular Vectors
Singular Vectors
Singular Vectors
Accuracy on QQP
(b) Similarity matrices for LoRA during continual learning.
0.90
after MNLI (#1)
after OOP (#2)
ofter FEVER (+6)
Accuracy
0.85
Singul al Vectors
In Wo
RoBENTE
MNLI
QOP
SS72
SIGA
Wine-
Grande
FEVER
02
base
02
Singular Vectors
Singular Vectors
Singular Vectors Wauses
Dataset Fine-tuned on
Winvo
Wasced
(a) Continual learning results.
(c) Similarity matrices for full fine-tuning during continual learning.
Figure 9: Full fine-tuning is better than LoRA at continual learning because of accumulating intruder dinensions. When sequentially training six tasks, full fine-tuning retains performance better than LoRA. in Fig. Pal horizontal dotted line indicates baseline pre-trained performance. Vertical solid line indicates when a specific dataset is fine-tuned on. Gray region represents performance before the model has been trained on that task. See Appendix[M) M. Fig. 17 for more. In Figs. 9b we see that LoRA accumulates intruder dimensions across tasks and contributes its degrading performance, whereas full fine-tuning does not.
our previous observations. However, we observe that all ranks of LoRA degrade much more rapidly than full fine-tuning. Low ranks of LoRA, which have the most intruder dimensions, degrade the LoRA appeared to forget less to the
most. We attribute this divergence from earlier accumulation of intruder dimensions during continual learning. which drive forgetting. To show this, we visualize how intruders are added across tasks in Fig. 9b Here, we see that each task adds its own intruder dimensions leading to a large amount intruder dimensions upon the completion of six task continual learning experiment. In contrast, in Fig. 9c
we see that full fine-tuning retains the pre-trained structure well justifying why full fine-tuning forgets less during continual learning.
Implications and prescriptions in fine-tuning, These findings suggest several implications and prescriptions during LoRA fine-tuning. We have shown that intruder dimensions drive forgetting, and therefore should be avoided when possible. Interestingly, this presents data free model evaluation method to examine which model most overfit to the fine-tuning task (forgotten the most): given two equally performing models on downstream test sets, you should select the one with fewer intruder dimensions. Intruder dimensions appear to be necessary part of fine-tuning, but they can be mitigated Further, these results show the danger of using LoRA during continual learning and justifies using many different adapters without combining them, like advocated in [Sheng et al. 2024
6
Conclusion
This paper describes the finding that LoRA and full fine-tuning update different parts of the parameter space resulting in distinct spectral properties: LoRA often introduces intruder himensions--ipe ranking singular vectors dissimilar to those in pre-trained weights. These structural differences persist across a series of ablations. Next, we find that models with fewer intruder dimensions exhibit better out-of -fiisiriitittioionn generalization and forget less of the pretraining distribution. Last, we show that intruder dimensions cause increased forgetting: We show that reducing the magnitude of high ranking intruder dimensions leads to minimal changes test performance but large drop in pretraining loss. We show that this is particularly relevant during continual training: even though LoRA forgets less than full fine-tuning after training on one task, sequentially training leads to an accumulation of intruder dimensions that causes more forgetting than full-finetun full-finctunin..

---

Acknowledgements
We would like to thank Jacob Portes and Dan Biderman for corresponding with us and releasing their LLaMA-2 7B checkpoints for us to use. This enabled us to study more comprehensive range of models. We would also like to thank Leshem Chosen, Lucas Hennigen, Han Guo, Vighnesh Subramaniam, Valerio Pepe, and the entire Language & Intelligence lab for their helpful feedback on this work. This research was supported in part by the National Science Foundation under grant IIS-2238240.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-'Tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, August 2021. URL httph:t://// htattps://aclaanhoolloe
org 2021 acl acl-long.5550 558
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. https://githh: com/
bigcode-prrject//iiccoee www.:/coddeeelllaaiio..... wwwelluaiionhhaaeeeee 2022.
Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jen- nings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research, 2024. URL httpsttp::// //tp:i/////xviv////a org/ abs abs 2eep/2405.0000 09673
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. httpss/tts:////ttt httpp:ss///avvv..eeeaeeassassaa/ org/ wwwp://////../.......0.
Peter Clark. Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. 2018. URL https:://arrtit org/ abs babs//883.0000
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URLhttss://arra org/ abs/22100. 14168.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems, 2023.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguisties, June 2019. URL httpp:://acaanthhllogg. .oogorgg//S -14133339111
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL httts::s:... http::::///enoo..oom htwpa/dd..rrgrreerrr///////0./0.
10

---

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S. Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. A Closer Look at the Limitations of Instruction Tuning. In Proceedings of the 41st International Conference on Machine Learning. International Conference on Machine Learning, 2024. htll.htttps://rci..orrg abs 220. 05119
Aaron Gokaslan and Vanya Cohen. OpenWebText Corpus. http://Skylion007..gg
io/
OpenieebexttCoppus 2019.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A Generic News Crawler and Extractor. In Proceedings the 15th International Symposium of Information Science, pages 218-223, March 2017. doi: 10.5281/zenode 4120316.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-Rank Adapters Are Secretly Gradient Com- pressors. In Proceedings of the st International Conference on Machine Learning. International Conference on Machine Learning, 2024. URL URI.htpss:/ arxiv org/ absaaaas/000
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient Low Rank Adaptation of Large Models, 2024. URL httttt////ariii wwwpexiv..orr/aeee abs 20abb///402.2355
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations, 2021.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels Changing Climate: Enhancing LM Adaptation with Tulu 2, 2023. URL
org/
abs 2211.
Damjan Kalajdzievski. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, 2023, URL https://arxiv. wtt.://////////////ee abt// 03732
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxiv.org org abtp////////1111/11 191.//112..9988
V. Klema and A. Laub. The singular value decomposition: Its computation and some applications.
IEEE Transactions on Automatic Control, 25(2):164 176, 1980. doi: 10. 10.1109/TAC.19800 102314,
Soroush Abbasi Koohpayegani, KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. NOLA: Compressing LoRA using Linear Combination of Random Basis. International Conference on Learning Representations, 2024. Uttps:ts::///rss
org/ 2310 02556
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. VeRA: baste Random Matrix Adaptation. International Conference on Learning Representations, 2024. URL https://arxx: //arxivv
org/abs/ atg/abs/2310..o 2310. 11454
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the Intrinsic Dimension of Objective Landscapes. International Conference on Learning Representations, 2018. URL
org/ abs/1804. 08838
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023. URL https: // //trxxixvvvv///////a//ea//aaaav/ org/ abs htw/ 06161

---

Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Cheng, and Min-Hung Chen. DoRA: Deeopp-Deeompseedd Low-Rank Adaptation. Proceedings of the s1 International Conference Machine Learning. International Conference on Machine Learning, 2024. URL JtRlhhttps://arxiii org/ abs/ 2402 09353
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining
Approach, 2019. /tttpt:: org abs/ 11692
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-arr Paraiiter-Efffiient Fine-Tuning methods. https://gitee
com htmm/uuggingface/eftttt 2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PISSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, 2024. URL httth://aaxxxi wwweiii.r.....eee.. wrggabs/////e///o.oo/...o...esr/ /a//ab/////// 2404 02948
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 2022.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. URL httthhttt//aaxxttt www.aawwwww..aaaaaeeeeeeeee 06786)
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pages 606-610, 2007.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, August 2021. ISSN 0001-0782.
doi: 10. 10.1145/3474381. URL. htttps::t//// /ttp::////o...r//1//// 1145/3474433 3474331
Julian Salazar, Davis Liang. Toan Q. Nguyen, and Katrin Kirchhoff. Masked Language Model Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020. doi: 10. 1/203///l/02 mai.. URL
http://dx: www.do........oo...............
18653/ 18653/v1/202/ acl main. 240
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense Reasoning about Social Interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 3463, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 1653//11/1191114555
URL /ttts::///cc
wrgg 1454
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selectiv Rank Reduction. The Twelfth International Conference on Learning Representations, 2024. URL
Ying Sheng. Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng. Kurt Keutzer, Joseph E. Gonzalez, and lon Stoica. S-lora: Serving thousands of concurrent lora adapters, 2024. URL hhtthststs/ttt/t// orrg abs /bbs/2111.. 03285
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionalityy Over a Sentiment Treebank. David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors. Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
URL LinuuuttioonllLLguuiitisssss
org D13 1170
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An Instruction-folloiinng L.LMMA model. https: //te:////tttttttt com
aalaaaalppaaapaaeaaeaaaaeea___le 2023,
12

---

James Thome,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a Large- scale Dataset for Fact Extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Pupers), pages 509-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1074, URL
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anm Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023a. URL
org/ abs///////// 2302 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez. Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open Foundation and Fine-Tuned Chat Models, 2023b. URL https: //ae:i///xxvvi org/ abs /b3/ 2307 09288
Tricu H. Trinh and Quoe V. Le. A Simple Method for Commonsense Reasoning, 2019. URL https://arxiv. org/ abs/ 1806. 02847
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GI.UE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations, 2019. URL httph: httptts:///peeeeeiiieteee net/ forum?id=r:44mm25577
Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient 11m finetuning, 2024. URL https::/://ax org/ abs/ 000s/400609044
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering Code Generation with OSS-Instruee In Proceedings of the 41st International Conference on Machine Learning. International Conference on Machine Learning, 2024. URL https://arxiv
org/abs 2212000022022
Adina Williams, Nikita Nangia, and Samuel Bowman, A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/w1/N18- 1101. URL htttp:://aallattoolllla
org NN8 1101
Zhengxuan Wi, Aryaman Arora, Zheng Wang, Attieus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. ReFT: Representation Finetuning for Language Models, 2024. URL
https://aaxxv.orrgabbb/ www.:////.........o./// 03592
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning. In Proceedings of the 41st International Conference on Machine Learning. International Conference on Machine Learning, 2024. URL https://arxxis org/ abs 0ts//2401.0415/
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In The Twelfth International Conference on Learning Representations, 2024. URL btstps://pennreieww.cco wtwp://wwwwwateeetooo//ooaaaaaan
13

---

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?. 2019. URL httppp::///rxiiv.or///////// aa////////111/ 07830
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. In The Eleventh International Conference on Learning Representations, 2023. URL stttps://operrevien net/
forum?id foruurid--(gEuuNRJji
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in Low-Rank Adapters of Foundation Models. In ICLR 2024 Workshop on Mathe- matical and Empirical Understanding of Foundation Models, 2024. URL htt.h:tp:/// htttps://openrevier:
bett/forum?dd-Prrrbiee
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. The IEEE International Conference on Computer Vision (ICCV), December 2015.
Terry Yue Zhuo, Armel Zebaze, Nitchakam Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter- Efficient Instruction Tuning Code Large Language Models, 2024. URL https:/ //tp///xvvv//./////v////////aav/ org/orgg//.ooo..ooooooo /a0p///201...0 00788
14

---

A Limitations
While we gather diverse of model (across sizes and types) and task types, it is always possible for findings to fail to hold for even larger models or in different scenarios. However, we have no reason to believe this is the case. Further, point that we lack the compute required to be able to fine-tune larger models.
B
Why do intruder dimensions exist & can we alleviate them?
Here, we discuss possible causes of intruder dimensions.
B.1 What do intruder dimensions do?
Conjecture: Intruder dimensions, as high-ranking singular vectors, contribute significantly to the norm and stability of the parameter matrix. In contrast pre-trained singular vectors that learned from large pre-training corpora, LoRA introduces intruder dimensions learned solely from the smaller dataset of the fine-tuning task, which overpower the pre-trained vectors, as seen in the experiments so far. This suggests that these intruder dimensions are very task specific. On the other hand, full fine-tuning, while adapting just effectively to the fine-tuning task, retains the spectral properties of the pre-trained model effectively. Our experiments that scale down intruder dimensions provide evidence for the claim that intruder dimensions specialized the fine-tuning task, since we observe that increasing the norm of an intruder dimension (using X >> 1) leads to little change in adaptation task performance but leads a significant increase in forgetting (pre-training loss).
B.2 Adding a random vector
Adding an random vector pre-trained matrix introduces an intruder dimension: To help provide intuition about how new singular vectors in the SVD can be added by LoRA, we examine mathematical conditions that lead to their creation. Certainly, when comparing SVD(W + Xxo?') and SVD(W), where W are the pre-trained weights in RTxx, is a randomly sampled vector in R", and À a scalar value greater than the largest singular value of W, we expect this update to create an intruder dimension (as v is nearly orthogonal to the existing singular vectors w.h.p.).
B.3 Differences in update rule
As described in Appendix F LoRA and full fine-tuning have characteristically different update rules, even for the same training examples. We highlight that LoRA has gradients projected into a low-rank space (Hao et al. 2024 leading to conditions similar the toy example in section B.2 above.
B.4 Impact of learning rate on intruder dimensions.
We study the impact learning rate has on intruder dimensions by sweeping a range of learning rates while keeping all other hyperparameters fixed for LoRA = 8 and fine-tune on MNLI. Across training epochs, we report the number of intruder dimensions, test accuracy, and pre-training loss (Fig- 77 Across models, we see they have similar test accuracies after 5 epochs but very different numbers of intruder dimensions. We note that small learning rates do not converge as fast as the ones tested and have difficulty reaching the maximum performance that larger learning rates are able to reach. We see that as we increase learning rate, the number of intruder dimensions also increases (left, Fig. 77. This illustrates tradcoff in the selection of learning rate of LoRA: picking larger learning rate may lead to faster convergence and potentially better test accuracy with more intruder dimensions and drift in overall language modeling performance, while smaller learning rates may lead to less drift but potentially lower test accuracy.
Because of this experiment, one may dispute our findings with the claim that they are due to our specific selection of hyperparameters. Therefore, we find important to note that we adopt hyperparameters from prior literature (Hu al. 2021), default settings in common machine learning
15

---

full
num iintruderrss
MNLI, B only
QQP, B only
SS12, B only
SIQA. B only
WinoGrande, B only
FEVER, B only
400
200
200
100
200
100
0.25
0.75
0.25 0.50
0.75
0.25 0:50 0.75
0.25 0.50 (threshold)
0,75
0.25 0.50 0.75 E (chreshold)
0.75 E (threshold)
0.25
E (threshold)
(threshold)
(threshold)
Figure 10: Impact of only tuning B the number of intruder dimensions. We randomly initialize A such that it has singular values of 1, freeze it, and only train B. When we do this, we see a sharp reduction in high ranking intruder dimensions in comparison those in normal LoRA (reported in Fig. 4d Graphs for a specific dataset have the same range as Fig. 4d for easy comparison.
l and also study externally trained open-sourced models. This means that our findings are reflection of common practices and not due selection bias by us. Learning rates used by prior work were likely determined based on variety of factors like speed of convergence and best resulting test accuracy and therefore selected large learning rates that still converge well but coincidentally result in intruder dimensions.
B.5 Matrix product parameterization of LoRA
Multiplying matrices together amplifies their spectral differences (their singular values) and in most cases leads to a lower effective rank. To test the impact of the product BA on the introduction of intruder dimensions, we randomly initialize A such that all its singular values are I and freeze it. We only tune B and keep the rest of our fine-tuning recipe the same. Comparing this with vanilla LoRA fair because Zhuu et al. (2024) found that tuning B is more impactful and important for generalization comparison A and Hao al. (2024) showed that only tuning B effectively approximates LoRA. As we can see in Fig. 10 we see a sharp drop in the number of high ranking intruder dimensions when only tuning B in comparison to the vanilla LoRA case where we train and B separately, as reported in Fig. 4 This suggests that the matrix product of LoRA is an important component in the introduction of intruder dimensions because of how amplifies the spectral differences of B and A.
C Implementation Details
C.1 Evaluation details
We follow the precedence Biderman et al. [2024] when evaluating LLaMA2-7B: When fine-tuned on code, we evaluate on HumanEval [Chen et al. 2021] using the bigcode-eval-hamees [Ben Allal al., 2022]. When fine-tuned on math, we evaluate on GSM8K [Cobbe et al. [02T] using the -nleves- [Gao et al. 2024). On both, we evaluate task forgetting by evaluating on Hellaswag, WinoGrande, and Are-Challenge using the nn--val-hhrneee [Gao et al. 2024].
We measure language modeling loss for all our LLaMA2-7B models on random sample of its pre-training data distribution, according to Touvron et al. [2023b). We measure "pseudo-loss" for all our fine-tuned RoBERTa models on a random sample of the four datasets that RoBERTa used for pre-training(Openll WebText [Gokaslan and Cohen 2019). CCNews (Hamborg al. 2017 Stories
TTinh and Le 2019], and bookcorpus [Zhu et al. 2015) and weigh them proportionally to their contribution as described by Liu et (2019
C.2 Compute Resources
internal, shared SXA100-SXM4480GB machine. All RoBERTa-base fine-tuning runs required a single A100 GPU. All evaluations and analyses also required a single A 100 GPU. Many experiments were run sequentially due need to share these computing resources. Due to these constraints, instead fine-tuning our own aLaMA2-7B models, we use publicly released fine-tuned models. For more information on these models, see Section Each RoBERTa-base
All experiments were
'PETT. the most popular LoRA library, use learning rates > 1e-3 in their tutorials and states "With LoRA-like methods, you afford to use higher batch size and learning rate." [Mangrulkar al. 2022
16

---

fine-tune run takes at most 6 hours on single GPU. Evaluating an arbitrary LLaMA2-7B model for both test accuracy and forgetting takes about 45 minutes on single GPU.
C.3 RoBERTa fine-tuning details
We generally follow the procedure used by Hu et al. [021]. For all models, we use a linear learning rate schedule with 0.06 linear warmup ratio and train for maximum of epochs with batch size 16. We use the Adam optimizer [Kingma and Ba 2017] with no weight decay and maximum sequence length of 512. We fine-tune all linear layers besides the embedding matrix. For full tine-tuning we use learning rate of le-5. For LoRA, we set a 2r, and train for all ranks in {1, 2, 4, 8, 16, 64). We hold the "total learning rate of LoRA", which is a 77, fixed as we sweep rank such that this product always equals 2. 2.4e-3. We fine-tune these models to equivalent accuracy on their downstream task. We fine-tune on six sequence classification tasks: sentiment analysis (Socher et al.) 2013]. entailment [Williams et al. 2018), duplicate identification [Wang et 2019), fact verification [Thoree et al. 2018), and common sense reasoning (Sap et al. 2019 Sakaguchi et al. 2021].
D Model Accuracies
We report the accuracies that RoBERTa models achieve in Table 1 and Table are based on the models Table
Our main results
Model
Type Full
MNLI
SST-2
QQP
WinoGrande
SIQA
FEVER
0.8745
0.9438
0.9152
0.6582
0.6499
0.6892
r=1
0.8647
0.9358
0.9045
0.6251
0.672
0.6712
r=2
0.8604
0.9415
0.9058
0.6172
0.6581
0.6673
RoBERTa-base
4
0.8607
0.9369
0.9079
0.6472
0.6505
0.6694
8
0.8648
0.9438
0.9108
0.6417
0.6586
0.6582
r= 16
0.8604
0.9427
0.9095
0.6235
0.6853
0.663
r=
0.8671
0.9484
0.9117
0.6614
0.6638
0.6601
Table I: Model accuracies on their given downstream task after fine-tuning for a = 8.
Model
Type Full
MNLI
SST-2
QQP
WinoGrande
SIQA
FEVER
0.8745
0.9438
0.9152
0.6582
0.6499
0,6892
r=1
0.8677
0.9415
0.9042
0.6275
0.6418
0.687
0.869
0.945
0.9054
0.6551
0.6438
0.6822
RoBERTa-baa
r=4
0.8698
0.9472
0.9089
0.6361
0.6602
0.6827
0.8704
0.9472
0.9093
0.6346
0.6607
0.6928
1=16
0.8739
0.9461
0.9093
0.6417
0.6571
0.6924
r=
0.8719
0.9472
0.9061
0.6212
0.6167
0.6864
Table 2: Model accuracies their given downstream task after fine-tumin fine-tuning for results are based on these models.
2r. Our main
E
Cosine Similarity with Orthogonal Vectors that Span a Space
Here we demonstrate why is possible for a vector to have low cosine similarity with every orthogonal vector that collectively span space if the dimensionality of the vectors is high.
Minimizing the Maximum Cosine Similarity. Lets take Z = min max cos(u, 2v)) where v an
v
arbitrary vector and each vector &;, which we collectively call X, make up an orthonormal basis that span the space. Z can be small in high dimensional space.
2-D case.
Assume X = / without loss of generality. It trivial to see that Z = . 1 and is when
V2
V2
17

---

3-D case.
Assume I without loss of generality. == v3
when U
1
v3
v3
v3
N-D case. In the N-D case, can see, via induction, that Z
V/
As see here, if n is large, the value of Z will be low, even though we are doing the cosine similarity of a vector with respect to a set of orthonormal vectors that span a space.
F
Derivation of LoRA Adapter's Gradients
Our calculations derived independently but follow similar line to that of Hao al. 2024
Derivation for Full Fine-tuning. Full fine-tuning is structured such that
= tumed = (Wo + AW)X,
where X Rexh are the inputs, Y Rmxh are the outputs, Wo Rmxa are the pre-trained weights, and AW E mR is the fine-tuning update. Accordingly, 85 81 XT" and the update is J&W øy
0L OY
AW, AW,
X.
where 1) is the learning rate.
Derivation for LoRA. LoRA is structured such that
Y = Whuned = (Wo + 2 BA)X.
where X E Raxb are the inputs, Y Rmxb are the outputs, Wo Rmxn are the pre-trained weights, TR" is randomly initialized, and a is a hyperparameter. B Rmxr r is initialized zero, AL B" X". Therefore, their respective updates are Accordingly, DB ay A and 0d
OL AL
a 0L / 0Y
B B
X T BY
and
dL
4, An
n- 0Y
B"
XT
where v) is the learning rate.
Differences in First Step. During the very first step of training, given identical examples both full fine-tuning and LoRA have the same X and Y for each layer since B is initialized zero. After the first step, full fine-tuning has a update matrix equal to
OL X
AW full -1 σy
In contrast, LoRA has an update matrix equal to
A Wioor = (B)
a 0L x (A)
- -/ -/ By OL
a
gy x²).
11 σY
Since Bo O,
a 0L AWtara = (- oY
x" (Ao))
From this, we can see that the gradient steps are clearly different, even with the same training examples.
18

---

Intruder Dimersions
Intruckr Cimensions
Forcto Frantier
0.03
0.600
0.023
0.825
0.775
0.575
0.00
0.500
0.523
Ferpetong
0.59
0.000
0.500
0.475
0.475
r-64
0.57
0.550
0
224
400
102
500
1000
200
400
500
100
1000
0.50
2:58
C.0
6.22
0.45
0.66
0.45
0.40
0.40
0.00
0.90
0.50
0.30
magicoddr r-10
0.2
maggccoder 6-
0.52
mmagicoder Pull
maagicoder 2-256
0.50
Forgetting ng
Figure 11: For LLaMA2-7B, intruder dimensions correlate with forgetting. Top row: MetaMath. Bottom row: Magicoder. We display intruder dimensions vs accuracy and intruder dimensions vs forgetting.
G Intruder Dimensions Correlate with Forgetting
G.1 For RoBERTa
As mentioned in the main text, when measuring the Spearman correlation between the number of intruder dimensions and forgetting Fig. we find an extremely strong fit, with p = 0.971 and p-value < 0.001. This shows us that intruder dimensions strongly correlate with forgetting. In contrast, when we correlate intruder dimensions with performance, we find no such correlation: for RoBERTa, we measure p 0.3388 with p-value- 0.218.
G.2 LLaMA2-7B
When measuring the Spearman correlation between number of intruder dimensions and forgetting for our LLaMA2-7B models across training epochs (Fig. (middle)) we find a statistically significant relationship with p 0.59 and p-value 0.0006. When correlating intruder dimensions and test accuracy (Fig. II (left),, we instead measure = -0.3178 with p-value = 0.0869. Again, we see that intruder dimensions correlates with forgetting.
H Intruder Dimensions Cause Forgetting (Scaling Experiments)
H.1 Performance Differences When Scaling Down Intruder Dimensions
H.1.1 RoBERTa
We report our findings for RoBERTa models fine-tuned on MNLI, QQP, and FEVER in Table 3 Remember that we scale down using the equation W = Wo + AW + (A ju," Here, we see that scaling down intruder dimensions leads sharp drop in forgetting (pre-training loss) and much smaller drop in test accuracy. Scaling down an intruder dimension by two (A 0.5) results always leads less than two percent drop in test accuracy but double digit percentage drops in forgetting. One particularly compelling example, as shown in Fig. Sd shows how scaling down the top intruder dimensions with À = 0.7 when fine-tuning on QQP and using LoRA r=8 result in essentially no (0.0%) drop in adaptation performance but a large (-33.29 drop in forgetting. Our findings of the impact of scaling down intruder dimensions hold across three datasets and both LoRA r=1 and 8= which were the two ranks that we found to have many intruder dimensions. Note that
19

---

this experiment is meaningless if a model has no intruder dimensions, since no singular vectors will be removed.
Task
LoRA Rank
=
0.1
À =
0.3
=
0.5
X =
0.7
X
0.9
TA
PTL
TA
PTL
TA
PTL
TA
PTL
TA
PTL
[=1
-18.7
-13.3
-8.3
-14.5
-2.7
-13.7
-0.6
-11.0
0.0
-5.1
MNLI
r=8
-5.1
-24.7
-1.9
-23.1
-0.6
-19.8
-0.2
-14.5
0.0
-5.9
1=1
-8.6
-35.5
-4.4
-35.8
-1.6
-34.1
-0.4
-28.9
0.1
-14.6
QQP
- -3.3
-52.0
-1.6
-50.6
-0.6
-45.0
-0.0
-33.2
0.1
-13.0
1=1 -11.1
-10.3
-4.2
-11.8
-0.4
-11.1
0.6
-8.7
0.6
-3.8
FEVER
r=8
-5.6
-14.6
-1.
-1.33
0.7
-13.4
1.3
-9.7
0.6
-4.0
Table Impact of scaling RoBERTa-base's intruder dimensions on fest accuracy (TA) and pre training loss (PTL). Numbers reported are the percent change in test accuracy and percent reduction in forgetting induced by fine-tuning. Scaling down intruder dimensions leads less forgetting. On RoBERTa. PTPL. is Pre-training loss and TA is test accuracy. Both are reported as percent change with respect to the unchanged fine-tuned model. Scaling down intruder dimensions has large impact on forgetting but little impact on test accuracy.
H.1.2 LLaMA2-7B
Our finding that scaling down intruder dimensions leads less forgetting but similar test accuracy holds to LLaMA2-7B. One particularly interesting example is for LLaMA2-7B fine-tuned on Meta- Math with r 256: when scaling the top intruder dimensions down with 0.5, we see large drop (-25.2%) in forgetting and an increase (+1.8%) in test accuracy.
Dataset
LoRA Rank
À
= 0.3
À
=
0.5
À =
=07.7
À
=
0.9
TA
PTL
TA
PTL
TA
PTL
TA
PTL
1= 16
-15.0
-46.5
-6.6
-40.4
-2.7
-28.2
0.0
-00.5
MetaMath
r-
-5.8
-29.1
-4.4
-22.3
0.3
-14.2
0.3
-4,9
r=256
-0.1
-33.3
1.8
-25.2
0.5
-15.5
-0.8
-5.2
16
-1.6
-37.5
-0.6
-24.4
-0.1
-13.2
0.1
-4.1
Magicoder
1-64
-5.0
-12.7
-3.3
-8.3
0.3
-4.3
2.2
-1.4
2556
-4.3
-12.8
-1.3
-10.2
-0.9
-7.0
-1.6
-2.8
Table Impact of scaling LLaMA2-7B's intruder dimensions on test accuracy (TA) and pre
training loss (PTL). Numbers reported are the percent change in test accuracy and percent reduction forgetting induced by fine-tuning.
H.2 Intruder Dimensions Cause Worse OOD Performance
As discussed in section of the main text, for LoRA models with main intruder dimensions (r 1 and r = 8 in our experiments) we measure the impact of intruder dimensions by identifying and scaling the top intruder dimension in every weight matrix such that W Wo AW + (A - 1),,,,u where is the index of the top intruder dimension (note that À = 0 is removal, À = 1 is no change, MNLI, QQP, and FEVER, we use X E {0, 0.25, 0.5, 0.75, 1.0, 1.5, 2.0) and for each measure the test accuracy and pre-training loss. For comparison baseline, select the neighbor of the intruder dimension to separately scale.
and X 2 doubles the intruder dimension). For our RoBERTa models fine-tuned
We report these results in Fig. 12a (MNLI,, Fig. 12b (QQP), and Fig. 12c (FEVER). We find that when scaling down intruder dimensions (A < 1), we observe clear and significant drop in forgetting (pre-training loss) but a negligible drop in test accuracy (adaptation). When instead scale up intruder dimensions (A > 1), we observe that forgetting increases significantly. We observe that when scaling up top intruder dimensions by 50% (A 1.5), adaptation performance remains relatively flat with a large increase in forgetting, providing further evidence that intruder dimensions are task specific. These trends hold across all 6 models we study datasets with two different LoRA ranks).
In contrast, when we scale neighboring (pre-trained) singular vector of intruder dimensions instead, we observe starkly different behaviors. When scaling down (A < 1) these pre-trained singular vectors,
20

---

MINLI
OUP
lnmode:
mok--
Scaling intuder's
Pro Po-iaanig (Foggeting)
Tox Accuray (chdppptio))
rank-1
Scaling ion.dee'
Pre-uaingg (uetiin)
60
20
(Alpptton))) Tout
00
10
00
0-5
1.5
15
Scoling top tnmader
(scaling (otD0)
makk
Scaling xop intrude's neighbor
A (Scaling Sot0O)
saaaia Loak Poggning
Tox hosaray 9 (kdappaai))
Scaling
Occaiig
Scaliing
(ssling
neighhor
Pre Tretnisiig Loso Le (Fegaii)))
50
Goaling
Soco))
(a) RoBERTa base tine-tuned MNLI
(b) RoBERTa-base tne-tuned QQP.
snas
Toal Su (Aleeprto)))
Scaling up intrnder
Scaling top iwwadeess reiglbo
0.5
1.8
0/
0.5
1.0
Gicaling Getor)
(oallig DoXo))
Toe Accency (Acippssi))
Scaling top intrucer
rack
Scaling top irmaders naighbee
Pre-taiing Loossriggitng (frggiing)
0.5
18
0.5
Aocaiin Getert
Axcclling (acue)
(c) RoBERTa-base fine-tuned FEVER.
Figure 12: Scaling RoBERTa-base intruder dimensions. We scale the top intruder dimension in each matrix by X, a multiplicative constant, such that W = Wo + AW + (Á 1),,o,e Using < leads to large drop in pre-training loss while only slightly impacting the test accuracy. For Figs. 12a 126 and 12c, we also scale the intruder dimension's 's neighbor, which a pre-trained singular vector. Changing these vectors negatively impacts both pre-training loss and test accuracy.
we observe we see that forgetting sharply increases and adaptation performance drops more sharply than when scaling intruders instead. When scaling up (A > 1) these pre-trained singular vectors, we observe similar drops in forgetting and adaptation performance. This is likely because pre-trained singular vectors are well tuned for language modeling, and therefore any change to them will have negative downstream impacts performance. This shows that are observations are not due to the robustness the model to scaling down specific singular vectors, but rather the difference in contribution to model performance of intruder dimensions vs. pre-trained singular vectors.
Here, we clarify some possible points of confusion. We only scale the top intruder dimension each weight matrix, if and only if an intruder exists in that matrix, so even when X = 0 we do not recover the pre-trained model because not all weight matrices have intruder dimensions and it possible for multiple intruder dimensions to exist in weight matrix if LoRA > 1. Furthermore, intruder dimensions are not perfectly orthogonal the pre-trained singular vectors. Due the orthogonality constraint imposed by the SVD, all singular vectors will be changed slightly in the matrix, so that even when we remove the intruder dimension, the resulting matrix will be slightly different. These reasons are why when À 0 our LoRA r models do not return to baseline performance.
These findings hold LLaMA2-7B: in Fig see that scaling down intruder dimensions leads much less forgetting but similar test accuracy.
21

---

magicoder Rank 16
metamath Rank 16
Pre tee-trainngg Loss (Foretting))
2275
0.24
22500.6
Test Accuracy
0.22
0.2
2.4
0.20
2.200
0.18
2.175
2.3
0.16
2.1500.3
2.2
0.14
2.125
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
magicoder Rank 64
metamath Rank 64
Pre-Tre Loss forgettig))
0.30
Test Accuracy
2:00
0.25
2.8
0.20
2.0
220
0.15
2.2
2.10
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
magicoder Rank 256
metamath Rank 256
Pre-train (eggeinn)
0.35
235
0.7
3.2
Test Accuracy
0.30
0.6
3.0
225
2.8
2.6
2.0
2.4
2.15
0.15
2.10
0.4
0.0
0.6
0.8
1.0
(scaling factor)
A (scaling factor)
Figure 13: Scaling down LLaMA2-7B's intruder dimensions leads to less forgetting and nearly equivalent test accuracy.
I The Effective Rank of the Update Matrix Depends on Alpha
Kalajdzievski 2023 found that LoRA can have gradient collapse when rank is high if alpha is not set properly. Biderman et al. 2024) found that setting a 2r is very important for the performance of high rank LoRÁ. In this section, provide additional evidence of the importance of setting a 2r and show that if a held fixed, high rank LoRA converges to low rank solutions. To do this, for both a 2r and updates for different LoRA ranks. Effective rank is a measure of the information density of matrix and be thought of weight matrix. It is computed using the singular values of a matrix and we expect the LoRA rank to be the upper bound on what the effective rank can be: LoRA r 8 should have an effective rank update of at most 8. We present the effective rank measurements in Fig. [4] In this plot, we find that 64 and 768 have much higher effective rank than 768 appearing to always have effective rank above 100. In stark contrast, we see
8, we measure the effective rank [Roy and Vetterli [007]] of the weight matrix
estimation of the rank needed to capture the information held in the
when 2r (Fig. 14a.. LoRA
and r16 with that when a 8 (Fig. 14b). high ranks of LoRA have much lower effective ranks, frequently even converging to the effective rank of much lower rank updates (like 8 and 16). For example, LoRA 768 has an effective rank that consistently below when a 8. note that full fine-tuning has effective rank update of above 400 consistently. These plots suggest that when kept fixed when scaling LoRA rank, the solutions are uable to take advantage of their higher expressability and instead converge to low rank solutions.
22

---

16
64
768
16
66
766
Effective Rank of update when
Effective Rank of update when a B
Wo
Wo
Wr
Effeative Rank
Effective Kank
20-
78-
18-
Wy
Wo
Wy
Wo
Eftletive Rank
i:flestive Kank
25
100
100
MLP,
MLP2
MLP1
MLP2
Elleetive Rank
Efffectivv Kank
200-
100
09
100
100
Layer Number
Laycr Number
Layor Number
Layer Number
(a) Effective Rank of the LoRA update when
(b) Effective Rank of the LoRA update when 8.
Figure 14: Effective rank of LoRA update matrices (AW) for RoBERTa fine-tuned on MNLI. We
observe that when than the same ranks of LoRA but instead with a = 8. Building on Kalajdzievski [2023], Biderman al. 2024), this suggests that a is necessary for high ranks of LoRA to utilize their expressive capacity. Note: full fine-tuning consistently has updates with effective rank above 400.
2r, higher ranks of LoRA
64, 768) have much higher effective rank
Impact of Matrix Percentage on Number of Intruder Dimensions
In this section, examine the extent which intruder dimensions exist throughout the entire weight matrix and how they are distributed. As described in the main text, we hold e fixed as c = 0.5 and measure the number of intruder dimensions while varying the proportion of the fine-tuned singular vectors that we examine (this means varying our k parameter in Algorithm I. Here, we can see that intruder dimensions than full fine-tuning, regardless of what fraction of
LoRA consistently has the singular values we examine. The only caveat to this is that, for some datasets, full fine-tuning passes LoRA due to the limited expressivity of rank I updates and is interesting because it suggests that in this case, full fine-tuning may be changing lower-ranking singular vectors more than LoRA. One interesting contradiction to our findings is in Fig. [5d] which shows that full fine-tuning and LoRA appear have very similar distributions of intruder dimensions within their matrix when fine-tuned on code. This is likely due to the large domain shift from natural language to coding tasks (Biderman al. [2024] also make this observation of large domain shift required for models fine-tuned on
when examining the last 20% of the fine-tuned singular vectors. This is likely
Magicoder Wei et al. 2024
23

---

F1
P2
F4
53
116 SIQA
F4
full
QQP
SST2
WinoGrande
FEVER
num intruders
10000
TECH0)
3000
4000
(0002
200
1000
1900
5000
0
0
100
100
50
100
examined
% matrix examined
% matrix examined
9 matrix examined
% marrix examined
matrix examined
(a) Impact of the number of singular vectors in the fine-tuned matrix examine, k. on the number of intruder dimensions for RoBERTa models fine-tuned on 6 different tasks. Here, we set 0.5.
LLaMA-7b on Alpaca
LLaMA2-76 MetaMathQA
LLaMA2-76 Magicoder
Fall
Bill
num intruders
satk-16
mad--16
1000000
mmk--64
meli-54
0020932
2015
30006
100000
400000
1000
202000
100
1/6 matrix examinad
% matrix examined
%. of mattis examined
(b) LLaMA-7B fine-tuned on (c) LLaMA2-7B fine-tuned on (d) LLaMA2-7B fine-tuned on Alpaca.
MetaMatthQ.
Magicoder-Evol-Inet
Figure 15: Impact of k, the number of fine-tuned singular vectors we examine, on the number of intruder dimensions. We see that models fine-tuned with LoRA tend to have more intruder dimensions than full fine-tuning, regardless the value of used.
K Impact of Dataset Size On Intruder Dimensions
The total number of intruder dimensions increases proportionally the size of the fine- tuning dataset. Using our training recipe (Appendix C.3 we tine-tuned models on data subsets of varying sizes. We trained RoBERTa-base on MNLI using LoRA with rank and 8 (cases where we originally saw intruder dimensions) and measure the number of intruder dimensions along with the impact of and & (Fig. 16). For r = 8, as we train on more data, more intruder dimensions are introduced. Interestingly, however, LoRA with rank appears to converge to similar amounts of intruder dimensions, regardless of the dataset size. This may be because of the limited expressivity of models with r 1. This experiments suggest that with smaller datasets, fewer intruder dimensions may be introduced by LoRA.
-Well
Imssct of Finetuning Detsset Shee High Ranaing Intruder Dimensions L-B
Impact of finetuning Dikast Sier
Miruder Dimaralors
LORK m
LoFA-w]
100
8002
15000
150
250
400
60
200
5200
lIhheeshoi)
matrir
Figure 16: (Left) Impact of cosine similarity threshold, e, on the number of intruder dimensions for LoRA models trained on different proportions of the MNLI dataset. (Right) Impact of the number of fine-tuned singular vectors we examine, k. on the number of intruder dimensions for LoRA models trained on different proportions of the MNLI dataset. We see that training on larger proportion of the dataset increases the number of intruder dimensions in the model.
24

---

L LLaMA/LLaMA- Instruction Tuned Models
Our LLaMA-7B checkpoints were fine-tuned on the Alpaca [Toor et al., 2023 and consist of two fully fine-tuned models, one LoRA model with rank 16, and one QLoRA Dettmers et al. 2023] model with rank 64. Our LLaMA2-7B checkpoints were fine-tuned on either code (IFT with Magicoder-Evol Instructtt 1 10K [Wei et al. 2024] or CPT with StarCoder [Li et al.
2023] or math
(IFT with MetaMathQA (Yu et al., 2024] or CPT with OpenWebMath (Paster et al. 2023 and
consist of one fully fine-tuned model and 3-4 LoRA' ed models of different ranks for each dataset and
generously provided by Biderraa al. 2024]. In Fig. [aa Full refers PKUU-Aligmmentalpacaa
7b-reproduced" and Full #2 refers to "chavimlo'alpac--natti
Hugging Face Path
Base Model
IT Dataset
tindettmers//lora alpaca--
LLaMA-70
Alpaca
tloen/al Siven//lpcaa-lor-//0
IlaMA-76
Alpaca
-LbMA-7b
Alpaca
chavinlofalpacc--attie
LLaMA-70
Alpaca
LoRA-TMI httpa//00/0//aaaaaeeer/ra/a/a///
ILaMA2-7b
Magicoder
LoRA-TMLR
LLaMA2-7b
Magicoder
LLaMA2-7b
Magicoder
LoRA-T
ILaMA2-7b
Magicoder
LLaMA2-7b
MetaMath
LLaMAA27b
MetaMath
LORA-TTMLRR2222//mecmmnaall
LLaMA2-7b
MetaMath
ILaMA2-7b
MetaMath
ILaMAA-7b
StarCoder
LORA-TMLL 202/////satacooooroooooaaaaao///
LLaMA2-7b
StarCoder
LORA-TMI.--200 TTT.M./...0AAkarrrodddd---------
ILaMA2-76
StarCoder
TOLR- MAA.TTMMM22222/aArrooooooooo 202444aarcooo--flll--nuuuningggn 00--::n-iiin-------------------- 30Betoken
LLaMA2-7b
StarCoder
LORA SORA-TMLR- 00244openweebmatt-loozz-aaaa 10---01-2-----e--
LLaMA2-7b
Open
LORA- TMLR 2024/ppenwwwbbbmtttlooo.... 64-00Bts
LoRA-TMLR- LARA--MM.R22224/0pppee 20044/ppwwwehhmmh-----aaaaaa----
ILaMA2-7b
OpenWebMath
LLaMA2-7b
OpenWebMath
TORA-TMLL-2022//peeeee hteel//pnweemmaat--lll--eeeeee
LLaMA2-7b
OpenWehMart
Table Hugging Face model paths for LLaMA-7b/L
models.
M
Continual Learning
M.1
Performance during continual learning
As described in Section in the main text, train sequentially 6 tasks and measure task performance on all of these tasks across tasks trained on (continual learning). We report the full graph of our findings in Fig. [7 In it, we find that when all our models are trained similar accuracy, lower ranks of LoRA, which coincide with more intruder dimensions, forget more of their previously learned tasks than higher ranks of LoRA and full fine-tuning.
M.2 Similarity matrices during continual learning
After each continual learning dataset we fine-tune on, we measure the similarity matrix between the current model and the pre-trained model. In Fig. T8b] we observe that LoRA accumulates intruder dimensions across fine-tuning datasets. In contrast, in Fig. [8a we observe that the pre-trained structure of the model is retained well across fine-tuning datasets. These experiments suggest why LoRA appears to degrade faster during continual learning.
25

---

Pall
LoRA,
LoKA, M6
Accuracy on
Accuracy QQP
Accuracy
0.90
0.8
0.85
0.7
Accuracy on SIQA
Accumey WinoGrande
Accuracy FEVER
0.65
0.5
0.60
0.1
0.525
0.55
0.500
RoBEETT
MNLI
Oge
9812
stgs
Win-
Goando
FEVER
RottERT
MNLI
9872
S044
Wine-
Graele
FEVER
RottE
MNII
cop
S872
SIQA
Wine-
Gronae
FEVER
have
hune
bear
Ditaset Fine-tuned
Figure 17: Full plot Fig. 9a
Continual Learning performance of RoBERTa for full fine-tuning and LoRA. We sequentially train six tasks, in order from left right. Horizontal dotted line indicates baseline pre-trained performance. Vertical solid line indicates when a specific dataset fine-tuned on. Gray region represents performance before the model has been trained on that task. We interested in the differences in accuracies of these methods both right after training (at the vertical black line) and later (in the white region). We that low ranks of LoRA forget previously learned tasks more.
Singule Wettors
Singuler Vectors Mause
Sinulttr Vectors Mune
Singular Vesters
Singibar Vectors
Singian Vestors Waoe
Singuls Vectors Wa
Muse
Continual learning similarity matrices for full fine-tuning
(#2)
Singeld Nettrr
Singular Vectors
Singultr Vectors Mi
Singular vestors Minm
Sinpibr Vectors Mine
Singuan Vestors Winee
Siquil-Wettree W.
(b) Continual learning similarity matrices for LoRA.
Figure 18: LoRA accumulates intruder dimensions, while full fine-tuning does not. The pre- trained structure of the model degrades across tasks trained
26

---

N
8 for Case Study: Setting Alpha=8 instead of Alpha=2r
Our main experiments
conducted with 2r. However, Hu et al. 2021 instead
RoBERTa-base While not the recommended practice now, we explore what impact this selection has on our findings. We report our key plots in Fig. 19a 19b 20 21 & 14b In Fig. 19a & 19b we see that LoRA'd models with high rank have significantly more intruder dimensions in comparison to 2r LoRA models with ranks like 64 had no very few intruder dimensions (see Fig. 4, they now have numerous intruder dimensions. These differences are corroborated by Fig. 14b where we see that the learned solutions of LoRA have significantly lower effective rank in comparison to when a 2r. For example. we see in Fig. 14b that when LoRA has rank of 768, the effective rank is never above 100. In contrast, we see in Fig. 1 4a that
when a 2r. Interestingly, whereas when
with the same rank of 768, LoRA always has an effective rank above 768. This suggests that when
= 8, LoRA converging to lower rank solutions than when a = 2r. This supports the finding that setting = 2r improves LoRA's performance when high rank is used [Biderman et al. 2024 Kalajdzievski 2023]. Behaviorally, we see in Fig. 21 that LoRA models with high rank have much more forgetting previously learned tasks in comparison to full fine-tuning and LoRA when
2r used (a 2r results are Fig. [7). Likewise, in Fig. 14b we see that when LoRA has high rank, it has much more forgetting the pre- rre-trained distribution in comparison to LoRA when 2r.
60
sinitiity foehhldd
sinilaatty dooshold
similaii theshledd
simiaryy Oressal
sitilaiit Oredoold
siillall fombold
(a) Number of intruder dimensions in RoBERTa models fine-tuned on 6 different tasks. Here, we set k 10. We use the same conditions in Fig. 4r but instead now set 8 instead of a 2r.
fal RoBERTe
RoiBERTa MNLI
RuvBEkTia
RoBERTa
Ie dimorerre
50
caamlted
(b) Impact of the number of singular vectors in the fine-tuned matrix we examine, k. on the number of intruder dimensions for RoBERTa models fine-tuned 6 different tasks. Here, conditions Fig. but instead now set
0.5. We use the
instcad of 2r.
Figure 19: We find that when 8 instead of 2r, our models have more intruder dimensions. (Top) Replication of Fig, 4d with a = 8 instead of a 2r. (Bottom) Replication of Fig. [5a with 8 instead of 2r.
O Impact of Random Seeds
To ensure that random seed does not play role in the number intruder dimensions we observe in model, we sample 5 different seeds and fine-tune RoBERTa-bass MNLI using the same methodology as Fig. 4d We find that the initialization has negligible role on the number of intruder dimensions. This shows that our findings not dependent the random initialization of the LoRA modules.
P LoRA Variants
We focus significantly on the standard LoRA method proposed by Hu et al. [222] in order to study it depth. However, many variants of LoRA have been proposed recently. A [Zhang et al. 2023 adaptively allocates LoRA rank to different modules in order to ensure optimal allocation of trainable parameters to certain modules. LoRA+ [Hayou et al. 2024] sets different learning rates for the A and B modules in LoRA. PiSSA [Meng et al. 2024 initializes the A and B modules with
27

---

MSLII
45
MNLT
SNT2
SNT2
46
SIQA
Pre-training Pscudoo
OC?
3,5
3.0
2.0
Dull Time-Tninn
LoKA
LoKA
Full
LeltA
LoRA
8-8
6-64
Finz Tin-Taring
--1
0-44
Figure 20: For a RoBERTa's performance on its pre-training data distribution after fine-tuning on particular task. We measure pseudo loss as described by (Salazar et al. [2020]. We compare these results to when a 2r (Fig. 6
Full
LoRA,
ассurRcy mnli
accuracy
accuracy sot
Accuracy
1.25
0.50
0.5
accuracy on diga
accuracy winegrande
accuraey on fever
1.0
142
5.9
1.0
0:35
RoUERTE
MNLI
DOP
S912
SQQ
Win-
Graudo
DIVER
RoBERTs
ANNLI
5572
SYGA
Wine-
Gradk
FEVER
RolERTa
MNLI
COP
5572
SIO4
Grande
FEVER
bas
dataset fine-tuned
Figure 21: For a = 8. RoBERTa's performance on six datasets during continual learning. We sequentially train six tasks, in order from left right. Horizontal dotted line indicates baseline pre-trained performance. Vertical solid line indicates when specific dataset is fine-tuned on. We compare these results when a 2r (Fig.
the top ranking singular vectors of the pre-trained weights. VeRA [Kopiezko al. 2024 models LoRA as the product of two random matrices with trainable parameters doing elementwise operations on the resulting vectors. These variants may have important impacts on the presence of intruder dimensions. For example, PiSSA initializes with the singular vectors and therefore may have an easier time changing them, possibly leading to more intruder dimensions. contrast, LoRA+ in effect lowers the learning rate, which we found to be important to introducing intruder dimensions, and may therefore reduce the number of intruder dimensions.
In order to examine if intruder dimensions are still relevant for these methods, we rerun our MNLI fine-tuning experiment with RoBERTa with each of these methods with default hyperparameters that they provide. These results are supplied in Fig. 23 Interestingly. we see that all the variations we examine have intruder dimensions. Some interesting observations include: LoRA+ and LoRA
= 1 appear to have nearly identical curves, suggesting they have very ismilar intruder dimension characteristics. We again see that with higher ranks (r = 64) these LoRA variants tend to have very few intruder dimensions. However, it does appear that methods that explicitly modify the singular vectors, like PiSSA, have many intruder dimensions. This makes sense since they are explicitly constructed to modify the singular vectors the pre-trained model. These findings emphasize that
28

---

MNLI
ceed+o
num intruders
scade)
200
szad-4
100
0.6
0.8
(threshold)
Figure 22: Impact of Random Seeds on intruder dimensions. We fine-tune RoBERTa- base across
5 random seeds and our same methodology as in Fig. 4d We find that the initialization has negligible role on the number of intruder dimensions. This shows that our findings are not dependent on the random initialization of the LoRA modules.
MiLooRAss LoRK- 2
PSSSA.8
LoRA.rd 11RAL-512
AdlloRA, 64 -.e.. LoRA*,
PISSA (64 ---- LoRAU - Full
Intruder Dimensions LoRA variants RoBERia
350
num intruders
250
150
100
8 (threshold)
Figure 23: Measuring LoRA variants for intruder dimensions. = 10. We compare variants of LoRA normal LoRA (blue) and full fine-tuning (black). We find that the LoRA variants examine still have intruder dimensions and shows that our findings are not just exclusive to normal LoRA.
intruder dimensions are not just an observed phenomenom in normal LoRA and suggests to future work the examination LoRA variants.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Each claim is clearly stated. We callout with bold each claim in the introduction and takeaways are bolded in later sections to call out our key claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes. We discuss the limitations of our work throughout the paper. We also add a new section the appendix to discuss limitations (Section A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [NA]
Justification: We have no theoretical results. We briefly justify results with calculations in the appendix(section F). but these not central or important our results. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We release code replicate our main results (code is not attached to avoid breaking double-bline review, link will be attached upon acceptance). This github repository contains exact commands needed replicate our findings. Additionally, methodology for reproducing experiments is located in the appendix. All our data is publicly available. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
31

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We release code to replicate our main results (code is not attached to avoid breaking double-bling review, link will be attached upon acceptance). This github repository contains exact commands needed to replicate our findings. Additionally, methodology for reproducing experiments is located in the appendix. All our data is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//apsa.ccppulic/gu.udee httpa////gggdeee/eeeuuubbbbibiia SCt////ooSSSeiiaaeeeeeeee seciissssoooooooi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details for replicating our training and evaluation methods are provided in the appendix. This includes datasets, models, hyperparameters, optimizer, etc. See section
C
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars for most experiments are not used because of the computational requirements for these experiments, making infeasible to, for example, perform more than one training run. Elsewhere, analyses are deterministic and do not need error bars. In addition, we do report statistical significance tests for our correlations of intruder dimensions and forgetting and intruder dimensions and test accuracy.
32

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All of this information provided a subsection of section Cin the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////eerppp..cc///ll//eeeee
Answer: [Yes]
Justification: this paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
33

---

Justification:
This paper does not have a true societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite every single model (Section L and codebase/dataset (Section C) we use. While difficult to find licenses for these resources, they are all open source.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
34

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLM usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httppe///earrrpccccc///oeeeerrre for what should or should not be described.
36


Reviews:
[{"review_id": "BoSimKJd9F", "reviewer": "Reviewer_BfKg", "summary": "This paper studies the difference between full fine-tuning and LoRA through spectral analysis of weight matrices. The authors identify intruder dimensions—high-ranking singular vectors in tuned weights that are dissimilar to any pre-trained directions—and show they are a key cause of forgetting. Scaling down these components reduces forgetting without harming downstream too much performance. The accumulation of intruder dimensions is also shown to hurt LoRA in continual learning.", "questions": "- The paper concludes that intruder dimensions cause forgetting. However, full-tuning with few intruder dimensions,  still exhibits more forgetting than LoRA with an appropriately chose $\\alpha$. How do we understand this based on findings in the paper?\n\n- In the continual learning setup, are the similarity matrices in Figure 9 all computed with the original model? I am wondering can we observe instruder dimensions over a model already tuned on another task(say do weights after task 3 have  instruder dimension over weights after task 2)?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The authors provide a novel perspective on the difference between LoRA and full fine-tuning. The concept of intruder dimensions reveals a concrete structural difference between full-finetuning and LoRA.\n\n- The link between intruder dimensions and forgetting is insightful. It offers both theoretical understanding and potential avenues for mitigation.\n\n\nWeaknesses\n\n- Logical mismatch in Section 5. Section 4 convincingly correlates the *number* of intruder dimensions with forgetting, yet Section 5 only experiments with scaling down the *magnitude* of the top intruder dimension. To solidify the argument, the authors should also manipulate the *count* of intruder dimensions and measure its effect on forgetting.\n\n- Limited practical guidance. The paper does not provide guidance on how to prevent the emergence of intruder dimensions during training, which is crucial for improving real-world applicability. As it stands, intruder dimensions appear to be primarily useful as a post-hoc model selection criterion. However, in practice, comparing two models based on their performance on pre-training data may be more straightforward.", "was_revised": true, "final_justification": "<xaiArtifact artifact_id=\"46a11797-61d4-444b-b2d8-0c570d1c2f93\" artifact_version_id=\"f7acb820-abe4-44da-8209-e4672a4d1da2\" title=\"Review_Response.md\" contentType=\"text/markdown\">\n\nResolved Issues:\n\n- Clarified the relationship between intruder dimensions and forgetting.\n- Provided a clear explanation of the method to prevent intruder dimensions and addressed potential logic mismatches.\n\nUnresolved Issues:\n\n- Additional evidence is needed to connect Sections 3 and 5 effectively.\n- Algorithmic suggestions for mitigating intruder dimensions remain limited.\n\nIn conclusion, this is a technically solid and engaging paper with some limitations. I assign it a rating of 4.", "rebuttal": "We thank the reviewer for their thoughtful review. We provide responses to their review below.\n\n> Logical mismatch in Section 5. Section 4 convincingly correlates the number of intruder dimensions with forgetting, yet Section 5 only experiments with scaling down the magnitude of the top intruder dimension. To solidify the argument, the authors should also manipulate the count of intruder dimensions and measure its effect on forgetting.\n\nWe actually do study altering the count of intruder dimensions in section 5. When scaling with $\\lambda=0$, we remove that intruder dimension. In Section 5, we remove the top intruder dimension *in each weight matrix*. This has the effect of removing up to 72 intruder dimensions from a RoBERTa model and has the effect of significantly reducing forgetting while causing less extreme drop in test performance. For examples, see Fig. 8 when $\\lambda=0$. We hope this clarifies the reviewers confusion.\n\n> Limited practical guidance. The paper does not provide guidance on how to prevent the emergence of intruder dimensions during training, which is crucial for improving real-world applicability. As it stands, intruder dimensions appear to be primarily useful as a post-hoc model selection criterion.\n\nThis paper provides several concrete suggestions on how to **reduce the emergence of intruder dimensions at training time**. We provide evidence that certain initializations (Fig. 10), lower learning rates (Fig. 7), and setting $\\alpha=2r$ (Section N) all lead to fewer intruder dimensions. These are concrete prescriptions that can be used to prevent the emergence of intruder dimensions during training.\n\nMoreover, we show that scaling down the top intruder dimensions after LoRA fine-tuning recovers the drop in pretraining performance without affecting downstream adaptation. In one such case, we see forgetting reduced by 33.2% with no change in test accuracy (lines 292-293). This effectively **resurfaces forgotten** information without compromising task performance—a novel and actionable intervention that leads to improved OOD generalization which was not previously known.\n\nThese are a few practical guidelines offered by the paper.\n\n> The paper concludes that intruder dimensions cause forgetting. However, full-tuning with few intruder dimensions, still exhibits more forgetting than LoRA with an appropriately chose alpha. How do we understand this based on findings in the paper?\n\nIt is important to note that intruder dimensions are **a cause of forgetting** in LoRA'd models, but not the **only cause** of forgetting in fine-tuning in general. This is obvious, since we should expect any sort of deviation from the pre-trained weights, which were trained to minimize language modelling loss, to lead to an increase in language modelling loss (aka forgetting). This finding demonstrates that these two methods update weight matrices in fundamentally different ways. This explains the structural basis of forgetting with LoRA'd models (and variants of LoRA). However, a functional explanation of forgetting in full-finetuned models can be different and an interesting avenue for future study.\n\n> In the continual learning setup, are the similarity matrices in Figure 9 all computed with the original model? I am wondering can we observe intruder dimensions over a model already tuned on another task(say do weights after task 3 have intruder dimension over weights after task 2)?\n\nYes, all similarity matrices in Figure 9 are computed using the original model. That said, we do observe what you're describing: as we train on more tasks, new intruder dimensions emerge in **distinct locations** with every round of adaptation to a new task (highlighted in pink), indicating that intruder dimensions can indeed arise relative to earlier task weights (e.g., after Task 3 vs. Task 2)."}, {"review_id": "UOFAXrz1mI", "reviewer": "Reviewer_GjPE", "summary": "The paper analyzes the differences between LoRA and full fine-tuning of LLMs by examining the spectral properties of the resulting weight matrices. Specifically, the authors compare the singular vectors of the pre-trained and fine-tuned weights and find that, while full fine-tuning largely preserves the original spectrum, LoRA introduces intruder dimensions—singular vectors with large singular values that have low similarity to any singular vector from the pre-trained model. The authors conduct thorough experiments to study how various hyperparameters, such as LoRA rank and alpha, learning rate, the parameters used to compute the number of intruder dimensions, and even variations of the LoRA method, affect these intruder dimensions. They then analyze the forgetting behavior of LoRA in both fine-tuning and continual learning settings, and argue that the intruder dimensions contribute to it.", "questions": "All questions and concerns are detailed in the Strengths and Weaknesses section. The main weaknesses 1–3 are the most critical and will have the greatest impact on my final evaluation after the rebuttal.", "limitations": "I believe the limitations section should provide a more thorough discussion noting that the paper focuses only on one aspect of the spectral differences between LoRA and full fine-tuning, the intruder dimensions, while leaving aside the effect of the overall rotation of the spectrum and the changes in singular values.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses an important problem: understanding the properties of solutions obtained with LoRA is highly relevant, given its widespread use.\n2. To the best of my knowledge, the analysis of spectral differences between LoRA and full fine-tuning is novel, and the results are nontrivial and interesting.\n3. The experimental setup is strong. The use of different architectures and datasets supports the generality of the intruder dimension phenomenon.\n4. The paper is thorough and includes a wide range of additional experiments and ablations.\n5. The paper is clearly written and easy to follow.\n\nMain weaknesses:\n1. The focus of the paper. The paper focuses on intruder dimensions as the main factor distinguishing the spectral properties of LoRA and full fine-tuning, but it is not entirely clear whether this choice captures the core difference. According to Figure 2b, the difference appears to be more uniform: all singular vectors in the LoRA solution tend to have lower cosine similarity, and there is no clear small subset of outlier directions with significantly lower similarity. This suggests that LoRA broadly rotates the basis rather than introducing a few distinct intruder dimensions. The paper does not provide a clear justification for why the top intruder dimensions should be considered more important than the overall rotation of the spectrum.\n2. Reasons behind intruder dimensions. The paper does not provide a clear explanation for why LoRA introduces intruder dimensions. Moreover, while the results convincingly show that LoRA solutions have intruder dimensions, the causal relationship is not fully established. In particular, it is possible that these dimensions are not specific to LoRA, but instead arise from larger overall changes to the weights. The experiments varying the learning rate for LoRA show that higher learning rates lead to more intruder dimensions, and it is plausible that full fine-tuning would show a similar pattern if higher learning rates were used. This raises the question of whether the observed differences between LoRA and full fine-tuning are due to specific properties of the LoRA method, or simply a consequence of differences in optimal hyperparameter ranges. Additional experiments that vary the learning rate in full fine-tuning and analysis of the relationship between the number of intruder dimensions and the magnitude of weight updates would help clarify this point.\n3. Practical importance of intruder dimensions. The analyses in Sections 4 and 5 do not clearly establish a causal link between intruder dimensions and forgetting or generalization. Moreover, they do not provide evidence that the top intruder dimensions contribute more significantly than the rest of the changes in the weights.\n    - The correlation between the number of intruder dimensions and forgetting observed in Section 4 may result from a shared underlying cause rather than a direct causal relationship. Specifically, larger changes in weights during fine-tuning (such as those resulting from higher learning rates in Figure 7) can lead to both a greater number of intruder dimensions and stronger adaptation to the fine-tuning data, which in turn increases forgetting. In this case, even though there is no strong correlation between the number of intruder dimensions and test accuracy on the fine-tuning task, there is likely a correlation with training accuracy/loss.\n    - While Section 4 claims that the number of intruder dimensions correlates with forgetting, Figure 7 shows that full fine-tuning has the fewest intruder dimensions, yet exhibits more forgetting than LoRA with a low learning rate. \n    - In Section 5, the experiments do not convincingly show that intruder dimensions contribute to forgetting more than other changes in weights. The comparison between downscaling intruder dimensions and regular singular vectors effectively compares removing part of the fine-tuning versus removing part of the pre-training. To show that intruder dimensions are specifically responsible for forgetting, a more appropriate comparison would be between downscaling intruder dimensions and downscaling the overall fine-tuning update. This distinction is important, since downscaling the overall fine-tuning update is also known to improve generalization (http://arxiv.org/abs/2109.01903). From the current experiments, it remains unclear whether intruder dimensions contribute more to forgetting than other changes introduced during fine-tuning.\n\nAdditional concerns, comments and questions:\n1. The analysis of the effect of LoRA rank on intruder dimensions could be improved. Figure 4 clearly shows a non-monotonic trend: as the LoRA rank increases, the number of intruder dimensions initially grows and then decreases. While the text notes that intruders decrease and converge toward full-rank behavior at high ranks, it does not discuss the initial increase at low ranks.\n2. The claim that LoRA forgets less than full fine-tuning, even at comparable performance levels, is not particularly strong. The results for LoRA and full fine-tuning in Table 2 differ significantly, and the observed differences in forgetting appear to correlate, at least to some extent, with differences in accuracy. For example, on the QQP task, both accuracy and forgetting differ between high-rank LoRA and full fine-tuning, while on MNLI, the results are almost identical in both respects.\n3. The results in Figure 17 do not seem to fully support the conclusions drawn from Figure 9a. Across different tasks, LoRA exhibits both more and less forgetting, suggesting that the effect may not generalize consistently.\n4. A more detailed analysis of intruder dimensions of different LoRA variants would be beneficial. There appears to be a non-monotonic dependence of the number of intruder dimensions on the epsilon parameter, which likely reflects properties of specific methods used.\n5. Are Figures 1, 2, and 3 based on the same experimental setting? In Figure 2, only one low-epsilon intruder dimension appears among the top 10 singular vectors, whereas Figures 1 and 3 seem to show a higher number of such dimensions.\n6. In Figure 2c, it seems odd to refer to a \"normal\" singular vector with cosine similarity equal to 1, given that Figure 2b clearly shows that the similarity is significantly lower for most vectors under both fine-tuning methods.\n7. It would be helpful to briefly define what alpha represents in the LoRA setup in the Background section, especially since its effect is analyzed later in the paper.\n8. Why do the full fine-tuning results differ Tables 1 and 2?\n9. In Figure 23, the line type for both VeRA experiments is the same.", "was_revised": true, "final_justification": "After careful consideration, I still hold the opinion that **the paper is not ready for publication and requires significant revision**. The rebuttal and discussion mostly addressed my concern about the focus of the paper (Weakness 1). However, the additional results largely confirmed that my concerns regarding the reasons behind the intruder dimensions and their connection to forgetting (Weaknesses 2 and 3) were reasonable. **The main reason for my negative score is the section on forgetting (Weakness 3):** the current claim that intruder dimensions are “special” with respect to forgetting is not convincing and may be incorrect.\n\n**Final comments on the main weaknesses:**\n* **Focus on intruder dimensions (Weakness 1).** After the discussion, I am convinced that intruder dimensions are indeed present in many LoRA experiments and that the paper’s focus is reasonable. However, some experiments still show very different behaviour (Figure 2b). Looking at Figure 4, both types of behaviours appear in practical experiments: clear outlier intruder dimensions result in an up–constant–up pattern (as in low-rank LoRA in the MNLI experiment), while a uniform change as in Figure 2b results in a constant–up pattern (as in high-rank LoRA in the MNLI experiment). I believe the paper should explicitly discuss this distinction.\n* **Reasons behind intruder dimensions (Weakness 2).** Given that the LoRA weight update has a much higher L2 norm, it is not clear whether intruder dimensions are specifically related to LoRA or simply to a larger weight update in general. Based on the current results, in my opinion, the only justified conclusion is that LoRA and full fine-tuning *under optimal hyperparameters* have different update structures. This conclusion is interesting in itself, but the paper would then need to adjust its claims accordingly. Alternatively, the paper could include additional experiments, e.g., using a higher learning rate or longer training for full fine-tuning (or a lower learning rate for LoRA), to confirm whether LoRA and full fine-tuning still produce different weight structures when the weight update norms are similar.\n* **Practical importance of intruder dimensions (Weakness 3).** After the discussion, I remain unconvinced that intruder dimensions are in any way special with respect to forgetting. Based on the previous point, the high correlation between the number of intruder dimensions and forgetting can most likely be explained by the strong correlation between the weight update norm and forgetting. Moreover, the similarity in the effects of downscaling intruder dimensions and downscaling the entire weight update in LoRA (Q4 in the discussion) contradicts the claim that intruder dimensions are particularly special.", "rebuttal": "Thank you for your thoughtful review. We have incorporated your suggestions, including the mistakes you caught like VeRA having the same line twice in Fig. 23, into our updated manuscript. \n\nWhere required, we provide responses to your review below:\n\n> Main weaknesses:\n> The focus of the paper. The paper focuses on intruder dimensions as the main factor distinguishing the spectral properties of LoRA and full fine-tuning, but it is not entirely clear whether this choice captures the core difference.\n\nOur primary objective was to assess **whether different fine-tuning methods converged to functionally equivalent models despite differing parameterizations**. To investigate this, we perform a spectral analysis to characterize how the sequence of transformations each method applied differ.\n\nSince the SVD breaks up the principal components of a matrix, studying how these principal components change (via cosine similarity) is an intuitive way to examine how a weight matrix is changed during fine-tuning. Empirically, we observe that while full fine-tuning, which directly manipulates the weights, makes small adjustments to the magnitude and direction of the singular vectors, LoRA, which uses a low rank product, introduces new singular vectors with low cosine similarity to the existing singular vectors. We provide some intuition for why intruder dimensions are the right framework in Section B.2, where we showed that adding the outer product of a random vector to a weight matrix introduces an intruder dimension. This is analogous to the matrix product of BA in LoRA when rank is 1, and implies that intruder dimensions occur because the B and A vectors are uncorrelated to the columns/rows of the weight matrix W_0.\n\nWe have conducted an extremely detailed examination and would have every reason to report a better measure had it been found. Does the reviewer have a suggestion of measure?\n\n> The paper does not provide a clear explanation for why LoRA introduces intruder dimensions. Moreover, while the results convincingly show that LoRA solutions have intruder dimensions, the causal relationship is not fully established.\n\nIt is important to note that intruder dimensions are an empirical observation of LoRA. However, we provide a few mathematical justifications for their occurrence. **We showed that adding the outer product of a random vector to a weight matrix introduces an intruder dimension. This is analogous to the matrix product of BA in LoRA when rank is 1, and implies that intruder dimensions occur because the B and A vectors are uncorrelated to the columns/rows of the weight matrix W_0 (Section B.2).** Further, we showed empirically that just training the B matrix, while freezing the A matrix with all singular values of 1, eliminates the amplification of the singular values because of the matrix product and therefore reduces the number of high ranking intruder dimensions. This suggests that this multiplicative property may cause new singular vectors to have large singular value. These experiments provide evidence towards why LoRA causes intruder dimensions to be introduced.\n\n> it is possible that these dimensions are not specific to LoRA, but instead arise from larger overall changes to the weights ... The experiments varying the learning rate for LoRA show that higher learning rates lead to more intruder dimensions, and it is plausible that full fine-tuning would show a similar pattern if higher learning rates were used. This raises the question of whether the observed differences between LoRA and full fine-tuning are due to specific properties of the LoRA method, or simply a consequence of differences in optimal hyperparameter ranges.\n\nThe reviewer is absolutely correct to be curious about this and we indeed investigated this. In our investigation, we used standard learning rates for both full fine-tuning and LoRA.\nWhen we conduct a large learning rate sweep for both methods, we observe that increasing learning rate significantly over the default setting leads to training instabilities and divergence. Decreasing learning rate has difficulty converging to similar performance, even with more training steps. Because of this, the resulting models perform significantly worse and therefore cannot be used in our analysis. **For models that do converge to similar, near optimal performance (Fig. 7), we observe that full fine-tuning contains no intruder dimensions while LoRA contains many.** \n\n> Practical importance of intruder dimensions. The analyses in Sections 4 and 5 do not clearly establish a causal link between intruder dimensions and forgetting or generalization.\n\nIn Section 4, we find a strong correlation between intruder dimensions and forgetting. The reviewer is absolutely right to point out that this could be the result of a third variable that is causing both to increase. **To show that this is not the case, in section 5 we intervene on the intruder dimensions.** In it, we scale down the singular values of the intruder dimensions, which has the effect of reducing their contribution on the fine-tuned weights. By performing this causal experiment, we find that there is little change in test accuracy but a large impact on forgetting. One example of this (reported in the main text in lines 292-293) shows that using $\\lambda=0.7$ on our model trained on QQP leads to *no change* in test accuracy but a 33.2% reduction in forgetting.\n\nThese findings lead to the following conclusion: intervening on intruder dimensions and scaling them down results in a big drop in forgetting but little change in test accuracy, showing that these intruder dimensions, and in particular their magnitude (singular vector), causes a large amount of the forgetting in LoRA models.\n\nWe do not claim that intruder dimensions are not the *only* possible cause of forgetting. Any sort of change to the pre-trained weights should be expected to impact a models base language modelling ability. Indeed, we observe that full fine-tuning makes small adjustments to the direction and magnitude of the pre-trained singular vectors. This still changes the fine-tuned weights, and we should expect forgetting to occur. Our causal experiment scaling intruder dimensions is specific to LoRA. \n\nIntruder dimensions and section 5 explain the structural basis of forgetting with LoRA'd models (and variants of LoRA). However, a functional explanation of forgetting in full-finetuned models can be different and an interesting avenue for future study.\n\n> Additional concerns, comments and questions:\nThe claim that LoRA forgets less than full fine-tuning, even at comparable performance levels, is not particularly strong. The results for LoRA and full fine-tuning in Table 2 differ significantly, and the observed differences in forgetting appear to correlate, at least to some extent, with differences in accuracy. For example, on the QQP task, both accuracy and forgetting differ between high-rank LoRA and full fine-tuning, while on MNLI, the results are almost identical in both respects.\n\nOur conclusion that \"LoRA forgets less than full fine-tuning, even at comparable performance levels\" stems from our results in Table 2 and Fig. 6. Table 2 contains the test accuracies of all of our models. In it, we see models trained to approximately the same accuracy. We are not sure why the reviewer claims that the results for LoRA and full fine-tuning in Table 2 differ significantly. For example, in Table 2, all our models fine-tuned on MNLI perform within 0.5% of each other. Looking horizontally, comparing full fine-tuning and LoRA r=16, without loss of generality, we see similar performance, with LoRA sometimes outperforming full fine-tuning and vice versa. Further, when correlating test accuracy and forgetting within datasets for Table 2 (as indicated by the reviewer), we get no statistically significant result (test: Spearman's rank-order correlation, p-value>0.33 for each.). **Therefore, this means that observed differences in forgetting do not correlate with differences in accuracy.**\n\nWhen looking at Fig. 6b, we see that full fine-tuning always forgets more than LoRA. This leads us to conclude that LoRA forgets less than full fine-tuning, even at comparable performance levels, which exends the findings of [1]. We hope this clarifies the reviewers concern.\n\n[1] - https://arxiv.org/pdf/2405.09673\n\n\n> A more detailed analysis of intruder dimensions of different LoRA variants would be beneficial. There appears to be a non-monotonic dependence of the number of intruder dimensions on the epsilon parameter, which likely reflects properties of specific methods used.\n\nWe provide a preliminary study to show that our findings scale to other variants in Section P. We find that LoRA variants (LoRA+, VeRA, AdaLoRA, PiSSA) all have intruder dimensions, showing that these methods are not adequate for preventing intruder dimensions. We emphasize that our study of LoRA variants is orthogonal to our main study, and that it is difficult to conduct every detailed experiment we have on all LoRA variants. While we save an in depth analysis for future work, we have no reason to suspect these will impact the claims we make in this paper."}, {"review_id": "zAzkhQXKZz", "reviewer": "Reviewer_JRuL", "summary": "This paper thoroughly investigates the structural differences between weight updates from LoRA and full-finetuning through the lens of SVD decomposition, and introduces a new concept *intruder dimension* to quantify these differences. Extensive empirical stuides and discussions are provided to better exemplify the properties and behaviors of this new concept.", "questions": "See Weaknesses.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written and clearly structured, with a logical and straightforward presentation.\n2. The concept of *intruder dimension* is well-defined and intuitively aligns with the mathematical underpinnings of representation learning, particularly in the context of SVD, where directions with higher variance tend to carry more informative content.\n\n**Weaknesses**\n\n1. The core concern is that the paper primarily presents extensive empirical comparisons between LoRA and full fine-tuning to illustrate the emergence and impact of *intruder dimensions*, without proposing new methods or actionable improvements based on these insights. As per the NeurIPS regulation, it may be better suited for the **Datasets & Benchmarks track** rather than the main conference.\n\n2. Despite the extensive empirical evidence for the impact of *intruder dimension*, the theoretical explanation for why they appear in fine-tuning LoRA is still lacking. Mathematically, it is expected that constrained updates (e.g., via LoRA) tend to concentrate on directions associated with the largest singular values, while full fine-tuning distributes updates more evenly with sufficiently large update space. Without rigorous theoretical support, the introduction of the *intruder dimension* appears to be a restatement of this known behavior within the specific framing of LoRA.\n\n3. I've several concerns regarding the empirical analyses:\n    1. Regarding the correlation between the number of *intruder dimension* and catastrophic forgetting (Lines 256-263), the experiments using a larger learning rate show that LoRA is not well optimized, as evidenced by a significant performance gap. In such cases, the increase in intruder dimensions could be a byproduct of suboptimal training rather than an inherent issue. For a well-optimized LoRA model (e.g., with lr=1e-4), the observation in [1] (LoRA learns less but forgets less) still holds. Thus, conclusions based on poorly trained models are less convincing.\n    2. In Appendix H.2, the observation that performance drops when learned knowledge is perturbed and LoRA's preserved knowledge is exaggerated is unsurprising. This behavior is consistent with general deep learning dynamics and does not convincingly support the claim that *intruder dimensions* degrade OOD performance. The argument feels more like an expected consequence of disrupting well-internalized features.\n    3. In Appendix I, was the hyperparameter search of learning rate conducted to ensure that LoRA with different $\\alpha$ converged to similar test accuracy? If not, it's trivial that improper/insufficient learning naturally leads to low rank solutions.\n\n[1] Dan, Biderman, et al., \"LoRA Learns Less and Forgets Less,\" TMLR 2024.", "was_revised": true, "final_justification": "Overall Assessment:\n1. I thank the authors for their clarifications, and I find this submission to be in alignment with the guidelines of the main conference.\n2. As noted by reviewers BfKg and GjPE, the exploration of an “intruder dimension” for LoRA seems to have limited practicality in real-world deployments, considering that the primary value of PEFT lies in enabling the rapid application of LLMs in industrial scenarios.\n3. I appreciate the authors’ substantial effort and the detailed clarifications provided.\n\nAfter considering the discussions throughout the rebuttal phase, I view this paper as being around the borderline. I have accordingly adjusted my rating to 4, while noting that this does not imply a definitive decision regarding acceptance.", "rebuttal": "We thank the reviewer for their thoughtful review. We provide responses to their review below.\n\n> The core concern is that the paper primarily presents extensive empirical comparisons between LoRA and full fine-tuning to illustrate the emergence and impact of intruder dimensions, without proposing new methods or actionable improvements based on these insights. As per the NeurIPS regulation, it may be better suited for the Datasets & Benchmarks track rather than the main conference.\n\nWe are confused by your suggestion to change tracks. First, under the main track call for papers, the website states that NeurIPS \"encourage in-depth analysis of existing methods that provide new insights in terms of their limitations or behaviour beyond the scope of the original work\". Moreover, this paper provides neither a dataset nor a benchmark.\nFurther, we do indeed provide actionable insights: we show several ways that the number of intruder dimensions can be reduced, including with certain initializations (Fig. 10), lower learning rates (Fig. 7), and setting $\\alpha=2r$ (Section N). We even identify a simple post-training intervention motivated by section 5: scaling down the magnitude of intruder dimensions can reduce the forgetting a model has while retaining nearly identical adaptation performance. These are concrete prescriptions that can be used to train better LoRA models that exhibit reduced forgetting and better out-of-distribution generalization.\n\n> Despite the extensive empirical evidence for the impact of intruder dimension, the theoretical explanation for why they appear in fine-tuning LoRA is still lacking. \n\nWe do provide a few mathematical justifications for the occurrence of intruder dimensions in Section B.1. We showed that adding the outer product of a random vector to a weight matrix introduces an intruder dimension (Section B.1). This is analogous to the matrix product of BA in LoRA when rank is 1, and implies that intruder dimensions occur because the B and A vectors are uncorrelated to the columns/rows of the weight matrix W_0. Further, we showed empirically that just training the B matrix, while freezing the A matrix with all singular values of 1, reduces the amplification of the singular values because of the matrix product and therefore reduces the number of high ranking intruder dimensions, aligning with our mathematical intuition.\n\n>Mathematically, it is expected that constrained updates (e.g., via LoRA) tend to concentrate on directions associated with the largest singular values, while full fine-tuning distributes updates more evenly with sufficiently large update space. Without rigorous theoretical support, the introduction of the intruder dimension appears to be a restatement of this known behavior within the specific framing of LoRA.\n\nWhat you describe as expected behavior actually doesn’t align with what our observations. Rather than changes occurring primarily along the existing high-ranking singular vectors, what happens is that a few new high-ranking singular vectors emerge, while the original ones remain largely unchanged or only slightly shifted. This suggests that the dynamics introduced by constrained updates like LoRA are more nuanced than simply concentrating on pre-existing dominant directions like you describe. Whereas full fine-tuning distributes updates across all singular vectors.\n\nIt is important to note that, before our work, it was not expected that LoRA would update a weight matrix in a **structurally different** manner than full fine-tuning. Rather, It was expected that LoRA approximated full fine-tuning (https://arxiv.org/pdf/2106.09685), which would result in the LoRA looking like an approximation of the fully fine-tuned weight updates that is concentrated on the top existing singular values, like you mention.\n\n> Regarding the correlation between the number of intruder dimension and catastrophic forgetting (Lines 256-263), the experiments using a larger learning rate show that LoRA is not well optimized, as evidenced by a significant performance gap. In such cases, the increase in intruder dimensions could be a byproduct of suboptimal training rather than an inherent issue. For a well-optimized LoRA model (e.g., with lr=1e-4), the observation in [1] (LoRA learns less but forgets less) still holds. Thus, conclusions based on poorly trained models are less convincing.\n\nWe are concerned the reviewer is misinterpretting this section.\nAll the models are equally well trained (as measured by equal validation and train loss) on the adaptation distribution--which are typical measures used when finetuning models. Could you please explain which measure would make one of the models sub-optimal over the other? \n\nIn the section you refer to, we sweep learning rates to see its effect on optimization. We find that we get *similar* test loss but significantly different forgetting profiles. For example, in Fig. 7, let us examine LoRA with lr=1e-4 and 2e-4. We see that the models have trained to within 0.5% test accuracy performance across epochs (middle plot). However, we see that 2e-4 consistently forgets a much more than 1e-4 when examining their corresponding pseudo loss. These are the datapoints that are used to indicate that intruder dimensions correlate with forgetting. We infact support the conclusion of (\"LoRA learns less but forgets less\") and extend it in this paper (lines 235-247).\n\nOur main conclusions of our paper are all based on well trained models. We welcome the reviewer to follow up and provide clarification if we have misunderstood their concern. If there is another criterion by which one model is considered suboptimal and is typically monitored at finetuning, we would appreciate further clarification.\n \n\n>In Appendix H.2, the observation that performance drops when learned knowledge is perturbed and LoRA's preserved knowledge is exaggerated is unsurprising. This behavior is consistent with general deep learning dynamics and does not convincingly support the claim that intruder dimensions degrade OOD performance. The argument feels more like an expected consequence of disrupting well-internalized features.\n\nYour suggested intuition is actually opposite to our findings, making this a surprising result. Our finding is not that \"performance drops when learned knowledge is perturbed\", but rather that it is *barely* changed when intruder dimensions are scaled down while forgetting has decreased *significantly*. We provide a key passage in our main text (lines 290-295) here:\n\n\"In one example for LLaMA2-7B fine-tuned on MetaMath with LoRA r = 256, we observe that scaling the top intruder dimension in each matrix with $\\lambda = 0.3$ leads to a 0.1% drop in test accuracy and a 33.3% drop in the forgetting induced by fine-tuning. In another for RoBERTa-base fine-tuned on QQP, using $\\lambda = 0.7$ leads to equivalent in test accuracy and a 33.2% reduction in the forgetting induced by fine-tuning. In certain scenarios, we even see test accuracy improve along with a drop in forgetting. If we instead increase their contribution ($\\lambda > 1$), we observe more forgetting.\"\n\nThis is an unexpected result. We welcome the reviewer to follow up and provide clarification if we have misunderstood their concern.\n\n\n>In Appendix I, was the hyperparameter search of learning rate conducted to ensure that LoRA with different alpha converged to similar test accuracy? If not, it's trivial that improper/insufficient learning naturally leads to low rank solutions.\n\nYes. We follow standard machine learning practice while conduct learning rate sweeps and ensure that the different models compared converged to similar test accuracy as shown in Table 2. In Appendix I, all models train to similar test loss and accuracy, making our investigation fair across ranks."}, {"review_id": "fVy0M3YRod", "reviewer": "Reviewer_9SVt", "summary": "This work introduces the concept of intruder dimensions that arise during fine-tuning of language models via low rank adaptation (LoRA).\nIntruder dimensions are directions in the parameter space that disrupt the original structure of pre-trained weights and strongly correlates with forgetting of pre-trained knowledge.\nFurthermore, the authors point out that fine-tuning via LoRA not always results in less forgetting and strongly depends on the number of introduced intruder dimensions.\nVia causal intervention, the authors verify that by scaling down singular values associated with intruder dimensions leads to recovering pre-trained knowledge while maintaining downstream performance.", "questions": "See weaknesses.", "limitations": "Some limitations are in the appendix, but they could be a bit more elaborate.\nTo accomodate those in the main body of the paper, I suggest moving Algorithm 1 in the appendix, it does not add too much value.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well written and most of the claims are well supported.\n\nDefinition of a new concept that results from fine-tuning with low-rank adaptation, which is very relevant.\n\nThe experiments are well designed with interesting results and takeaways.\n\nThe key takeawas are based on multiple rigorous experiments, not just single seeds/experiments.\n\n**Weaknesses**\n\nI am generally very positive about this work, the following weaknesses are minor issues.\n\n**Unsupported claims**\n\nThere are a few claims that are not entirely supported:\n- line 166: \"LoRA introduces new singular vectors that have a large contribution to the norm of the\nupdated parameter matrix.\" - As far as I can see there is no support for a change in norm, only in directions because of using cosine similarity.\n- line 210: \"LoRA consistently has more intruder dimensions than full fine-tuning\" - not entirely correct, see Appendix J Figure 15, LoRA r=64 has less intruder dimensions than full fine-tuning\n- line 244: \"This extends the finding that LoRA learns less \" - I assume this is a typo and should be \"forgets less\" rather than \"learns less\", as there is no support for the learning less argument as far as I can see\n\n**Different distance measures**\n\nCurrently, the authors only consider cosine similarity as distance measures between singular vectors.\nIt would be interesting if intruder dimensions could be identified via different distance measures, or whether it is really only the difference in directions that enables identifying them.\n\n**Potential additional experiments**\n\nIt is very interesting that initialization has a huge effect on intruder dimensions (Figure 23).\nIt would be interesting how the number of intruder dimensions vary for different initialization schemes, specifically for data-driven initialization, e.g. [1,2,3].\n\n[1] Wang et al., LoRA-GA: Low-Rank Adaptation with Gradient Approximation, NeurIPS 2024\n\n[2] Paischer et al., One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation, NeurIPS 2024 ENLSP Workshop\n\n[3] Yang et al., CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS 2024\n\nAnother very interesting point would be showing where exactly the intruder dimensions appear in.\nAre they only apparent in certain projection matrices (Q/K/V, etc.) and layers and can they be related with implicit biases as introduced in [4]?\n\n[4] Sun et al., Massive Activations in Large Language Models, COLM 2024\n\nFinally, it would be interesting how the number of intruder dimensions changes with larger models.\n\n**Presentation**\n\nGenerally, I recommend increasing the fontsize of the plots, they are partly a bit hard too read without zooming in a lot.\nIt would be cool to have a zoomed in version of the top left part of Figure 1, right, to better identify intruder dimensions (same for Figure 2b).\nThere are typos in line 188 and 202.", "was_revised": true, "final_justification": "My initial rating of the paper was already positive and all my raised points have been addressed during the rebuttal.\nI see this work having an impact in understanding and improving LoRA-style fine-tuning, therefore I reside with my original rating.", "rebuttal": "We thank you for your thoughtful review and are grateful for your positivity about our work. We have incorporated the suggestions you have requested, including typos, presentation changes, and updating claims, into the updated manuscript. We provide responses and more context where required below:\n\n> There are a few claims that are not entirely supported:\nline 166: \"LoRA introduces new singular vectors that have a large contribution to the norm of the updated parameter matrix.\" - As far as I can see there is no support for a change in norm, only in directions because of using cosine similarity.\n\nThank you for pointing this out. While Figure 5 shows an increase in the singular value corresponding to the intruder dimension—suggesting a potential increase in the matrix norm—we currently don't report the norm itself. We will include this statistic in future revisions to more concretely support the claim.\n\n> Different distance measures\nCurrently, the authors only consider cosine similarity as distance measures between singular vectors. It would be interesting if intruder dimensions could be identified via different distance measures, or whether it is really only the difference in directions that enables identifying them.\n\nThank you for the suggestion; it's a valuable point. While we focus on cosine similarity, we don’t claim it's the only viable measure. We chose it because, in the context of SVD, comparing singular vectors via cosine similarity offers a natural and interpretable way to track changes in the principal directions of the weight matrix. Since these directions define how the matrix transforms input, shifts in them are meaningful. That said, exploring alternative distance measures is an interesting direction for future work and could offer complementary insights.\n\n> Another very interesting point would be showing where exactly the intruder dimensions appear in. Are they only apparent in certain projection matrices (Q/K/V, etc.) and layers and can they be related with implicit biases as introduced in [4]?\n\nAcross our experiments, intruders appear in all weight matrices, with no systematic difference in the number of intruder dimensions between MLP and attention matrices. We will add in a section outlining these details to the appendix. \n\n> It would be interesting how the number of intruder dimensions changes with larger models.\n\nWe thank the reviewer for there extensive suggestions of additional experiments. Due to the limited length of the rebuttal, we will leave these experiments to future work to extend this work."}]

Author rebuttal:

Meta-review: This paper introduces the concept of intruder dimensions in LoRA fine-tuning, showing their strong correlation with forgetting and demonstrating, via causal interventions, that mitigating their effect can recover pre-trained knowledge without sacrificing downstream performance. Reviewers agreed the paper is novel, clearly written, and supported by extensive experiments across models, datasets, and hyperparameters.

While minor issues were raised (e.g., phrasing of claims, reliance on cosine similarity as a distance measure), these do not undermine the core contributions. The rebuttal effectively addressed concerns about hyperparameter bias, causal importance of intruders, and whether LoRA differences are simply basis rotations. Overall, the paper provides valuable insights into the spectral properties of LoRA and practical strategies for mitigating forgetting.
Decision: Accept (poster)