Title: LoRA vs Full Fine-tuning: An Illusion of Equivalence
Venue: NeurIPS 2025
Accepted: yes
Keywords: Parameter Efficient Fine-tuning (PEFT)

Low Rank Adaptation (LoRA)

LLMs

Transformers
Abstract: Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, \emph{are their learned solutions really equivalent?} We study how LoRA and full-finetuning change pre-trained models by analyzing the model's weight matrices through the lens of their spectral properties. We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure: weight matrices trained with LoRA have new, high-ranking singular vectors, which we call \emph{intruder dimensions}, while those trained with full fine-tuning do not. Further, we extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension -- by causally intervening on the intruder dimensions by changing their associated singular values post-fine-tuning, we show that they cause forgetting. Moreover, scaling them down significantly improves modeling of the pre-training distribution with a minimal drop in downstream task performance. Given this, we should expect accumulating intruder dimensions to be harmful and lead to more forgetting. This will be amplified during continual learning because of sequentially fine-tuning, and we show that LoRA models do accumulate intruder dimensions here tend to perform worse in this setting, emphasizing the practicality of our findings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces the concept of intruder dimensions that arise during fine-tuning of language models via low rank adaptation (LoRA).\nIntruder dimensions are directions in the parameter space that disrupt the original structure of pre-trained weights and strongly correlates with forgetting of pre-trained knowledge.\nFurthermore, the authors point out that fine-tuning via LoRA not always results in less forgetting and strongly depends on the number of introduced intruder dimensions.\nVia causal intervention, the authors verify that by scaling down singular values associated with intruder dimensions leads to recovering pre-trained knowledge while maintaining downstream performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper is well written and most of the claims are well supported.\n\nDefinition of a new concept that results from fine-tuning with low-rank adaptation, which is very relevant.\n\nThe experiments are well designed with interesting results and takeaways.\n\nThe key takeawas are based on multiple rigorous experiments, not just single seeds/experiments.\n\n**Weaknesses**\n\nI am generally very positive about this work, the following weaknesses are minor issues.\n\n**Unsupported claims**\n\nThere are a few claims that are not entirely supported:\n- line 166: \"LoRA introduces new singular vectors that have a large contribution to the norm of the\nupdated parameter matrix.\" - As far as I can see there is no support for a change in norm, only in directions because of using cosine similarity.\n- line 210: \"LoRA consistently has more intruder dimensions than full fine-tuning\" - not entirely correct, see Appendix J Figure 15, LoRA r=64 has less intruder dimensions than full fine-tuning\n- line 244: \"This extends the finding that LoRA learns less \" - I assume this is a typo and should be \"forgets less\" rather than \"learns less\", as there is no support for the learning less argument as far as I can see\n\n**Different distance measures**\n\nCurrently, the authors only consider cosine similarity as distance measures between singular vectors.\nIt would be interesting if intruder dimensions could be identified via different distance measures, or whether it is really only the difference in directions that enables identifying them.\n\n**Potential additional experiments**\n\nIt is very interesting that initialization has a huge effect on intruder dimensions (Figure 23).\nIt would be interesting how the number of intruder dimensions vary for different initialization schemes, specifically for data-driven initialization, e.g. [1,2,3].\n\n[1] Wang et al., LoRA-GA: Low-Rank Adaptation with Gradient Approximation, NeurIPS 2024\n\n[2] Paischer et al., One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation, NeurIPS 2024 ENLSP Workshop\n\n[3] Yang et al., CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS 2024\n\nAnother very interesting point would be showing where exactly the intruder dimensions appear in.\nAre they only apparent in certain projection matrices (Q/K/V, etc.) and layers and can they be related with implicit biases as introduced in [4]?\n\n[4] Sun et al., Massive Activations in Large Language Models, COLM 2024\n\nFinally, it would be interesting how the number of intruder dimensions changes with larger models.\n\n**Presentation**\n\nGenerally, I recommend increasing the fontsize of the plots, they are partly a bit hard too read without zooming in a lot.\nIt would be cool to have a zoomed in version of the top left part of Figure 1, right, to better identify intruder dimensions (same for Figure 2b).\nThere are typos in line 188 and 202.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSome limitations are in the appendix, but they could be a bit more elaborate.\nTo accomodate those in the main body of the paper, I suggest moving Algorithm 1 in the appendix, it does not add too much value.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper thoroughly investigates the structural differences between weight updates from LoRA and full-finetuning through the lens of SVD decomposition, and introduces a new concept *intruder dimension* to quantify these differences. Extensive empirical stuides and discussions are provided to better exemplify the properties and behaviors of this new concept.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well-written and clearly structured, with a logical and straightforward presentation.\n2. The concept of *intruder dimension* is well-defined and intuitively aligns with the mathematical underpinnings of representation learning, particularly in the context of SVD, where directions with higher variance tend to carry more informative content.\n\n**Weaknesses**\n\n1. The core concern is that the paper primarily presents extensive empirical comparisons between LoRA and full fine-tuning to illustrate the emergence and impact of *intruder dimensions*, without proposing new methods or actionable improvements based on these insights. As per the NeurIPS regulation, it may be better suited for the **Datasets & Benchmarks track** rather than the main conference.\n\n2. Despite the extensive empirical evidence for the impact of *intruder dimension*, the theoretical explanation for why they appear in fine-tuning LoRA is still lacking. Mathematically, it is expected that constrained updates (e.g., via LoRA) tend to concentrate on directions associated with the largest singular values, while full fine-tuning distributes updates more evenly with sufficiently large update space. Without rigorous theoretical support, the introduction of the *intruder dimension* appears to be a restatement of this known behavior within the specific framing of LoRA.\n\n3. I've several concerns regarding the empirical analyses:\n    1. Regarding the correlation between the number of *intruder dimension* and catastrophic forgetting (Lines 256-263), the experiments using a larger learning rate show that LoRA is not well optimized, as evidenced by a significant performance gap. In such cases, the increase in intruder dimensions could be a byproduct of suboptimal training rather than an inherent issue. For a well-optimized LoRA model (e.g., with lr=1e-4), the observation in [1] (LoRA learns less but forgets less) still holds. Thus, conclusions based on poorly trained models are less convincing.\n    2. In Appendix H.2, the observation that performance drops when learned knowledge is perturbed and LoRA's preserved knowledge is exaggerated is unsurprising. This behavior is consistent with general deep learning dynamics and does not convincingly support the claim that *intruder dimensions* degrade OOD performance. The argument feels more like an expected consequence of disrupting well-internalized features.\n    3. In Appendix I, was the hyperparameter search of learning rate conducted to ensure that LoRA with different $\\alpha$ converged to similar test accuracy? If not, it's trivial that improper/insufficient learning naturally leads to low rank solutions.\n\n[1] Dan, Biderman, et al., \"LoRA Learns Less and Forgets Less,\" TMLR 2024.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the differences between LoRA and full fine-tuning of LLMs by examining the spectral properties of the resulting weight matrices. Specifically, the authors compare the singular vectors of the pre-trained and fine-tuned weights and find that, while full fine-tuning largely preserves the original spectrum, LoRA introduces intruder dimensions—singular vectors with large singular values that have low similarity to any singular vector from the pre-trained model. The authors conduct thorough experiments to study how various hyperparameters, such as LoRA rank and alpha, learning rate, the parameters used to compute the number of intruder dimensions, and even variations of the LoRA method, affect these intruder dimensions. They then analyze the forgetting behavior of LoRA in both fine-tuning and continual learning settings, and argue that the intruder dimensions contribute to it.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper addresses an important problem: understanding the properties of solutions obtained with LoRA is highly relevant, given its widespread use.\n2. To the best of my knowledge, the analysis of spectral differences between LoRA and full fine-tuning is novel, and the results are nontrivial and interesting.\n3. The experimental setup is strong. The use of different architectures and datasets supports the generality of the intruder dimension phenomenon.\n4. The paper is thorough and includes a wide range of additional experiments and ablations.\n5. The paper is clearly written and easy to follow.\n\nMain weaknesses:\n1. The focus of the paper. The paper focuses on intruder dimensions as the main factor distinguishing the spectral properties of LoRA and full fine-tuning, but it is not entirely clear whether this choice captures the core difference. According to Figure 2b, the difference appears to be more uniform: all singular vectors in the LoRA solution tend to have lower cosine similarity, and there is no clear small subset of outlier directions with significantly lower similarity. This suggests that LoRA broadly rotates the basis rather than introducing a few distinct intruder dimensions. The paper does not provide a clear justification for why the top intruder dimensions should be considered more important than the overall rotation of the spectrum.\n2. Reasons behind intruder dimensions. The paper does not provide a clear explanation for why LoRA introduces intruder dimensions. Moreover, while the results convincingly show that LoRA solutions have intruder dimensions, the causal relationship is not fully established. In particular, it is possible that these dimensions are not specific to LoRA, but instead arise from larger overall changes to the weights. The experiments varying the learning rate for LoRA show that higher learning rates lead to more intruder dimensions, and it is plausible that full fine-tuning would show a similar pattern if higher learning rates were used. This raises the question of whether the observed differences between LoRA and full fine-tuning are due to specific properties of the LoRA method, or simply a consequence of differences in optimal hyperparameter ranges. Additional experiments that vary the learning rate in full fine-tuning and analysis of the relationship between the number of intruder dimensions and the magnitude of weight updates would help clarify this point.\n3. Practical importance of intruder dimensions. The analyses in Sections 4 and 5 do not clearly establish a causal link between intruder dimensions and forgetting or generalization. Moreover, they do not provide evidence that the top intruder dimensions contribute more significantly than the rest of the changes in the weights.\n    - The correlation between the number of intruder dimensions and forgetting observed in Section 4 may result from a shared underlying cause rather than a direct causal relationship. Specifically, larger changes in weights during fine-tuning (such as those resulting from higher learning rates in Figure 7) can lead to both a greater number of intruder dimensions and stronger adaptation to the fine-tuning data, which in turn increases forgetting. In this case, even though there is no strong correlation between the number of intruder dimensions and test accuracy on the fine-tuning task, there is likely a correlation with training accuracy/loss.\n    - While Section 4 claims that the number of intruder dimensions correlates with forgetting, Figure 7 shows that full fine-tuning has the fewest intruder dimensions, yet exhibits more forgetting than LoRA with a low learning rate. \n    - In Section 5, the experiments do not convincingly show that intruder dimensions contribute to forgetting more than other changes in weights. The comparison between downscaling intruder dimensions and regular singular vectors effectively compares removing part of the fine-tuning versus removing part of the pre-training. To show that intruder dimensions are specifically responsible for forgetting, a more appropriate comparison would be between downscaling intruder dimensions and downscaling the overall fine-tuning update. This distinction is important, since downscaling the overall fine-tuning update is also known to improve generalization (http://arxiv.org/abs/2109.01903). From the current experiments, it remains unclear whether intruder dimensions contribute more to forgetting than other changes introduced during fine-tuning.\n\nAdditional concerns, comments and questions:\n1. The analysis of the effect of LoRA rank on intruder dimensions could be improved. Figure 4 clearly shows a non-monotonic trend: as the LoRA rank increases, the number of intruder dimensions initially grows and then decreases. While the text notes that intruders decrease and converge toward full-rank behavior at high ranks, it does not discuss the initial increase at low ranks.\n2. The claim that LoRA forgets less than full fine-tuning, even at comparable performance levels, is not particularly strong. The results for LoRA and full fine-tuning in Table 2 differ significantly, and the observed differences in forgetting appear to correlate, at least to some extent, with differences in accuracy. For example, on the QQP task, both accuracy and forgetting differ between high-rank LoRA and full fine-tuning, while on MNLI, the results are almost identical in both respects.\n3. The results in Figure 17 do not seem to fully support the conclusions drawn from Figure 9a. Across different tasks, LoRA exhibits both more and less forgetting, suggesting that the effect may not generalize consistently.\n4. A more detailed analysis of intruder dimensions of different LoRA variants would be beneficial. There appears to be a non-monotonic dependence of the number of intruder dimensions on the epsilon parameter, which likely reflects properties of specific methods used.\n5. Are Figures 1, 2, and 3 based on the same experimental setting? In Figure 2, only one low-epsilon intruder dimension appears among the top 10 singular vectors, whereas Figures 1 and 3 seem to show a higher number of such dimensions.\n6. In Figure 2c, it seems odd to refer to a \"normal\" singular vector with cosine similarity equal to 1, given that Figure 2b clearly shows that the similarity is significantly lower for most vectors under both fine-tuning methods.\n7. It would be helpful to briefly define what alpha represents in the LoRA setup in the Background section, especially since its effect is analyzed later in the paper.\n8. Why do the full fine-tuning results differ Tables 1 and 2?\n9. In Figure 23, the line type for both VeRA experiments is the same.\n\nQUESTIONS:\nAll questions and concerns are detailed in the Strengths and Weaknesses section. The main weaknesses 1–3 are the most critical and will have the greatest impact on my final evaluation after the rebuttal.\n\nLIMITATIONS:\nI believe the limitations section should provide a more thorough discussion noting that the paper focuses only on one aspect of the spectral differences between LoRA and full fine-tuning, the intruder dimensions, while leaving aside the effect of the overall rotation of the spectrum and the changes in singular values.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the difference between full fine-tuning and LoRA through spectral analysis of weight matrices. The authors identify intruder dimensions—high-ranking singular vectors in tuned weights that are dissimilar to any pre-trained directions—and show they are a key cause of forgetting. Scaling down these components reduces forgetting without harming downstream too much performance. The accumulation of intruder dimensions is also shown to hurt LoRA in continual learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The authors provide a novel perspective on the difference between LoRA and full fine-tuning. The concept of intruder dimensions reveals a concrete structural difference between full-finetuning and LoRA.\n\n- The link between intruder dimensions and forgetting is insightful. It offers both theoretical understanding and potential avenues for mitigation.\n\n\nWeaknesses\n\n- Logical mismatch in Section 5. Section 4 convincingly correlates the *number* of intruder dimensions with forgetting, yet Section 5 only experiments with scaling down the *magnitude* of the top intruder dimension. To solidify the argument, the authors should also manipulate the *count* of intruder dimensions and measure its effect on forgetting.\n\n- Limited practical guidance. The paper does not provide guidance on how to prevent the emergence of intruder dimensions during training, which is crucial for improving real-world applicability. As it stands, intruder dimensions appear to be primarily useful as a post-hoc model selection criterion. However, in practice, comparing two models based on their performance on pre-training data may be more straightforward.\n\nQUESTIONS:\n- The paper concludes that intruder dimensions cause forgetting. However, full-tuning with few intruder dimensions,  still exhibits more forgetting than LoRA with an appropriately chose $\\alpha$. How do we understand this based on findings in the paper?\n\n- In the continual learning setup, are the similarity matrices in Figure 9 all computed with the original model? I am wondering can we observe instruder dimensions over a model already tuned on another task(say do weights after task 3 have  instruder dimension over weights after task 2)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}