Title: The Primacy of Magnitude in Low-Rank Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LoRA; Low-Rank Adaptation; Deep Learning; Large Pre-trained Models
Abstract: Low-Rank Adaptation (LoRA) offers a parameter-efficient paradigm for tuning large models. While recent spectral initialization methods improve convergence and performance over the naive “Noise \& Zeros” scheme, their extra computational and storage overhead undermines efficiency. In this paper, we establish update magnitude as the fundamental driver of LoRA performance and propose LoRAM, a magnitude-driven “Basis \& Basis” initialization scheme that matches spectral methods without their inefficiencies. Our key contributions are threefold: (i) Magnitude of weight updates determines convergence.
We prove low-rank structures intrinsically bound update magnitudes, unifying hyperparameter tuning in learning rate, scaling factor, and initialization as mechanisms to optimize magnitude regulation. 
(ii) Spectral initialization succeeds via magnitude amplification.
 We demystify that the presumed knowledge-driven benefit of spectral component essentially arises from the boost in the weight update magnitude.
(iii) A novel and compact initialization strategy, LoRAM, scales deterministic orthogonal bases using pretrained weight magnitudes to simulate spectral gains. Extensive experiments show that LoRAM serves as a strong baseline, retaining the full efficiency of LoRA while matching or outperforming spectral initialization across benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a unifying principle for understanding and improving Low-Rank Adaptation (LoRA), arguing that the magnitude of weight updates is the primary driver of its performance. The authors present a theoretical framework that connects learning rate, scaling factor, and initialization as different means to the same end: regulating update magnitude. Using this, the authors demystify the success of recent spectral initialization methods, positing that their effectiveness comes from amplifying update magnitudes, not from a more abstract preservation of pretrained knowledge.\n\nBuilding on this insight, the paper proposes LoRAM that initializes the LoRA matrices using deterministic orthogonal bases derived from the Discrete Sine Transform and scales them by a factor carefully calculated to mimic the magnitude gains of spectral approaches. Through extensive experiments on a diverse set of language, coding, and vision-language tasks, the authors demonstrate that LoRAM consistently matches or outperforms more complex initialization schemes while retaining the full efficiency and plug-and-play simplicity of the original LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper successfully challenges the findings of previous research by proposing an alternative explanation for the improved performance of spectral decomposition PEFT algorithms.\n\nLoRAM displays comparable or slightly superior performance to spectral PEFT algorithms using a simpler strategy.\n\nThe findings are mathematically supported and insights into the training dynamics are presented in Figures 2 and 3.\n\nLoRAM's performance is evaluated across a range of tasks to comfort as to it efficiency\n\nWeaknesses:\n\n(Minor) The ablation studies do show that while magnitude is the dominant factor, the choice of basis vectors (e.g., DST vs. SVD-derived) and the update direction (as influenced by data-driven methods like LoRA-GA) can still lead to minor performance differences outside of magnitude alone. The paper does acknowledge this.\n\nQUESTIONS:\nNo further questions\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work in a dedicated section. They correctly note that LoRAM mimics, rather than optimizes, spectral magnitudes and that further gains might be found by jointly optimizing magnitude with other hyperparameters like rank and learning rate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the parameter-efficient fine-tuning of large language models. The authors investigate the initialization methods of LoRA to improve its convergence and performance. Specifically, they theoretically re-interpret the success of spectral initialization in terms of weight magnitude and unify the hyperparameter for LoRA tuning as the magnitude regulation optimization. Furthermore, the authors propose an initialization method called LoRAM, which serves as a strong baseline for replacing spectral initialization across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. This paper provides a comprehensive theoretical analysis of the correlation between the LoRA training dynamics and the weight magnitude.\n2. This paper interprets the effectiveness of spectral initialization and unifies the hyperparameters of LoRA in terms of weight magnitude.\n3. The proposed LoRAM is easy to implement and improves LoRA in multiple scenarios.\n4. The proposed LoRAM is evaluated on multiple tasks and achieves comparable performance to spectral initialization methods.\n5. This paper is well-organized and easy to follow.\n\n**Weaknesses**:\n1. The proposed LoRAM is designed to avoid the additional matrix decomposition and storage overhead of spectral initialization methods. Nevertheless, the current experiments only evaluate LoRAM's performance in comparison to spectral methods. Including comparisons of computational costs and memory usage would further demonstrate the effectiveness and efficiency of the proposed approach.\n2. In the experiment of Figure 2, the pre-trained model and dataset should be described.\n3. Since the authors assume that gradients $\\nabla_W L^{(t)}$ follow a normal distribution in Proposition 2, it would be insightful to investigate whether this assumption holds reliably across different models, layers, and downstream tasks.\n4. The LoRAM is designed for improving naive LoRA, which might lack the flexibility to accommodate the current practice of incorporating additional operations into LoRA [1, 2, 3].\n\n[1] Expanding Sparse Tuning for Low Memory Usage, NeurIPS'24\n\n[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, NeurIPS'24\n\n[3] Parameter-Efficient Fine-Tuning with Discrete Fourier Transform, ICML'24\n\nQUESTIONS:\nAccording to the above weaknesses, I expect the authors to (i) demonstrate the efficiency of the proposed method, (ii) provide the reliance of their assumption.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the important problems of training dynamics and initialization in Low-Rank Adapters (LoRA). The authors develop an initialization (LORAM) based on deterministic orthogonal bases with controllable magnitude scaling parameters. This developed theoretically by treating the evolution of LoRA parameter magnitudes as a linear dynamical system and observing the magnitude changes during training. A correspondence between different optimization schemes is developed based on a scaling factor applied to adapters. \n\nThe authors demonstrate that the LORAM initialization achieves competitive performance to leading LoRA schemes across a range of Natural Language Generation, Natural Language Understanding, and Vision-Language tasks with superior performance achieved in several individual benchmarks. They additionally provide a number of additional analyses and ablations for their method, including the use of a 'tracking mode' that backsolves scaling parameters for LORAM based on target initialization schemes.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well motivated. Despite the many variants of LoRA there are still many open questions to its training dynamics and optimal initialization. The paper approaches this in an ambitious way: by seeking to develop a correspondence between the training dynamics under different optimization schemes and adapter magnitudes, the authors aim to unify understanding for a number of hyperparameter and design choices involved in LoRA. \n\nThe main strength of this paper is in its theoretical insight. In particular, the linear dynamical system approach for LoRA is interesting. In addition, the use of a deterministic orthogonal initialization (in particular the Discrete Sine Transform) is a very interesting approach which may motivate further exploration into other deterministic bases. \n\nThe experimentation and ablations are thorough and appropriate. Experiments appear to support the central idea that appropriately scaling parameters can boost LoRA convergence, and results show competitive or improved performance on a wide range of benchmarks across modalities (NLG, NLU, Vision-Language). \n\nIn terms of written description, the manuscript is relatively clear - with the related literature presented especially well. \n\nThere are a few important areas where the clarity is reduced: in particular, the figures are very information dense (for example, Figure 2a attempts to combine insights into multiple [a', a_A, a_B] hyperparameter ablation, optimizer [SGD, Adam], losses, and norm differences into a single double-axis figure with overlapping convergence lines [which I believe is meant to indicate they achieve equivalences as per Proposition 1]). This makes the figures very difficult to parse as a reader. Figure font sizes are often much too small to be visible when printed and when zoomed in. Adding in labels for axes (e.g. on Figures 2, 3, 5) would similarly improve the figures. \n\nTables 1, 2, and 3 are presented clearly and show that LORAM is competitive with leading methods, and achieves superior performance for several tests. Presentation for Table 4 is relatively dense (showing ablations for different scaling factors, bases, benchmark initialization methods, backsolved `tracking' initialization, etc) and could benefit from being split into separate tables in the supplementary materials to improve clarity. Bold figures could help make this comparison and the takeaway message clearer. \n\nMy general impression is that this paper has a lot of information that it wants to convey in a short amount of space. This unfortunately makes it relatively difficult to digest as a reader. Spacing these ideas out by focusing on the key message / simplified clear figures in the main paper, and moving less critical ablations to the supplementary materials would greatly improve this paper's potential impact. (E.g. - separating a few of the ablations in Figures 2 and 3, and providing sufficient space to describe this in the supplementary). \n\nSimilar issues appear in some of the mathematical descriptions (very densely written, with some overloading of notation [e.g. it is not clear how B^{(0)} differs from B_{init}], and many details feeling a little \"cramped\" to fit in the full technical details of ablations, e.g. logarithmic gain factor, tracking mode, and the DST basis). Including many technical components in the main paper can mean that there is less space available to develop, motivate, and connect each idea clearly. In contrast, I found the proofs in Supplementary B were relatively clear and well developed. Focusing in on the most important message (things that directly support the LORAM initialization and magnitude analysis), and leaving fuller description in the supplementary would make it much easier to digest (e.g. details of the LORAM initialization are provided relatively late in the method on page 6, and felt difficult to directly connect to the earlier formal linear dynamical system analysis and update magnitude scaling).\n\nQUESTIONS:\nMy current view is that this is a decent paper which presents interesting theoretical analysis along with well benchmarked and ablated experiments, but which would benefit substantially from improving clarity (especially for figures). In the current form many of the details are not immediately clear which detracts from the analysis and results. My suggestion would be that simplifying figures and sections by splitting less essential details to the supplementary materials could make this a good paper. \n\nIn addition to the details raised in the Strengths and Weaknesses, there are a few questions which it would be useful to clarify: \n\n* Could the authors clarify Figure 2a and 2b? As described in the Strengths and Weaknesses section these figures are difficult to digest (e.g. the meaning of the SGD region and the Adam region, and the interpretation of the loss and norm differences). My interpretation of b) is that it shows how changing the initial variance of the A and B matrices changes the total magnitude across training, but I am not certain \n* Could the authors clarify Figure 3? It appears to either show how the spectral concentration and gain factor change over training or across the dimension of the layers? In addition the insert in the p[r] row seems to refer to the early Q[r] values. \n* There are a few numbers that are used in the paper (e.g. L126-130, sigma_A = 1/20, k1 = 1/16) where it isn't immediately clear how these were selected and the how it connects with the theoretical analysis \n* In the Abstract the method is described as a `Basis and Basis' initialization scheme, however this isn't clearly defined later\n* The linear dynamical system approach used within Proposition 2 is an interesting and potentially novel way to analyze LoRA. As section 2.3 and the supplementary proof don't directly include citations, could the authors briefly discuss the papers they feel most relevant to the approach used in Proposition 2.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLoRAM introduces a magnitude-aware initialization scheme for LoRA that replaces SVD with deterministic orthogonal bases scaled by pretrained weight statistics. While preserving the standard LoRA training pipeline, this approach achieves comparable or superior performance with significantly reduced computational overhead. By reinterpreting spectral methods through the lens of update magnitude, the paper provides a unifying perspective on various hyperparameter strategies and highlights magnitude scaling as a key lever for efficient adaptation.\n\nSTRENGTHS AND WEAKNESSES:\n(+) Provides a unified theoretical framework linking LoRA hyperparameters through update magnitude. This perspective allows for a coherent interpretation of prior LoRA variants such as RsLoRA and LoRA+, which represents a clear theoretical contribution.\n\n(+) The ablation studies are appropriately designed and help isolate the effects of key components in the proposed method.\n\n(-) The authors reinterpret SVD-based initialization—commonly understood as preserving principal directions—as primarily contributing to performance through magnitude amplification. This is an insightful and valid perspective. However, it is important not to dismiss the spatial or spectral interpretations of SVD entirely. The original LoRA paper analyzes the similarity between $W$ and $\\Delta W$ in weight space, and later methods such as MiLoRA and CorDA explicitly leverage the structure of principal and minor components to mitigate forgetting and enhance transferability. Notably, MiLoRA achieves strong performance even when modifying only small singular components, making its underperformance in this paper somewhat surprising and potentially in need of further clarification.\n\n(-) The overall narrative may slightly overstate the universality of the magnitude principle. In practice, the benefit of increasing update magnitude likely depends on factors such as rank size, model architecture, and task characteristics. A more nuanced discussion of when magnitude scaling is helpful—and when it may result in diminishing returns or instability—would strengthen the overall framing.\n\n(-) As shown in Figure 5(a), the improvement in convergence speed may not be as pronounced as implied, and appears moderate relative to other methods.\n\n(-) While LoRAM achieves consistent performance gains in Tables 1 and 2, the improvements over strong baselines are relatively modest. The results are solid and reliable, but in some cases the margin may be too small to be considered practically significant.\n\nQUESTIONS:\nPlease refer to weakness\n\nLIMITATIONS:\nPlease refer to weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}