Title: PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Venue: NeurIPS 2025
Accepted: yes
Keywords: low-rank adaptation

architecture-optimizer co-design

large language models

lora

low-rank adapter

fine-tuning
Abstract: We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PoLAR, a new method for PEFT of LLMs that leverages the polar-like decomposition of LoRA matrices into unitary and a low-dimensional components. The authors provide theoretical motivation with rigorous analysis and demonstrate empirical gains over strong LoRA-based baselines across various model sizes and tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is well-organized and clearly written, with effective use of figures to illustrate key concepts and results.\n2.  The authors present a theoretical connection between PEFT and matrix factorization. They analyze convergence properties under this framework in a rigorous way.\n3. The proposed “landing trick” serves as a computationally efficient alternative to the more expensive Stiefel retraction, enabling the enforcement of orthonormality constraints with minimal overhead.\n4. The experimental results include both small- and large-scale language models, demonstrating the scalability of the approach across model sizes.\n\nWeaknesses:\n1. The theoretical analysis on matrix factorization does not clearly translate into deeper understanding of PEFT or address any of the known theoretical challenges in the field. The connection feels superficial and lacks strong justification for its practical relevance.\n2. The performance improvements shown in Table 2 and other benchmarks are modest. Given the competitiveness of the PEFT field, it is unclear whether polar decomposition offers a meaningful advantage over existing methods such as LoRA or QLoRA. More extensive ablation or comparisons would help clarify the benefits.\n\nQUESTIONS:\n1. A comment: \nThe authors reference Zeng and Lee (2024) to motivate the expressiveness of low-rank adaptation, noting that LoRA can approximate any target transformer under mild conditions. However, the cited result crucially depends on using relatively high ranks—specifically, LoRA rank being at least half the embedding dimension—which makes the theoretical result more about approximation capacity than practical efficiency. In contrast, this paper aims to improve fine-tuning efficiency by operating in a very low-rank regime, which is far from the high-rank setting assumed in Zeng and Lee. Therefore, the claim that “the low-rank space offers sufficient expressiveness” is not directly supported by that citation.\n\n2. In the theoretical analysis (e.g., lines 168–170), the convergence is derived under the assumption of zero training loss, which implies that the model operates in a high-rank regime—something the authors also acknowledge. However, this seems inconsistent with the practical low-rank settings  used in PEFT. Could the authors clarify how much insight the analysis in the full-rank or zero-loss regime provides for understanding the behavior of LoRA or PoLAR in low-rank regimes commonly used in practice?\n\n3. In Theorem 1, the expression uses T \\geq O(\\cdot), but the Big-O notation already implies an upper bound. Would it not be more precise to write T = O(\\cdot) or clarify the inequality with appropriate asymptotic notation?\nSimilarly, in the supplementary material (line 760, Condition C4), the equality marked as (a) is said to follow from the data being whitened. However, whitening ensures the identity covariance only in expectation, not deterministically. Shouldn’t this step be marked as an approximation, rather than a strict equality? While these may be minor points, they raise concerns about the precision and rigor of the theoretical claims in a paper that is otherwise theory-driven.\n\n\n4. In Algorithm 1, the matrix \\Theta is updated at the beginning of each iteration, whereas in Algorithm 2, it is updated at the very end. This inconsistency raises a question: what would happen if we swapped the update order in Algorithm 2 to match that of Algorithm 1? Is there a theoretical or empirical reason for this discrepancy? Some clarification would help understand whether the difference is deliberate or incidental.\n\n5. In Table 2, both LoRA and PoLAR show lower average performance at rank 32 compared to some smaller ranks. This seems counterintuitive since higher ranks should, in principle, offer more capacity. Can the authors comment on why this drop occurs? Additionally, did you train all configurations (across different ranks) with the same number of epochs? If so, could underfitting or overfitting at higher ranks be a factor?\n\nLIMITATIONS:\nThe authors briefly mention some limitations of their approach, such as the orthogonal constraint’s potential optimization challenges, but they do not clearly discuss broader limitations or possible negative societal impacts. It would strengthen the paper if they included a more explicit reflection on the practical constraints of PoLAR (eg. the selection of \\lambda) and any potential risks in real-world deployment scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduces a new method for fine-tuning large language models that addresses LoRA's underuse of the allocated rank, leading to low stable rank and collapsed update directions. PoLAR uses polar decomposition to factor low-rank updates into orthogonal direction matrices and a scale matrix, improving expressiveness. Combined with a Riemannian optimization strategy that avoids costly retractions, PoLAR achieves faster convergence and better GPU efficiency. PoLAR outperforms LoRA and DoRA  (the most recent reproducible SoTA) across models and tasks by maintaining higher stable rank.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Comprehensive and detailed theoretical grounding.\n- The method scales well with model sizes.\n- The implementation of the landing method seems stable and significantly improves convergence time.\n\nWeaknesses:\n- Optimizing over Steifel manifolds still introduces overhead and significant complexity over DoRA. This is considering that the performance improvements are noticeable but not necessarily significant in the tested benchmarks.\n- It would also be interesting to see the difference in the jump in stable rank from this work compared to multi-lora applications. I say this because multi-lora methods bump up the rank by adding multiple matrices together, which could result in a rank that is higher than the sum of all ranks without changing anything about the typical adapter.\n- The state of the art being compared against is a bit limited. I understand that in most of the tested tasks, AdaLoRA barely performs better than DoRA, but both AdaLoRA and LoRA+ are missing.\n- Addressing the diversity collapse with this work leads to better stable rank within the adapter. Interpreting this from a LoRA’s perspective, the singular value spectrum is more uniform as opposed to being peaky. Other works like DoRA and LoRA+ have directly or indirectly addressed this issue, and their claims were mostly directed towards matching the training dynamics of full fine-tuning. Similar analyses would be interesting to see if the same behavior is seen from PoLAR.\n\nNitpick:\nThe references to the Appendix in the main text are too frequent, and it disturbs the flow of reading.\n\nQUESTIONS:\nIn Figure 5, the stable rank of LoRA seems to have a wider range compared to PoLAR different from other datasets. You mention this in L1060, but I couldn't find an explanation or hypothesis about it. Why do you think PoLAR doesn't provide any improvements here?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PoLAR, a parameter-efficient fine-tuning (PEFT) method designed to enhance the directional diversity of weight updates ($\\Delta W$) in LoRA, thereby addressing the low stable rank issue. Experimental results demonstrate that PoLAR achieves consistent accuracy improvements across diverse datasets and base models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Clear motivations.** The paper effectively identifies the directional diversity collapse in low-rank adapters, articulates its impact, and explains how PoLAR addresses this issue.\n2. **Theoretically sound analysis.** The study provides a comprehensive theoretical framework, including problem definition and convergence guarantees, strengthening the method’s foundation.\n3. **Practical implementation.** This paper introduces an alternating update method to solve the low-rank approximation problem with reduced computational requirement. \n\n**Weaknesses**\n\n1. **Limited discussions of related  work.** One major reason for the low directional diversity of LoRA may be attributed to the non-orthogonal projection. There are other related work that attempts to solve the same problem, such as SVD-based GaLore (Zhao et al., 2024) and other related works. The discussion on the similarity and distinctions between the proposed method and GaLore-like methods are necessary.\n2. **Misalignment between theory and practice.** While convergence is analyzed under Riemannian gradient descent, the practical implementation relies on an alternating update algorithm. An analysis of the convergence properties of this algorithm would bridge this gap.\n3. **Insufficient experiments.** The experiments compare PoLAR only with LoRA and DoRA. Including additional methods like GaLore, Fira (Chen et al., 2024), SLTrain (Han et al., 2024), and full-rank fine-tuning would provide a more comprehensive evaluation.\n4. **Limited memory efficiency.** Unlike recent PEFT methods that prioritize both parameter and memory efficiency, PoLAR does not demonstrate memory advantages over LoRA, potentially limiting its practical applicability.\n5. **The organization and presentations can be further improved.**\n   - Key content, such as Appendix C.1, is relegated to the appendices despite its importance and should be integrated into the main text. Frequent references to appendices disrupt the reading flow.\n   - [Minor] The core idea’s presentation could be more concise, avoiding excessive use of technical terms and complex concepts.\n\n**References**\n\n[R1] X. Chen, et al. (2024). Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? *arXiv preprint: 2410.01623*.\n\n[R2] A. Han, J. Li, W. Huang, M. Hong, & A. Takeda, (2024). SLTrain: A Sparse Plus Low-Rank Approach for Parameter and Memory Efficient Pretraining. *NeurIPS 2024*.\n\nQUESTIONS:\n1. How does PoLAR compare to GaLore and other methods addressing directional diversity (*e.g.*, SVD-based approaches)? Please provide a discussion on similarities and distinctions.\n\n2. Is it possible to provide a convergence analysis for the alternating update algorithm used in PoLAR’s implementation, given that the theoretical analysis relies on Riemannian gradient descent?\n\n3. How does the proposed method performs compared to methods like GaLore, Fira, SLTrain, and full-rank fine-tuning?\n\n4. Does PoLAR offer any memory efficiency advantages over LoRA, or are there plans to address this limitation in future work?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors start out with looking at PEFT -- parameter efficient fine-tuning. Their key enabling insight, which serves as strong motivation for this work, is that vanilla LoRA adapters demonstrate low stable rank, and severe underutilization of available algebraic rank and, thus, expressivity. Thus, while structurally, capacity is available, it is not utilized due to demonstrated directionality collapse (DC). What's interesting is that the effective stable rank over time converges to 1. Armed with this insight, the authors propose to factor low rank updates (ie \\delta{W} updates to W_0) into direction matrices and a separate scale matrix. This parameterization is then suported by a Riemannian optimization. As a result of this, the authors demonstrate exponentially faster convergence rate compared to vanilla LoRA! Findings generalize across tasks, models, and model sizes. Impressive work.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* very good presentation. Well written, thoughtful, thorough disposition throughout the paper. I enjoyed reading it. Best in the pile.\n* the proposed technique is mathematically sound\n* strong key enabling insight, showing directional collapse in standard LoRA adapters as well as very low stable rank and, therefore, poor utilization of available expressivity. The fact that varying r typically doesn't help is strongly aligned with this insight.\n* findings generalize across tasks, models, and even model sizes (up to 27B)\n\n\n**Weaknesses**\n* some important LoRA initialization work hasn't been cited [1]. It would also be great to see if PoLAR can generalize to this type of LoRA parameterization (see questions). \n\n[1] Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning, https://arxiv.org/abs/2411.19557. Submitted on 29 Nov 2024 (v1).\n\nQUESTIONS:\n* would you be able to demonstrate your proposed technique on the Silver Bullet, which reduces a set of FT-trainable parameters to R (rxr), instead of the typical A and B LoRA matrices. This reduction to R may lead to higher stable rank. I'm wondering whether Silver Bullet led to improving/mitigating directional collapse as well.\n* To what extent is PoLAR sensitive to initialization? Conversely, if we use the best possible initialization, does this help mitigate the stable rank or DC in LoRA? In other words, can initialization help with this problem, or is it fundamentally an issue with the optimization step and, therefore, truly requires proposed decomposition? \n\n[1] Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning, https://arxiv.org/abs/2411.19557. Submitted on 29 Nov 2024 (v1).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}