Title: COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

model compression

low-rank approximation

matrix decompositions

regularization
Abstract: Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks.
In this type of approximation, a norm is weighted by a matrix of input activations, significantly improving metrics over the unweighted case. 
Nevertheless, existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion. 
We demonstrate that this can degrade the approximation quality or cause numerically singular matrices.

To address these limitations, we propose a novel _inversion-free regularized framework_ that is based entirely on stable decompositions and overcomes the numerical pitfalls of prior art. 
Our method can handle all possible challenging scenarios: (1)&nbsp;when calibration matrices exceed GPU memory capacity, (2)&nbsp;when input activation matrices are nearly singular, and even (3)&nbsp;when insufficient data prevents unique approximation.
For the latter, we prove that our solution converges to a desired approximation and derive explicit error bounds.


Peer reviews:
comments: SUMMARY:
This paper proposes a numerically stable and efficient framework to perform block-wise context-aware low rank approximation on LLMs. Specifically, the paper proposes a QR decomposition-based inverse free optimization algorithm an a regularization to avoid overfitting. Experiments are conducted on multiple LLMs across different tasks.

STRENGTHS AND WEAKNESSES:
## Strength
1. The paper provides a novel inversion-free regularized optimization framework to solve the low-rank approximation problem of LLMs
2. The theoretical derivations provided in the paper are clear and solid. The paper has a high writting quality and is easy to follow
3. Adequate experimental results are included to show the improvements compared with baseline methods.

## Weakness
1. Low rank approximation generally leads to a worse performance-efficiency tradeoff compared to pruning and quantization. To this end, though the proposed method improves over existing baseline, it has limited significance on pushing the compression state-of-the-art forward. It is interesting to see if the proposed approximation algorithm or derivations can be used on quantization or pruning as well.
2. The effectiveness of the regularization $\mu$ is worth analyzing. Though the paper provides an theoretical derivation on the impact of $\mu$, an experimental study on different choices of $\mu$ is preferred

QUESTIONS:
Please provide experimental results on the effect of using different $\mu$ in Equation (4)

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoTAn -- a numerically stable and inversion free framework to minimize the optimization problem $$\\text{minimize}_{W' : \\text{rank}(W') \\leq r} ||WX - W'X||_F,$$ where $X$ is a calibration data matrix. This problem is referred to as the *context-aware low-rank approximation* problem in the paper. Unlike prior approaches mentioned (e.g., SVD-LLM, ASVD), CoTAn avoids computing and inverting the Gram matrix of the calibration data, i.e., $X^\\top X$, which is often ill-conditioned or singular in practice. Hence, the name, \"numerically stable\" in the title of the paper.\n\nThis is done using stable matrix factorizations such as QR and SVD decompositions directly on the transformed input data (rather than the Gram matrix) -- optionally combined with regularization to handle ill-posed or underdetermined settings. The paper also discusses an efficient Tall-Skinny QR decomposition (TSQR) for scalability to large matrices, and proves convergence and error bounds for the regularized variant. Experiments on Llama-3 family of models and Mistral, and 8 downstream evaluation tasks (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC-e, ARC-c, OBQA) show that CoTAn achieves comparable or better performance with greater stability and faster runtime, when compared to ASVD and SVD-LLM.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\nThe paper address a relevant problem in the field of LLM compression -- for compressing weight matrices in an LLM, it is not sufficient to preserve just the Frobenius norm error of the weight matrix, but rather the output activations of each layer. Consequently, several LLM compression methods have resorted to using a small calibration dataset for this purpose, and this paper studies low-rank approximation with a calibration dataset. The paper addresses a practical issue regarding matrix inversion errors while inverting (near) singular Gram matrices, while providing clean mathematical formulations, correctness proofs, and convergence guarantees for the same. Numerical experiments also validate the theoretical claims.\n\nIn terms of clarity, the paper, in general, seems well-organized and technically sound with a clear motivation for each design choice. The pseudocode and figures also help in understanding and are highly appreciated. Some important derivations are delegated to the Appendix, but could benefit from an intuitive explanation in the main text. \n\n**Weakness**:\nI believe the paper can do a better job placing itself in the context of some relevant works. For instance, [this paper](https://proceedings.neurips.cc/paper_files/paper/2024/hash/a20e8451ffb07ad25282c21945ad4f19-Abstract-Conference.html) is quite relevant and adopts a similar approach to solve the context-aware low-rank approximation problem. How does CoTAn compare with the approach presented in this paper? For instance, Lemma 4.2 (and its complete derivation in Appendix B) of this paper, also solves the same problem, without inverting any singular matrix. The rank-constrained regression problem (eq. 5 in this paper ) seems to be the same problem as CoTAn solves (up to some change in variables). The authors seem to have missed discussing this in related works. \n\nMoreover, low-rank decomposition of the weight matrix, by itself, introduces a very large gap in downstream accuracies with respect to the uncompressed baselines. This is true for all methods like ASVD, SVD-LLM and including CoTAn. This often renders low-rank factorizations not-so-useful in practical application where accuracy requirements are paramount. Some discussion needs to be added about how CoTAn is compatible with other methods that can help reduce the accuracy gap between low-rank factorized and uncompressed models.\n\nI would be happy to revise my score if my concerns are addressed.\n\nQUESTIONS:\nI have some questions and would appreciate it if they are addressed:\n\n1. Can the authors comment on how sensitive CoTAN is to the choice of the regularization parameter $\\mu$? Is there a principled way to choose it (for example, via calibration)? Such ablation studies would be helpful in understanding the role of a proximal regularizer that penalizes deviation, such as in (4).\n\n2. Any ablation study with the choice of rank? \n\nPlease also refer to other important concerns in the previous strengths and weaknesses section.\n\nLIMITATIONS:\nYes. Would appreciate if authors would comment of the limitation that low-rank approximation by itself leads to significant accuracy drop.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposes an approach based on decompositions that is numerically stable compared to methods that relies on Gram matrix and their subsequent inversion. CoTAn introduces an inversion-free solution based on stable matrix decompositions like QR and SVD. The framework is comprehensive, proposing the TSQR algorithm for memory-constrained scenarios and a novel regularization technique for data-scarce settings. The author shows that the method is appliable on large language models namely LLama 3.1 and Mistral 7B and is compared on wide datasets, demonstrating superior performance and stability in compression compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n* Tackles a Critical Problem: The paper addresses the crucial and practical issue of numerical stability during the compression and fine-tuning of large language models, a significant bottleneck for current methods.\n\n* Comprehensive and Versatile Framework: The proposed solution is not a single trick but a complete framework. The extension to handle memory-constrained scenarios via TSQR and data-scarce situations via regularization makes CoTAn robust and widely applicable.\n\n* Diversified and Strong Evaluation: The method is thoroughly evaluated on multiple modern architectures (LLaMA3-8B, LLaMA3-1B, Mistral-7B) across a wide range of  datasets, providing convincing evidence of its effectiveness. The work also clearly shows how to choose between SVD and QR decomposition for efficiency depending on the hardware and matrix dimensions\n\n\n* Solid Theoretical and Practical Support: The claims are well-supported by clear mathematical propositions and proofs. Furthermore, the inclusion of pseudocode and detailed hyperparameters in the appendix significantly aids reproducibility.\n\n\n### Weakness :\nThe following weaknesses are minor and more for readability \n* Guidance on Regularization Parameter: The method's effectiveness is clearly shown, yet the sensitivity of the performance to the regularization parameter µ is not fully explored. For instance, the paper states with each results the value of μ ,\n > CoTAnμ​ is regularized approach, we get µ = 5 · 10−4\", but the link between this choice and the final accuracy is not established.\"\n\nQUESTIONS:\n*  To improve the practical utility of the method, could you provide a sensitivity analysis or a plot showing the effect of different µ values on model performance? This would offer valuable guidance for selecting this hyperparameter.\n*  Adaptive rank to compare the methods fairly though it is an important aspect for compression ( L225 - 226) could you please either showcase the overhead those methods adds. Models are compressed once and then  deployed, if the margin in performance is large sometimes it is worth the effort to tune\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCoTAn aims to preserve the most important directions of the activations. The method computes a low-rank decomposition that approximately minimizes error on the layer’s output activations, rather than weight reconstruction error itself. To achieve this efficiently, CoTAn uses an inversion-free algorithm: instead of explicitly inverting large covariance matrices or solving normal equations, it leverages orthogonal projections and QR factorizations to find the low-dimensional subspace spanned by the data, and then performs a truncated SVD or low-rank factorization in that subspace. This approach avoids  matrix inverses and can be implemented in a distributed manner.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- The paper addresses a meaningful problem in model compression: numerical instability in context-aware low-rank approximations. The authors propose an inversion-free formulation that avoids inverting Gram matrices, the use of QR and TSQR decompositions is well-motivated from numerical linear algebra.\n\n*Weaknesses*\n\n- Related literature is not well addressed. Statement \"Structured pruning removes unnecessary parameters and simplifies the\nmodel architecture without significant loss of accuracy In turn, the low rank decomposition approach is often memory-efficient and can accelerate model inference\" is not correct in general,  compression rate/accuracy drop depends significantly on the data, neural network architecture, way of training, tensor decompositions sometimes can lead to slow down of inference if rank is not small enough or due do increased number of reading-writing accesses to the memory.\n- The paper's  claims like stability and  generalization are not well demonstrated by empirical results\n\nQUESTIONS:\n- What happens when the context data used for compression is misaligned with test-time data—how robust is the method?\n- Can the authors clarify how CoTAn would be integrated into a full training or fine-tuning pipeline, or is it solely for post-training compression\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}