COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Authors: Uliana Parkina
Maxim Rakhuba
Venue: NeurIPS 2025 poster ()
Keywords: LLM
model compression
low-rank approximation
matrix decompositions
regularization
Primary area: deep_learning
TL;DR: We introduce an inversion-free regularized framework using stable decompositions to overcome numerical instabilities in context-aware low-rank approximation for neural networks.

Abstract: Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks.
In this type of approximation, a norm is weighted by a matrix of input activations, significantly improving metrics over the unweighted case. 
Nevertheless, existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion. 
We demonstrate that this can degrade the approximation quality or cause numerically singular matrices.

To address these limitations, we propose a novel _inversion-free regularized framework_ that is based entirely on stable decompositions and overcomes the numerical pitfalls of prior art. 
Our method can handle all possible challenging scenarios: (1)&nbsp;when calibration matrices exceed GPU memory capacity, (2)&nbsp;when input activation matrices are nearly singular, and even (3)&nbsp;when insufficient data prevents unique approximation.
For the latter, we prove that our solution converges to a desired approximation and derive explicit error bounds.

COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Uliana Parkina
Maxim Rakhuba HSE University
HSE University
uliana, pakkinaaark com com
Abstract
Recent studies suggest that context-aware low-rank approximation is useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximation, a norm is weighted by a matrix of input activations, significantly improving metrics over the unweighted case. Nevertheless, existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion. We demonstrate that this can degrade the approximation quality or cause numerically singular matrices.
To address these limitations, we propose novel inversion-free regularized frame- work that is based entirely on stable decompositions and overcomes the numer- ical pitfalls of prior art. Our method can handle possible challenging scenarios: (1) when calibration matrices exceed GPU memory capacity, (2) when input acti- vation matrices are nearly singular, and even (3) when insufficient data prevents unique approximation. For the latter, we prove that our solution converges to a desired approximation and derive explicit error bounds.
Introduction
Large Language Models (LLMs) have demonstrated high performance across a variety of tasks [5] leading to significant advancements in artificial intelligence. However, the increasing size of these models brings efficiency challenges, including inference speed and model size when resources are constrained [55]
54. To address these issues, various approaches have been proposed, such as model compression and fine-tuning. Specifically, basic and context-aware low-rank approximation
techniques have proven to be an effective tool for compressing [46] 50
6
27 21
and fine-
tuning 33 47 43 modern large-scale neural networks.
In the context-aware approach, trix W R given input data X Ro*k where k is the batch size multiplied by the context length. The goal is to find low-rank approximation W of W that maintains the performance of the neural network while reducing its computational complexity. This objective leads to the minimization of
consider the task of approximating weight
L(W") = |Wx - W'XIIe;
where -1 denotes the Frobenius norm. We aim to minimize L(W") with respect to the matrix W" Rm under the constraint that the rank of W does not exceed r.
Despite its seeming simplicity, the context-aware low-rank approximation poses various challenges from the computational point of view:
Numerical instabilities.
The first source of difficulty stems from numerical instability. Prior frequently depend on the inversion of large or nearly singular Gram matrices
methods 46 25 47
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of the columns of X, which can degrade performance and introduce substantial computational errors in practice 27). Although theoretical guarantees for such inversion-based strategies often assume that the Gram matrix is of full rank, this condition can fail under real-world constraints and floating-point arithmetic 44 27
Large calibration datasets. Another challenge relates to memory limitations, particularly notice- able in large-scale scenarios. For instance, calibrating LLaMA3-8B [15] using 100 examples of length 2048 tokens and an internal dimensionality of 14336 leads z 10.9Gb matrix X in a single precision. Thus, the method should be memory-effiicee and, if necessary, support batch processing without explicitly constructing the whole matrix X.
Limited data. A final challenge emerges when dealing with severely limited data. In low-data regimes (e.g., 3-5 images in generative model adaptation [17 37]) the problem becomes ill- posed and susceptible to non-unique solutions and overfitting. Similar difficulties appear in model compression tasks when only constrained datasets are available
In this paper, overcome all these issues in single framework, called COALA (COntext-Aware Low-rank Approximation). Our contributions are as follows.
We propose to use a regularization term that balances fitting the available examples with preserving the model's capacity to generalize. This regularized formulation yields unique solution for any X. Moreover, it boosts metrics of compressed neural networks by mitigating overfitting. Under certain assumptions, we establish theoretical convergence of the regularized solution.
We show how fully avoid both inversion and computation of Gram matrices, which prone to numerical instabilities [12]. Also, to avoid computational challenges associated with large matrices X, we preprocess them via the reliable TSQR algorithm [[]] which computes a QR decomposition in smaller chunks.
2
Related Work
In recent years, the compression of deep learning models has become a critical focus within the field.
Several approaches have been proposed address this challenge, including quantization [48] 24).
structured pruning (30 2 53 and low-rank approximation methods 12 49 45].
Quantization
allows for reducing the bit-width of model weights, thereby decreasing memory consumption and accelerating computations. Structured pruning removes unnecessary parameters and simplifies the model architecture without significant loss of accuracy. Also, the low rank decomposition approach is often memory-efficient and can accelerate model inference, which is particularly important for tasks related to response speed and model size when deploying on mobile devices [50
28
55).
The primary concept behind model compression using low-rank approximations to represent the weight matrix W as the product of two low-rank matrices: W UV, where W Rmxn Rm V E " This representation allows for storing Omrr + nr) elements instead of the original Omm)), and enables the propagation of data X E Ra*k through a layer with computational complexity O(nkr + mkr) rather than O(mnk), which is advantageous when < m, n.
In this context, the theory of low-rank matrix approximations developed to preserve certain properties of the original matrix. For instance, the Eckart- singular value decomposition (SVD) [33 of matrix, allows for the construction of a low low-rank matrix
theorem [13]. based on the
W' that best approximates W in the sense of minimizing the norm || Will for any unitarily invariant norm, such as the Frobenius norm. However, as demonstrated in works [46] 50), such approximations do not always efficiently preserve the model's performance and are outperformed by compression methods based on alternative ideas, such as quantization, structured pruning, and unstructured pruning.
The work ASVD 50
proposes a solution that manages activation outliers by transforming the weight matrix based on the activation distribution. However, this solution does not achieve the best approximation error in the posed problem, providing a reasonable yet suboptimal solution [46]. Other studies, such
[46 44), [25) and (6). present solutions that attain the theoretical minimum of the

---

error in the Frobenius norm. Nonetheless, they still rely on the formation of Gram matrices and/or inversion of small singular values.
Weighted Low-Rank Approximation
Building upon the previous discussion, by introducing context-aware approach each layer using X, we aim to reduce the number of parameters needed to store the matrix W by finding its low-rank approximation. Formally, this can be formulated as the following minimization problem:
min Tank(W"))
II( W')
This problem a special case of the general weighted low-rank approximation problem 31
32
vec[ - w Q vee[W W'), rank(W")s
min
where Q is positive definite symmetric matrix and vecf.} denotes the column-wiss vectorization of a matrix. By applying [3] Theorem 3] (see Appendix A) in our specific case of the matrix Q, we obtain:
W = UE,V'S
where S = (X/ is the unique positive definite square root of XXX,, UEVT is the SVD of WS, and E, is the matrix obtained from E by setting the last n singular values zero.
One can show that using the symmetric matrix square root is not the only possible way to obtain the solution. Any decomposition of the form SS XXT with square matrix S is applicable as well. For example, S can be an R. factor from the Cholesky decomposition of the matrix X X Alternatively. it be based on SVD of X For example, these
approaches were used in 46 and utilized other studies 25 47), see Appendix B As we will see further, forming the Gram matrix in this context may already lead numerical problems the ill-conditioned case, see also a theoretical example from Appendix G.I Inverting nearly singular matrices afterwards only deteriorate this effect. In the next section, we show how naturally avoid both problems at once.
Inversion-Free Solution
The following result provides a simple yet effective orthogonal-projection-- formula, avoiding matrix inversion and Gram matrices.
Proposition 1. Let W E m and X E BRak be arbitrary matrices. A solution to the optimization problem
min IWX -IX-W'Xlle
(3)
rank[W Sr
is attained at W' = U.U,T W, where U, consists of the first left singular vectors of the matrix W.
Proof. Let define A = W and B W'X. Then,
rank B) < min (rank(W'), rank(X)) I rank(W') S
It is well-known that the minimizer of ||A B|]F under the constraint rank (B) < rri given by B = U,U.T A, where U, contains the first As left singular vectors of A (see Corollary 2 for details). Substituting back for and B, we have B = U.U,T A = U,U,T WX, implying
W'X U,U, WX.
Hence, of the possible solutions W looks as follows:
W' U,U," W.
As desired, the rank W" does not exceed r because U, has rank r. Note that in general there can be many solutions depending the matrix X.
D
3

---

Although this result is well-established for the unweighted case (X* = I). we include proof here since we were unable find a weighted analogue in the literature. Note that this formula does not require any additional constraints on X, such as the assumption of full column rank, which is required in [46 44). However, let us note that the number of columns of the matrix X grows with the number of samples and can be fairly large quantity, exceeding m and n by many times. Nevertheless, can can be efficiently computed with the help of the reliable QR decomposition.
Proposition 2. Suppose that n S. k. Then, we can get Ur in Proposition 7 as the first v left singular vectors of the malrix WR', where R is the upper triangular matrix from the QR decomposition of XX.
Proof. Let QR X be the QR decomposition of X Then, using orthogonal invariance of |
II(W" 1 = ((W--)) Q T
tr (tW" WV)RTQTQRIN
- -1
= tr ((W' - WWR R(W" - -)) = - 1|2
We complete the proof by applying Proposition to the new minimization task.
Note that in the proof of Proposition -
we only use the fact that RD R XX so any matrix for
which this is true will suffice.
The pseudocode of the final solution is summarized in Algorithm
!
Algorithm A Stable Solution to the Weighted Low-Rank Approximation Problem
Require: W E Rm X E Raxk r E.N,N S k
Ensure:
A E Rmxr B E Rexn
Compute the upper-triangular factor R by performing a TSQR factorization of X
[Q, R] < QR(XT)
& Use Tall-Skinny QR (TSQR) method, see Section
Compute the SVD of W WT:
[U, VT] + SVD(WRR))
Let U, r(
Set A + U,
Set + U W
6 return A, B
4.1 Stability
Let us analyze potential issues that arise on real-world data. particular, we use the LLaMA3 [5]]
model on the WikiText2 (34] dataset and construct weighted low-rank approximation of matrices in three ways: (1) via Cholesky decomposition of (XX X as in SVD-LLM, (2) via SVD of (XX) as in SVD-LLM v2, and (3) via the QR-based approach.
Figure T shows that the approaches relying on the Gram matrix suffer from large errors that are independent of the chosen rank and appear already during the construction of the approximation of W. We evaluate the error in the spectral norm || ll2. the operator norm induced by the Euclidean vector norm via || supzy/0 || / /||==]|2. Being defined through supremum over all possible inputs, this bound cannot be exceeded by any particular vector x.
The size of these errors is linked to the distribution of singular values: very small singular values cause numerical instabilities when inversion of the Gram matrix involved. As illustrated Figure 2 several layers exhibit sharp drop in the smallest singular values of the input matrix X. Our findings indicate that computing X introduces noticeable numerical errors, which may subsequently impact the final results.
4.2 Efficiency
In this section, we also discuss the compression time for large models. In our approach we preprocess X using the QR decomposition, see Proposition 2 decomposition provides further acceleration.
The need for only the R factor in the QR

---

10
10
10
10
Hoer Wels
QR
Cholesky
10
SVD-based
10"
10-3
10
500
1000
1500
2000
500
2000
singular value number
Figure 1: Relative approximation error versus ap- proximation rank obtained by different methods on layer q-proj. The reference weight ma- trix Wiee was computed using the inversion-free COALA method and in high working precision (fp64) to serve as the ground-truth solution. The LLaMA3-IB [IS] model was used with 64 exam- ples from the Wikitext [3] dataset.
Figure 2: Distribution of singular values of matrix X, obtained from the outputs of layer q_proj in the ILLMMA-1B [5] model, computed over 64 samples from the WikiText 34 dataset.
Table I: Computation times produced by different methods.
Model
#Samples
Strategy
Time, s
SVD-LLM
273.93-:22. 12
LLaMA3-IB
64
SVD-LLM V2
404.88
COALA
196.34 16.48
SVD-LLM
3624.88
512.4
LLLMA3-SB
SVD-LLM V2
4210.5
63.3
COALA
1811.0 +15.6
We compared the time required by different methods in Table I Additionally, examined the break- ing point at which computing the SVD of X becomes faster than performing a QR decomposition of X. We have observed that dimension exceeding the other by several tens of times - the QR decomposition remains the preferred performed on single NVIDIA A100 GPU. To
when the matrix has a highly unbalanced aspect ratio with one
method, see Figure left graph. All calculations preserve the integrity of the experiment, the SVD on the GPU was executed with Porochh's "gesvd" method, because the default "gesvdj" method, although faster, produces noticeably larger error.
80
4.0
time (seconds)
2.0
time (scconds)
1.0
8.0
TSUR
0.5-
0.25
8.0
BVDOXX
4-10'
8-10'
2- 10
3-
7 1-
6-10-2
1-10-1
2-10-1
1-10P chunk size (Gb)
5-10
2-10
4-10"
number of columns
Figure 3: Runtimes for computing S: SST X using two approaches. Left: Matrix
R 1-006xx for different n. Right: Matrix E IR 1R106xx--10 split into chunks of different size. In this case, QR is computed using the TSQR method and the Gram matrix using XXT = Σ. X,X,T
When dealing with matrices so large that they cannot be accommodated in fast memory, one still easily compute the Gram matrix by splitting it into p batches of smaller sizes that fit in
5

---

memory, resulting into XX" = =E X,X." In our case, we can also efficiently compute the QR decomposition using the Tall Skinny QR (TSQR) method [TT. allows for reducing QR decomposition of the whole matrix to y QR decompositions of the smaller sizes. For example, we can sequentially apply the QR decomposition to each new block, incorporating the R matrix obtained from the previous step, i.e., for 3:
X X
QoRo X
Qo
Rg
x
I
X T
X2
X2
I
X2
Qo
QQ
Qoi
Ryj X2
Qai
i]
Q012Ro12.
T
1
1T
Since a product of matrices with orthonormal columns also has orthonornal columns, we conclude that this is indeed QR decomposition of X As can be seen in Figure (righ), this approach not only eliminates the need to store large matrices, but also speeds up the solution time for large-scale matrices X.
Moreover, multiple GPUs are available, the scheme can be transformed into a binary tree structure to enable parallel execution, thereby achieving speedup in computational time:
X + Ro
Roi
X1 + R1
Ro123
X2 + R2
R23
X3 + Rs
If one or more arrows point to the same matrix, then this matrix represents the R-factor obtained from the QR decomposition of the matrix formed by stacking all the matrices the opposite ends of the arrows. This process is described in more detail in
Weighted Low-Rank Approximation with Regularization
So far, have discussed various numerical aspects of solving the problem (3. However, in practice, we want to adapt the model to fit the available examples, but not excessively, as we aim to avoid overfitting and preserve the model's 's knowledge in other domains. This situation becomes particularly pronounced when data is scarce and matrix X may have more columns than rows. For example, in model compression, data are often limited due confidentiality, yet there's a need deploy models on devices with restricted resources 3 [9]. A similarly relevant challenge is adapting pre-trained
generative models to new concepts using just handful of images (usually 3-5) [17. Thus, we can formulate the following minimization problem:
min
[ - W' + Wlll - WII
rank(W))
where μ 0 is a given parameter. Notably, this strategy yields systematic improvements even in data-sufficient scenarios.
Our methodology presented in earlier sections continues to provide an efficient and robust solution to the problem 4 well.
Proposition 3. Let W E pmxn and X E Bank be arbitrary matrices. Then problem is equivalent to
min Tank(W")s SK
IIW - W'
where X [X Vall.
Proof. We have
II(W" - W WI WII = [(W" W)X
Va(W" -W) 12
1[WWW' W) IX V 11..

---

This equivalence means that we can use the same approach as in the unregularized problem by augmenting the data matrix X with the scaled identity matrix vud. By transforming the regularized problem into this form, we can apply efficient algorithms such as Proposition 2 for its solution. The corresponding pseudocode is presented in Algorithm 2
Algorithm 2 A solution to the weighted low-rank approximation problem with regularization
Require: W E R X Raxk
1 E R4, r N
Ensure:
A Rmxr
Rrxn
Form the matrix X = [X vP where is the n x n identity matrix.
2 Call Algorithm with imput (W,X', .X'r)) rr) to compute A and B.
return A, B
What is the limit of W,, + 0? Another natural question is what happens with the regularized solution W, for small If X is of full row rank, then it is natural to assume that it W converges to unique solution of the unregularized problem. It is, however, unclear what happens in the general case and what is the convergence rate. We establish that W,, converges to a well-defined solution Wo. which corresponds to the solution obtained from Proposition The following theorem provides precise estimate for the convergence rate.
Theorem Let W E Ruxx and X E Bank Suppose that has rank(X) k 2 r and that the singular values of WX satisfy () * (WX1(WX), where a+(-) denotes the i-th largest singular value. Let Wo = U,U,T W denote the solution to the problem (3. and let Wi, p denote the solution to the regularized problem ④
then the following estimate holds:
2/
| Wullr
o? (WX) - 1 (WX) 1
Proof. See Appendix E
In the case where X has full rank, we have a more precise estimate with a better constant, which, however, also involves the multiplier 1/gap, where
gap = or (WX) - a (WX),
see Appendix D This gap-dependent behavior is intrinsic to the problem, as demonstrated in Example G.2
Our estimate suggests that even in the degenerate case W,, approaches Wo linearly with respect to l as μ - 0, which we also observe in numerical simulations. The estimate may also be useful for practical reasons as it shows asymptotic dependence on key parameters such as the gap value and the regularization parameter Jr. For example, the estimate quantifies how sensitive our solution is to the choice of which can inform practical decisions about selecting an appropriate regularization parameter. This can be of particular interest applications where the balance between fitting the data and preventing overfitting is delicate.
6 Experiments
6.1 Model compression
In this section, we evaluate the effectiveness of our regularization-base compression approach in practice We first fix the procedure for selecting the regularization parameter μ.. Specifically, we determine y relative to the unregularized solution Wo (i.e., the one obtained for μ = 0) according to the formula below:
|| X WXI2
X
(5)
Wo WI
where À serves as hyperparameter controlling the adjustment. This step crucial because different layers of large language models exhibit substantially different norms of the weight matrices W.
OOr code is available at httttttt///tthubbbbbb won//uuaaariiia///o....
7

---

(fp16) calibration matrices X, and their products WX, see, e.g.. [9 Moreover, we compare these two strategies, analyzing how the metric depends on adaptive and non-adaptive choices of μ across layers, see Figure The Mistral-7B-Instree model was selected due to its pronounced variation of layer-wise norms.
ahpiic:
not-adopiie
58
acccuracy,
56
54
10
10
109
regularization hyperparameter
Figure 4: Comparison of the impact of parameter tuning with (Equation 55) and without considering layer-wise norms on model quality at 70% compression, evaluated on common-sense reasoning dataset using the Mistral-7B-Instruu model.
Figure 5) presents sensitivity analysis parameter demonstrating that the optimal value of ja remains relatively stable (in the region from to 10) across different settings, including various model architectures, datasets, and compression ratios.
50
accuracy,
accuracy,
45
Misrrl 7B
85
60
Misurl 7B
35
45
10
10
10
10
I0P
2
Figure 5: The dependence of average accuracy the parameter À on the commonsense reasoning dataset for different models. On the left: 70% compression ratio, on the right: 80% compression ratio.
Table 2 compares several methods without adaptive rank selection under conditions. We observe that our more numerically stable formulations improve performance, with regularization providing the most consistent gains. We compare use fine-tuning or adaptive rank selection. However, our solution can be potentially used not only as standalone compression technique, but also integrated into other works as part of a problem-solving framework.
method to approaches that do not
Finally, Table compares our approach with state-of-the-art methods that report the relevant metrics their manuscripts. Our method achieves comparable or even superior results solely due the use of regularization, without any additional heuristics fine-tuning.

---

Table Metric values of various compression methods. All computations, except for solving the weighted low-rank approximation problem, were performed in half precision (fp16). Experiments were conducted using the L. 11-2IIr-nsttrct model compressed 90% using text samples from
the commonsense reasoning dataset, which was also used for validation.
Method
boolQ
PIQA
WiNoG
HSwag
ARC-E
ARC-C
OBQA
Original
69.5
74.4
59.5
60.7.0.5
63.24..9
38.1421
34.621.4
ASVD
58.0
52.5+1.0
51.321.3
27.8
30.040.9
25.942.1
26.8=144
SVD-LLM
54.1
60.6#10
53.8=18
34.620.5
44.3
25.5+2.1
26.0=1.4
COALAy
57.6
60.941.0
53.2
34.6.0.5
43.440.9
27.3
26.0
COALA, 59.0+0.7
62.8+1.0
54.0+1.3
36.6+0.5
46.2+0.9
29.2+-2.1
27.6+1.44
Table 3: Metric values of various compression methods. Experiments were conducted using the
Mistral-7B model the WikiText2 dataset and commonsense reasoning used for validation. The results for SliceGPT 22 and FLAP [1] were taken from the work [25].
Ratio
Method
MMLU
BoolQ
PIQA
WiNoG
HSweg
ARC-E
ARC-C
OBQA
100%
Mistral-7B
62.50
83.98
82.05
73.95
81.02
79.55
53.92
44.00
FLAP
25.90
62.26
72.31
64.09
55.94
51.05
31.91
36.80
SliceGPT
28.60
37.86
60.66
59.43
45.10
48.15
30.03
32.00
80%
SVD-LLM
41.80
68.29
73.39
68.43
61.75
71.31
10.53
36.60
SoLA
44.20
66.09
73.67
68.75
63.32
69.99
39.76
39.20
COALA
41.20
78.07
77.04
68.82
65.06
72.13
43.43
40.20
FLAP
26.40
65.26
69.59
64.80
55.61
48.91
30.55
35.80
SliceGPT
25.00
37.83
54.41
51.62
32.54
35.02
22.95
26.80
70%
SVD-LLM
28.20
64.62
64.91
64.17
47.36
58.25
30.72
34.20
SoLA
33.80
62.57
68.39
64.48
53.00
60.90
32.76
37.60
COALA
27.35
63.82
70.40
62.43
51.02
63.63
35.49
36.00
We conducted experiments on the models LLaMA3-SB, LLaMA3-IB [6] and Mistral-7B [5] (including Insrtuct versions), comparing our approach with existing methods across various datasets: boolq 8 OpenbookQA [35], WinoGrande (38), HellaSwag [5T] Are_e 1. Are_e [TO], PIQA (4) MMLU [8]. We used A100 GPU and Tesla T4 GPU for our experiments. The results indicate that
all the considered settings our regularized algorithm systematically achieves better metrics during compression.
6.2 Fine-Tunie
Table 4: Results of fine-tuning LLaMA3-IB--nstmnn at rank r = using different PEFT initializa-
tion methods on the commonsense reasoning dataset with 24 examples for initialization. In exact arithmetic, "COALA a 2" is equivalent to CorDA. See hyperparameters in Appendix
Method
BoolQ
PIQA
SIQA
HSweg
WiNoG
ARC-e
ARC-c
OBQA
Avg.
LoRA PiSSA
64.5
76.1
71.5
82.4
53.8
76.8
58.5
68.2
75.0
64.5
76.0
71.5
83.0
52.0
78.4
60.8
70.4
75.4
CorDA
61.4
62.1
60.8
52.4
68.7
40.1
52.8
60.9
COALA & 2
64.4
75.9
72.6
82.7
54.3
78.2
59.5
68.0
75.4
COALA a
61.1
76.1
72.8
82.8
56.0
77.5
59.8
68.4
75.5
Training and fine-tuning models with specific constraints or regularization applied to the weights
has proven to be an effective technique in recent years 23 40
Fine-tuning methods often utilizes
concept of low-rank matrix approximations for initialization, see PiSSA (33 and CorDA [47 appraoches. We investigate the application of our method for initializing LoRA A 23 adapters and demonstrate advantages. The following proposition unifies these methods and also leads to new method for I.

---

Proposition 4. The solution to the optimization problem
min tr ( (WX-W'(XX) T
(6)
rank WW
for an arbitrary a 2 0, a E 2, is given by the formula:
W = U,U, W, where UEV = W(XX))
and U, consists of the first r columns of the matrix U.
Proof. Note, that
Lr - -W'" W))) 1(W - 1
where (XXT) = S is such a square positive definite matrix that SST = (XXT) )o. Thus, applying Proposition we obtain the desired solution.
Note that to obtain (XXT) one does not have to compute XXT explicitly. One possible strategy to take the SVD of X: UEVT and then (XXT)$ = U.
Remark 1. For a = 2, the task (6) becomes equivalent to the following minimization problem:
min tr ((W XW (S
min rank(WW))r
11W ((WWW))) I(W-W)XXX
Thus, applying Corollary 7. arrive at the solution
W' = U,E.V, (XXT)-1
where UEVT T and U, Ex,, V truncated matrix. This solution is presented algorithm in the CorDA method.
By applying our Proposition]] 7. obtain another way of solving this problem:
W U,U, W
where U, consists of the first r left singular vectors of the matrix W W
We show that the solution provided by the CorDA method solves the problem described in (6), when and also applied our formulas for robustness purposes. Without them, in some scenarios, inversions of XX raised runtime errors due to singular matrices or lead to large numerical errors. Note also that for a the minimization problem (6) leads the PiSSA method. We conduct experiments on the LLaMA3- 1B-Instruct [15] model. Table
4 Juggesst that the robustified version of CorDA (COALA, a 2) significantly boosts the performance. Both robust versions for a 1 and
2 yield results similar to PiSSA, though 1 performs slightly better.
7 Limitations
The limitations of work closely linked the applicability and effectiveness of the weighted approximation approach. Thus, its efficiency is limited to tasks and domains where these methods perform well.
Conclusion
In conclusion, we have presented new, regularized inversion-free framework for context-aware low-rank approximation of LLM. We aimed to address the issue of numerical instability seen in previous works, and developed solutions for challenging scenarios such as large calibration matrices exceeding GPU memory capacity and near-singular input activation matrices. In our experiments, we observed favorable results in both model compression and fine-tuning scenarios compared to previous methods.
10

---

Acknowledgments
The work was supported by the grant for research centers in the field of Al provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C31 3925P4E0002 and the agreement with HSE University 139-15-2025-009, The cal- culations were performed in part through the computational resources of HPC facilities at HSE University [26].
The authors are also grateful to A. Osinsky for insightful suggestions that led to an improved theoretical bound.
References
[1] Yongqi An. Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-bass adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865-10873, 2024.
[2] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, In ICLR. OpenReview.net, 2024.
[3] Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2020.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[5] Devendra Singh Chaplot. Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel. guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut lavril, thomas wang, timothée lacroix, william el sayed. arXiv preprint arXiv:23100 (tt.:23110068/0 06825, 2023.
[6] Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsich. Drone: Data-aware low- rank compression for large nlp models. Advances in neural information processing systems, 11:29321-29333 2021.
[7] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. 10 2017.
[8] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: (8X3::8803.055577 2018.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.054555 2018.
[11] James Demmel, Laura Grigori, Mark Hoemmen, and Julien Langou. Communicationnopiinnal parallel and sequential qr and lu factorizations. SIAM Journal on Scientific Computing, 34(1):A206AA299, 2012.
[12] James W Demmel. Applied numerical linear algebra. SIAM, 1997.
[13] G.H. Golub, Alan Hoffman, and G.W. Stewart. A generalization of the eckart-young- matrix approximation theorem. Linear Algebra and its Applications, 88-89:317-322 1987.
11

---

[14] Mikhail Gorbunov, Kolya Yudin, Vera Soboleva, Aibek Alanov, Alexey Naumov, and Maxim Rakhuba. Group and shuffle: Efficient structured orthogonal parametrization. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024.
[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey et al. The llaa herd of models. arXiv preprini, arXiv:2407.22/78,, version 3, 2024.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407:21783, 2024.
[17] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[19] Stefan Hex and Turn Trout. Residual stream norms grow exponentially over the forward pass. htwp://wwwogggoooo//////////naaa
2023.
[20] Roger A Horn and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994.
[21] Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022.
[22] Yen- Ven-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In ICLR. OpenReview.net, 2022.
[23] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[24] Wei Huang. Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for 1lm.. In ICML. OpenReview.net, 2024.
[25] Xinhao Huang, You-Liang Huang, and Zeyi Wen. Sola: Leveraging soft activation sparsity and low-rank decomposition for large language model compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 17502, 2025.
[26] PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. Hpc resources of the higher school of economics. In Journal of Physics: Conference Series, volume 1740, page 012050. IOP Publishing, 2021.
[27] Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Adasvd: Adaptive singular value decomposition for large language models. CoRR, abs/2502.01403. February 2025.
[28] Zichang Liu, Jue Wang, Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22176. PMLR, 2023.
[29] Raphael Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. 10 2017.
[30] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In NeurIPS, 2023.
12

---

[31] Jonathan H Manton, Robert Mahony, and Yingbo Hua. The geometry of weighted low-rank approximations. IEEE Transactions on Signal Processing, 51(2):00--514, 2003.
[32] Ivan Markovsky. low-rratd approximation and its applications, Automatica, 44(4):891-
909, 2008.
[33] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PISSA: Principal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference on Neural Information P Processing Systems, 2024.
[34] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:/609.000 /te.///99000777 07843, 2016.
[35] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 2018.
[36] Scan O' O'Rourke, Van Vu, and Ke Wang. Random perturbation of low rank matrices: Improving classical bounds. Linear Algebra and its Applications, 540:26-59, 2018.
[37] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-225100 10, 2023.
[38] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communicutions of the ACM, 64(9):99-106, 2021.
[39] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[40] Askar Tsyganov, Evgeny Frolov, Sergey Samsonov, and Maxim Rakhuba. Matrix-free two-to- infinity and one-to-two norms estimation. arXiv preprint arXiv:2508. 044444 2025.
[41] Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu. Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdbury, and Mi Zhang. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024. Survey Certification.
[42] Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient IIm finetuning. arXiv preprint arXiv:2406.0000 2024.
[43] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approxima- tion. Advances in Neural Information Processing Systems, 37:54905-54931, 2024.
[44] Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing sin- gular value truncation for large language model compression. arXiv preprint 1233: 2025.
[45] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: truncation-aware singular value decomposition for large language model compression. CoRR, abs/2403.07378, 2024.
[46] Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware singular value decomposition for large language model compression. In The Thirteenth International Conference Learning Representations, 2025.
[47] Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter-efflcciett fine-tuning, In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[48] Zhihang Yuan, Yuzhang Shang, and Zhen Dong. PB-LLM: partially binarized large language models. In ICLR. OpenReview.net, 2024.
13

---

[49] Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: activation-awarr singular value decomposition for compressing large language models. CoRR. abs/2312.05821, 2023.
[50] Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-ars aware singular value decomposition for compressing large language models, 2025.
[51] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 2019.
[52] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303.112222 18223, 2023.
[53] Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. Blockpruner: Fine-grained pruning for large language models. CoRR, abs/2406. 10594, 2024.
[54] Zixuan Zhou, Xuefei Ning. Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. survey on efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024.
[55] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey model compression for large language models. Transactions of the Association for Computational Linguisties, 12:1556-1577, 2024.
[56] Difan Zou, Philip M. Long, and Quanquan Gu. On the global convergence of training deep linear resnets. In International Conference on Learning Representations, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. The abstract and introduction give concise overview of the paper's methods and findings, accurately reflecting its contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Yes. The paper includes all necessary assumptions and offers thorough, correct proofs for each theoretical result.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes. We provide all the necessary information. Moreover, the Appendix Section F contained hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The section with the code repository can found in Section 6, which contains the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligguddeeeeeeeeeeee Cooee///ddeeebmimiii ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The necessary details of the experiments presented in Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we provided error bars for experiments where there was indeterminacy and the possibility provide them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The necessary details of the experiments are presented in Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes. Out research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper has no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes. All used papers properly cited in the text.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release any new assets in our papper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No. The paper does not mention any significant LLM usage in its core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: /2tppp/p./..//ooeeeeeee//////// for what should or should not be described.
21

---

A General Weighted Low-Rank Approximation Problem
Definition A.1 (General Weighted Low-Rank Approximation Problem), Given matrix W E R we aim to find a matrix W of rank most r, such that the objective function
min (W - W' Q vec[W-W"]
(7)
Waaank Wo
minimized, where 3 denotes the vectorization operator, transforming a given matrix into column vector by stacking its columns on top of each other. The matrix Q E Runnxm represents positive definite matrix.
Theorem 2. (From (311) In 7. Q - Q @ Q2 where Q E Rmxx and Q2 E R are both positive definite and symmetric, then the solution W' of 7 is given by the following closed- form expression. Let 1/ 22)
UEVT be the compact SVD, where Q/ is the unique positive definite symmetric matrix such that Q 12 Q Qi and similarly for Q2 1/2 Then, W = Q2 U., 1/2 where E, is obtained from E by setting all but the first v singular values to zero. Here, & is the Kronecker product [20].
Proof. See work 31
Observe that if we choose Q2 I and Q1 XXT, we immediately obtain a solution to the problem (I) More generally, note that any square matrix S satisfying SST = XXX can be employed in this construction. For instance, a standard choice would be the Cholesky factor of XX
Corollary 1. Let W and X be arbitrary matrices belonging to gm and Ba respectively, with X having full row rank.. The solution to the optimization problem formula
be obtained using the
W' = -U=-VVV (xXT))111
where USVT~ ((XXT)u is SVD.
Proof.
Note, that
[ -W II(W - w')x|l ">t))\l==x<(W>
= [tT(AB) vecfA]T vec[
= =voefW- - (oo (WWWWWXXX)
= woof - -ook voc[I(W - -''XXX [vocf = = (CT & vec(B] =
vec(W- Doe[ "VX'TXXx @ 1) WW')
So, if X has full rank, can apply Theorem 2 where Q1 X.T. .Q2=1:
W = 1-1/ (XX 1/2 UE V (XXX)(XXX)XX""
(XXT)-1/2
where UEVT= ((XX
B SVD-LLM Method
In this section, we present pseudocode for several approaches to solve the problem, including the method outlined in Section A an approach leveraging the Cholesky decomposition, and one utilizing the square root of the matrix XXx
The Algorithm 3 from the work 46 provides the solution via Cholesky decomposition for the matrix XXT, while Algorithm from the work 44 finds the solution via the search for symmetric matrix square root of X.X through SVD.
22

---

Algorithm 3 SVD-LLM method 46
Input: W E RRax X E Raxk N
Output:
A Rmxr, E
R2xx
Compute the upper triangular matrix S from the Cholesky decomposition of X.x::
S 4 (xooesskyXXX
Compute the singular value decomposition of WS:
[U, DUE < (Vd(WS)
Let U,, Ey: V Ul: r|, r, r). rl
Set A t U,
Set + E, V.V, S-1
return A, B
Algorithm SVD-LLM V2 method
Input: W E R X E Raxk reN
Output: A Rmxrr B E TRxx
Compute the SVD of XXT.
S, V < S)
Compute M + WU,
Compute the SVD of M:
[U, Σ, V - Svd(M)
Let U,, E, V, r] E r. r], [[
Compute S-1/2
Set A +- Up
Set B + Σ, V, V,T T
return A, B
C Basics of Low-Rank Approximation
This section presents statements of established results well references to their original sources. Although readers may choose to skip this part, serves to provide greater clarity in the subsequent proofs when referring these well-known findings.
Theorem 3. (Eckart-Young- Houng-Mirsly Let A E Rxx have the SVD
A A-UEVT
where E = diag(o 02...., op) with on 2 2 2 op 2 0 and min(m, n). For any integer r with Sr < p, define the rank-r matrix
A, U, Exp V.T
by keeping only the top r singular values o1...., Gr in E, along with the corresponding columns of U and V. Then A, is best rank-r approximation to in the Frobenius norm, i.e.
Ar
arg min ||- 11-BB- rank(B) ST
Moreover, af + ar41, then this best rank-r approximation A, is unique.
Proof. See 13
Corollary 2. The solution to the low-rank approximation problem
min rank(AA]
||A
AslFF,
Sk
can be obtained using the formula A URU Ar A
AV&V. where the SVD of matrix A is
given by
Σ3
Up
Uid
Vx
V V
Σ
23

---

Theorem 4 (Davis-Kanl Kahan- Wedin sin(O) Theorem), Let A E R be matrix such that its r-th and +r+ + 1)-th singular values satisfy or (A) + (+)) Let E E R==x" be a perturbation matrix, and define A = A+ E. Let U, laxrr and U, R2 be matrices whose columns consist of the first r left singular vectors of A and A, respectively. Then,
|,,77 - 0,0," (A( (r)
2
Proof. This result is proved in "Random perturbation of low rank matrices: Improving classical bounds" [36]
Lemma 1. Let A E Rdxr be a rank-r matrix. Then for any B B holds that
Omiu miin (A) I || F S mmx (A) | ||]
Proof. The proof is classical and can be found, e.g., in 56
D Convergence Proofs for the Full-Rank Regularization Problem
Theorem Let W E mRxx and X E RRax Suppose that X has full row rank (i.e,, rank(X) n) W and that the singular values W X satisfy o, (W X) 0y41 (W+1(XX) where ag(:) denotes the i-th
largest singular value. Then, the solution Wo to the problem ① is unique. Furthermore, if W denotes the solution the regularized problem (4) then the following estimate holds:
I 12 ||
I Walle 0, (WX) (WX)
Om (X)
where, || ||2 denotes the spectral norm.
Before we proceed to the proof of Theorem 5 let us establish an auxiliary lemma.
Lemma 2. Let X E R m S n, rank(X)) (X) = m. Then
[(XXT) -xx + ( 11)
20 2xm
Proof. Let UAUT = x define the eigendecoopposition of a symmetric positive definite matrix.
Then, U is orthogonal, and the elements of are positive.
[IxxxT) (XXT + 1/) = | - (UAUT + -UUT))/221 = 11 - (A+ (M+ (1+
III ll2 is unitarily invariant] =
1//1//2 - (A 11)
(VA+Æ-VV) vA).
max XEO(XXT)
Note that
V -x= (V++u VA)(A= VA)
1
< "
VA+u+V VT
VA+W+ 20
Then
[(XXT])112 -XXXT 11)
(VA+H-VA) VV)
max XEO(XXT) 1
S max P (EO(XXT)
11
max
2 oEO(X)
20
20 (X)
Proof of Theorem The uniqueness of Wo follows from the fact that if (W * (rFi(X)) then the rank-r low-rank approximation Y, of the matrix W is unique (by Eckart- Mirsky Theorem 3. Hence, Wo solution if and only if WoX Y,. Morcover, since the kerel of X
24

---

is empty, if such matrix Wo exists, it must be unique. However, by Proposition indeed exists.
I.
such matrix
We now establish the estimate from the theorem's condition. By Proposition
we obtain
Wo UgU0 W
where Uo denotes the first left singular vectors of W Ho. and Ho = (XX 11/2 Analogously, using Proposition we have
W U W
where Un denotes the first left singular vectors of W WHE, and H (XXT + u/) From Lemma Glt follows that
|Ho Hall2 S (xm
Consequently,
| 20 (X) n.
| Ho WH, S W Hnll2 -
By applying Davis-Kahan Theorem
4 we obtain
2|IW Ho WH,ll12
|U 11
of (W 0 0 1
2||
||
20 (X) (W H) 0r+ (WH)) 4 0m (X) H1) or+ (WH) u.
Thus,
W. - = 11000 W - UU W = 11UbU/ - UWU
|W||2||W||
< |UDU0 - -U, |12| |F
0. (X) 0,+1 (W)) 1b
E
Convergence Proofs (Without the Full-Rank Condition)
Proof of Theorem
7. By Proposition
D. we obtain
Wo = Up W,
where Un denotes the first r left singular vectors of WX, X, and Ho = (X2 Analogously, using
Proposition 3 we have
Wp U,U,W,
where U,. denotes the first r left singular vectors of W Hp. and Hμ. (XXT k/))1/2, However, can get the matrices Un and U,, are defined as the matrices of the first r left singular vectors obtained from the singular value decompositions:
Uo XX
Uμ + SVD(W(XX W))
Here we use the fact that the left singular vectors of a matrix coincide with the eigenvectors vectors (same that left singular in this case) of AA
Consider the perturbation of the matrix XXXTWWI
X T - w(XXT + # 1|2 wI|lW 1|2 1l
By Applying Davis-Kahai Theorem 4
obtain
- W(XX # 1
|UOUT Un 1l2 o, (XX WT) (WXX WXX
25

---

Since ox WT - (W(W)) X), this yields
W |UDUR - U 112 x2(W)) - 0f (W X) /.
Combining this bound with Wo UoUo and W U W, we arrive at
W - Welle = 11(U0U0 U Wil
$ 11U0U - -U,U lla I|
o X) 0, (WX) p.
F
Implementation Details
Table 5: Choice of hyperparameters for different methods, which were applied to the matrices Q, K,
V. O, Up, Down.
Hyperparameter
LoRA
PiSSA
CorDA
COALA
Rank
8
Q
12
Dropout
0.0
0.0
0.0
0.0
Optimizer
Adam W
AdamW W
Adam W
Adam W
Learning Rate
x 10-4
x 10-4
x 10-4
x 10-4
LR Scheduler
Cosine
Cosine
Cosine
Cosine
Batch Size
16
16
16
16
Warmup Steps
100
100
100
100
Epochs
Fine-tuning: All training runs were conducted on the same dataset consisting of 40,000 examples, presented in the same order across all experiments. Training single model required approximately 10 hours, with an additional 2 hours allocated for evaluating the response accuracy on the valida- tion dataset. All experiments were performed
NVIDIA Tesla T4 GPU with Driver Version 535.183.01 and CUDA Version 12.2. The parameter a was individually selected for each initialization method since the norms resulting from different initialization methods varied, impacting the gradient norms. See Table 5 for the other parameters.
Compression: We compressed the Q, K, V, O, Up and Down matrices, approximating each of them with the same rank r to achieve the desired parameter ratio.
G Examples
In this section, we present examples supporting various assertions of our work.
Example G.1 (Loss of Precision When Computing the Gram Matrix [12).. When "squaring" matrix and then taking square root, we can lose accuracy in computing its smaller singular values. This phenomenon can be illustrated on the following matrix:
(
1 V8
x =
where & == Em/22, and Em is small positive number, representing the machine epsilon (the smallest number such that + Em o 1 in machine arithmetic).
26

---

First, we compute the singular values of matrix X. The singular values are the square roots of the eigenvalues of X X"X: X:
x = (1
(b
0
(
v²) (1 1+8) 1+8)
1
V8)
det fet(XTx)
Tx X - XI) = 0.
This leads to:
1² (2 + 8) =
A 2 + v (2 + -
2
Thus, the eigenvalues are:
2 2+ + 2
E 8²
A1
=2
+0(=)),
2
2
8
(2 2
A2
2+ E
4
8
8
+
O
2
8
The singular values of X are the square roots of the eigenvalues:
σ1 VA
2
+
E
g²
=
V2
1 +
5
E²
v2
1
1
5²
+ +(s))
2 8
4
16
8
128
02 0= Vx =
E
22
=
3
E
=
=
Ve
1
8
e2
+ 0
2
8
2
4
22
8
128
Finally.
01 V2 v2
+ 0(=2),
8
Ve E
+ (
02 = V2 8/2
However, machine arithmetic, we will obtain:
XXT 1 1).
and also
(X) V2, xx( 0.
As a result,
(x2(X) - -xx)
-
So we lost approximately square root of machine epsilon.
y gap-1 real angle
107
angle
105
10a
10-5
4
10
10
10
10"
gap
Figure 6: Figure illustrating the dependence of the convergence slope angle of regularized models compared to non-regularized models, with all other factors held constant.
27

---

Example G.2 (Dependence on gap-1).
We fixed dimensional parameters, left and right singular vectors of the matrix WX, as well as all singular values except for the r-th and (r + 1) J-th ones. Then, we varied the difference between or (W X) and (y+ X). The convergence rate of the regularized solution to the unregularized solution with respect to this gap is presented Figure [ We observe that the dependence on the gap is intrinsic to the problem and that we catch the correct asymptotic behaviour in our thcoretical bound in the full rank case.
28


Reviews:
[{"review_id": "7ksytCJubn", "reviewer": "Reviewer_GaTG", "summary": "CoTAn aims to preserve the most important directions of the activations. The method computes a low-rank decomposition that approximately minimizes error on the layer’s output activations, rather than weight reconstruction error itself. To achieve this efficiently, CoTAn uses an inversion-free algorithm: instead of explicitly inverting large covariance matrices or solving normal equations, it leverages orthogonal projections and QR factorizations to find the low-dimensional subspace spanned by the data, and then performs a truncated SVD or low-rank factorization in that subspace. This approach avoids  matrix inverses and can be implemented in a distributed manner.", "questions": "- What happens when the context data used for compression is misaligned with test-time data—how robust is the method?\n- Can the authors clarify how CoTAn would be integrated into a full training or fine-tuning pipeline, or is it solely for post-training compression", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n- The paper addresses a meaningful problem in model compression: numerical instability in context-aware low-rank approximations. The authors propose an inversion-free formulation that avoids inverting Gram matrices, the use of QR and TSQR decompositions is well-motivated from numerical linear algebra.\n\n*Weaknesses*\n\n- Related literature is not well addressed. Statement \"Structured pruning removes unnecessary parameters and simplifies the\nmodel architecture without significant loss of accuracy In turn, the low rank decomposition approach is often memory-efficient and can accelerate model inference\" is not correct in general,  compression rate/accuracy drop depends significantly on the data, neural network architecture, way of training, tensor decompositions sometimes can lead to slow down of inference if rank is not small enough or due do increased number of reading-writing accesses to the memory.\n- The paper's  claims like stability and  generalization are not well demonstrated by empirical results", "was_revised": true, "final_justification": "During the rebuttal the authors  addressed my questions  on  misaligned data, as well as method applicability for both post-training and fine-tuning scenarios, thus I intend to raise my score.", "rebuttal": "We sincerely thank you for your time and effort. We appreciate your valuable comments and feedback. \n\n\n\n> (1) Related literature is not well addressed. Statement \"...In turn, the low rank decomposition approach is often memory-efficient and can accelerate model inference ...\" is not correct in general, compression rate/accuracy drop depends significantly on the data, neural network architecture, way of training, tensor decompositions sometimes can lead to slow down of inference if rank is not small enough or due do increased number of reading-writing accesses to the memory.\n\nWe do agree that the efficiency/accuracy trade-off of any compression technique is highly problem-dependent, and also that more more complex structures such as tensor decompositions can additionally slow down computations. We will be sure to highlight these points and also incorporate suggestions from the other reviewers in the related work section to enhance its correctness and completeness.\n\n> (2) What happens when the context data used for compression is misaligned with testtime data -- how robust is the method?\n\nThank you for this question, it is indeed an important aspect to consider. On the one hand, we are in the setting of weighted matrix approximation for compression and our method inherits its properties. On the other hand, we have an additional regularization term. This term serves to penalize deviations from the original (unweighted) matrix, mitigating this problem and, hence, should positvely affect generalization. In the table below we present results with misaligned testtime data. We observe that the regularization parameter does increase the metrics. \n\n\n**Table. 1**  The dependence of the  _LLaMA-3-1B_ model performance on the $\\mu$ hyperparameter at a compression ratio 90\\%, using Wiki2 as compression dataset. The average value of the accuracy on the common reasoning dataset is taken as the model's performance.\n\n| **$\\boldsymbol \\mu$** | 0 | 0.01 | 0.1 | 0.5 | 2 | 5 | 10 | 20 | 50 | 100 |\n|:---------:|:---:|:-----:|:----:|:----:|:----:|:--:|:--:|:---:|:--:|:--:|\n| **Avg.**  | $41.08{\\color{gray}\\small\\pm0.4}$ | $41.15{\\color{gray}\\small\\pm0.4}$ | $41.37{\\color{gray}\\small\\pm0.4}$ | $41.41{\\color{gray}\\small\\pm0.4}$ | $41.61{\\color{gray}\\small\\pm0.4}$ | $42.01{\\color{gray}\\small\\pm0.4}$ | $\\underline{42.04}{\\color{gray}\\small\\pm0.4}$ | $42.03{\\color{gray}\\small\\pm0.4}$ | $40.91{\\color{gray}\\small\\pm0.4}$ | $39.59{\\color{gray}\\small\\pm0.4}$ |\n\n\n> (3) Can the authors clarify how CoTAn would be integrated into a full training or finetuning pipeline, or is it solely for post-training compression?\n\nThank you for your question. We focus on two settings involving weighted low-rank approximation. Beyond post-training compression (Section 6.1), we also demonstrate its use in constructing initializations for fine-tuning (Section 6.2). Due to the generality of our framework, we presume that it can be also useful in other scenarios where weighted low-rank approximation is of interest. Furthermore, it could be combined with other techniques — for example, by integrating different training methods after post-training compression. We consider this an interesting research direction and plan to investigate it further in future work.\n\nAs for the full training procedures, the potential usefulness of weighted low-rank approximation remains unclear for us at this stage. While it might be relevant for low-rank gradient approximations (e.g., as in GaLore), this is speculative rather than empirically validated.\n\n___\n### Additional experiments\n\nWe would also like to highlight that in addition to addressing your specific questions, we have included the results of a new experiment responding to a common concern raised by several reviewers. Specifically, multiple reviewers requested an analysis of the parameter $\\mu$ dependence, which we present in the table below.\n\n**Table 1.** The dependence of the  _LLaMA-3-8B-Instruct_ model performance on the $\\mu$ hyperparameter at different compression ratio values. The average value of the accuracy on the common reasoning dataset is taken as the model's performance.\n\n\n| $\\boldsymbol \\mu$$\\rightarrow $  Ratio $\\downarrow $   | **0** | **0.001** | **0.01** | **0.1** | **0.2** | **0.5** | **1** | **2** | **5** | **7** | **10** | **20** | **100** |\n|:------:|:--:|:-----:|:----:|:-:|:--:|:---:|:-:|:---:|:--:|:--:|:--:|:--:|:--:|\n| **90\\%** | $59.67_{\\color{gray}\\pm 0.5}$ | $59.59_{\\color{gray}\\pm 0.5}$ | $59.94_{\\color{gray}\\pm 0.5}$ | $60.35_{\\color{gray}\\pm 0.5}$ | $60.56_{\\color{gray}\\pm 0.5}$ | $60.76_{\\color{gray}\\pm 0.5}$| $61.07_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $\\underline{61.59_{\\color{gray}\\pm 0.5}}$ | $61.49_{\\color{gray}\\pm 0.5}$ | $59.80_{\\color{gray}\\pm 0.4}$\n| **70\\%**   | $45.39_{\\color{gray}\\pm0.4}$ | $45.44_{\\color{gray}\\pm0.4}$ | $45.75_{\\color{gray}\\pm0.4}$ | $46.27_{\\color{gray}\\pm0.4}$ | $46.18_{\\color{gray}\\pm0.4}$ | $46.67_{\\color{gray}\\pm0.4}$ | $47.13_{\\color{gray}\\pm0.5}$ | $47.25_{\\color{gray}\\pm0.5}$ | $\\underline{47.54_{\\color{gray}\\pm0.5}}$ | $47.44_{\\color{gray}\\pm0.5}$ | $47.16_{\\color{gray}\\pm0.5}$ | $46.68_{\\color{gray}\\pm0.4}$ | $41.00_{\\color{gray}\\pm0.4}$\n| **60\\%** | $41.09_{\\color{gray}\\pm 0.4}$ | $41.04_{\\color{gray}\\pm 0.4}$ | $41.23_{\\color{gray}\\pm 0.4}$  | $41.48_{\\color{gray}\\pm 0.4}$ | $41.62_{\\color{gray}\\pm 0.4}$ | $41.76_{\\color{gray}\\pm 0.4}$ | $\\underline{41.88_{\\color{gray}\\pm 0.5}}$ | $41.85_{\\color{gray}\\pm 0.5}$ | $41.39_{\\color{gray}\\pm 0.5}$ | $41.18_{\\color{gray}\\pm 0.5}$ | $40.60_{\\color{gray}\\pm 0.5}$ | $39.50_{\\color{gray}\\pm 0.4}$ | $36.25_{\\color{gray}\\pm0.4}$\n| **50\\%** | $37.12_{\\color{gray}\\pm 0.4}$ | $37.2_{\\color{gray}\\pm 0.4}$ | $37.28_{\\color{gray}\\pm 0.4}$ | $37.77_{\\color{gray}\\pm 0.4}$ | $\\underline{37.90_{\\color{gray}\\pm 0.4}}$ | $37.51_{\\color{gray}\\pm 0.4}$ | $37.26_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.22_{\\color{gray}\\pm 0.4}$ | $36.64_{\\color{gray}\\pm 0.4}$ | $36.18_{\\color{gray}\\pm 0.4}$  | $35.29_{\\color{gray}\\pm 0.4}$"}, {"review_id": "taJfrOkFWs", "reviewer": "Reviewer_kPsW", "summary": "The authors proposes an approach based on decompositions that is numerically stable compared to methods that relies on Gram matrix and their subsequent inversion. CoTAn introduces an inversion-free solution based on stable matrix decompositions like QR and SVD. The framework is comprehensive, proposing the TSQR algorithm for memory-constrained scenarios and a novel regularization technique for data-scarce settings. The author shows that the method is appliable on large language models namely LLama 3.1 and Mistral 7B and is compared on wide datasets, demonstrating superior performance and stability in compression compared to existing methods.", "questions": "*  To improve the practical utility of the method, could you provide a sensitivity analysis or a plot showing the effect of different µ values on model performance? This would offer valuable guidance for selecting this hyperparameter.\n*  Adaptive rank to compare the methods fairly though it is an important aspect for compression ( L225 - 226) could you please either showcase the overhead those methods adds. Models are compressed once and then  deployed, if the margin in performance is large sometimes it is worth the effort to tune", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n\n* Tackles a Critical Problem: The paper addresses the crucial and practical issue of numerical stability during the compression and fine-tuning of large language models, a significant bottleneck for current methods.\n\n* Comprehensive and Versatile Framework: The proposed solution is not a single trick but a complete framework. The extension to handle memory-constrained scenarios via TSQR and data-scarce situations via regularization makes CoTAn robust and widely applicable.\n\n* Diversified and Strong Evaluation: The method is thoroughly evaluated on multiple modern architectures (LLaMA3-8B, LLaMA3-1B, Mistral-7B) across a wide range of  datasets, providing convincing evidence of its effectiveness. The work also clearly shows how to choose between SVD and QR decomposition for efficiency depending on the hardware and matrix dimensions\n\n\n* Solid Theoretical and Practical Support: The claims are well-supported by clear mathematical propositions and proofs. Furthermore, the inclusion of pseudocode and detailed hyperparameters in the appendix significantly aids reproducibility.\n\n\n### Weakness :\nThe following weaknesses are minor and more for readability \n* Guidance on Regularization Parameter: The method's effectiveness is clearly shown, yet the sensitivity of the performance to the regularization parameter µ is not fully explored. For instance, the paper states with each results the value of μ ,\n > CoTAnμ​ is regularized approach, we get µ = 5 · 10−4\", but the link between this choice and the final accuracy is not established.\"", "was_revised": true, "final_justification": "I stay with my recommendation of accepting the paper. The authors answered my concerns", "rebuttal": "We sincerely thank you for your time and effort. We appreciate your valuable comments and feedback.\n\n> (1) To improve the practical utility of the method, could you provide a sensitivity analysis or a plot showing the effect of different $\\mu$ values on model performance? This would offer valuable guidance for selecting this hyperparameter.\n\nWe appreciate this question. The table below presents the results for different compression ratios. Our current findings suggest that different compression ratios within the same model yield comparable in magnitude optimal values of $\\mu$, with smaller $\\mu$ values tending to correspond to higher compression.  As part of our revision, we will expand this analysis with a more comprehensive ablation study on $\\mu$ and carefully integrate these insights into the manuscript.\n\n**Table 1.** The dependence of the  _LLaMA-3-8B-Instruct_ model performance on the $\\mu$ hyperparameter at different compression ratio values. The average value of the accuracy on the common reasoning dataset is taken as the model's performance.\n\n\n| $\\boldsymbol \\mu$$\\rightarrow $  Ratio $\\downarrow $   | **0** | **0.001** | **0.01** | **0.1** | **0.2** | **0.5** | **1** | **2** | **5** | **7** | **10** | **20** | **100** |\n|:------:|:--:|:-----:|:----:|:-:|:--:|:---:|:-:|:---:|:--:|:--:|:--:|:--:|:--:|\n| **90\\%** | $59.67_{\\color{gray}\\pm 0.5}$ | $59.59_{\\color{gray}\\pm 0.5}$ | $59.94_{\\color{gray}\\pm 0.5}$ | $60.35_{\\color{gray}\\pm 0.5}$ | $60.56_{\\color{gray}\\pm 0.5}$ | $60.76_{\\color{gray}\\pm 0.5}$| $61.07_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $\\underline{61.59_{\\color{gray}\\pm 0.5}}$ | $61.49_{\\color{gray}\\pm 0.5}$ | $59.80_{\\color{gray}\\pm 0.4}$\n| **70\\%**   | $45.39_{\\color{gray}\\pm0.4}$ | $45.44_{\\color{gray}\\pm0.4}$ | $45.75_{\\color{gray}\\pm0.4}$ | $46.27_{\\color{gray}\\pm0.4}$ | $46.18_{\\color{gray}\\pm0.4}$ | $46.67_{\\color{gray}\\pm0.4}$ | $47.13_{\\color{gray}\\pm0.5}$ | $47.25_{\\color{gray}\\pm0.5}$ | $\\underline{47.54_{\\color{gray}\\pm0.5}}$ | $47.44_{\\color{gray}\\pm0.5}$ | $47.16_{\\color{gray}\\pm0.5}$ | $46.68_{\\color{gray}\\pm0.4}$ | $41.00_{\\color{gray}\\pm0.4}$\n| **60\\%** | $41.09_{\\color{gray}\\pm 0.4}$ | $41.04_{\\color{gray}\\pm 0.4}$ | $41.23_{\\color{gray}\\pm 0.4}$  | $41.48_{\\color{gray}\\pm 0.4}$ | $41.62_{\\color{gray}\\pm 0.4}$ | $41.76_{\\color{gray}\\pm 0.4}$ | $\\underline{41.88_{\\color{gray}\\pm 0.5}}$ | $41.85_{\\color{gray}\\pm 0.5}$ | $41.39_{\\color{gray}\\pm 0.5}$ | $41.18_{\\color{gray}\\pm 0.5}$ | $40.60_{\\color{gray}\\pm 0.5}$ | $39.50_{\\color{gray}\\pm 0.4}$ | $36.25_{\\color{gray}\\pm0.4}$\n| **50\\%** | $37.12_{\\color{gray}\\pm 0.4}$ | $37.2_{\\color{gray}\\pm 0.4}$ | $37.28_{\\color{gray}\\pm 0.4}$ | $37.77_{\\color{gray}\\pm 0.4}$ | $\\underline{37.90_{\\color{gray}\\pm 0.4}}$ | $37.51_{\\color{gray}\\pm 0.4}$ | $37.26_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.22_{\\color{gray}\\pm 0.4}$ | $36.64_{\\color{gray}\\pm 0.4}$ | $36.18_{\\color{gray}\\pm 0.4}$  | $35.29_{\\color{gray}\\pm 0.4}$\n\nAs for the different models, we observe that they exhibit distinct scaling properties in the matrices $W$ and $X$, which directly influences the optimal choice of $\\mu$. While our scale-dependent heuristics produce $\\mu$ values that improve model performance, we find that further tuning yields additional gains. We will also be sure to study this aspect in more detail and conduct more thorough ablation studies regarding $\\mu$ in our revised manuscript.\n\n\n> (2) Adaptive rank to compare the methods fairly though it is an important aspect for compression (L225 - 226) could you please either showcase the overhead those methods adds. Models are compressed once and then deployed, if the margin in performance is large sometimes it is worth the effort to tune.\n\nWe agree that adaptive ranking is an important aspect of compression. Unfortunately, there is no definitive answer regarding the overhead that these methods introduce, as they vary significantly and may require different computational times. For example, the method from work [1] emperically chooses rank based on error approximation in the Frobenius norm, so the choice of rank is not a computationally demandning step. On the other hand, work [2] presents a differentiable method to solve for optimal truncation positions, which requires 8 GPU hours for the LLama-7B model. In our work, we wanted to study how different algorithms for compressing matrices affect quality, so we eliminated the interfering factor using a single shared strategy. Nevertheless, as you noted, while the extended initialization time due to rank adaptation may be justified by the model's improved deployment performance, combining our method with these techniques can  certainly be beneficial. We plan to explore optimal strategies in future work.\n___\n[1] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. \"SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression.\" . In The Thirteenth International Conference on Learning Representations (2025).\n\n[2] Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. \"Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives.\" . In The Thirteenth International Conference on Learning Representations (2025)."}, {"review_id": "YhG03iXx69", "reviewer": "Reviewer_FrEc", "summary": "This paper introduces CoTAn -- a numerically stable and inversion free framework to minimize the optimization problem $$\\text{minimize}_{W' : \\text{rank}(W') \\leq r} ||WX - W'X||_F,$$ where $X$ is a calibration data matrix. This problem is referred to as the *context-aware low-rank approximation* problem in the paper. Unlike prior approaches mentioned (e.g., SVD-LLM, ASVD), CoTAn avoids computing and inverting the Gram matrix of the calibration data, i.e., $X^\\top X$, which is often ill-conditioned or singular in practice. Hence, the name, \"numerically stable\" in the title of the paper.\n\nThis is done using stable matrix factorizations such as QR and SVD decompositions directly on the transformed input data (rather than the Gram matrix) -- optionally combined with regularization to handle ill-posed or underdetermined settings. The paper also discusses an efficient Tall-Skinny QR decomposition (TSQR) for scalability to large matrices, and proves convergence and error bounds for the regularized variant. Experiments on Llama-3 family of models and Mistral, and 8 downstream evaluation tasks (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC-e, ARC-c, OBQA) show that CoTAn achieves comparable or better performance with greater stability and faster runtime, when compared to ASVD and SVD-LLM.", "questions": "I have some questions and would appreciate it if they are addressed:\n\n1. Can the authors comment on how sensitive CoTAN is to the choice of the regularization parameter $\\mu$? Is there a principled way to choose it (for example, via calibration)? Such ablation studies would be helpful in understanding the role of a proximal regularizer that penalizes deviation, such as in (4).\n\n2. Any ablation study with the choice of rank? \n\nPlease also refer to other important concerns in the previous strengths and weaknesses section.", "limitations": "Yes. Would appreciate if authors would comment of the limitation that low-rank approximation by itself leads to significant accuracy drop.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\nThe paper address a relevant problem in the field of LLM compression -- for compressing weight matrices in an LLM, it is not sufficient to preserve just the Frobenius norm error of the weight matrix, but rather the output activations of each layer. Consequently, several LLM compression methods have resorted to using a small calibration dataset for this purpose, and this paper studies low-rank approximation with a calibration dataset. The paper addresses a practical issue regarding matrix inversion errors while inverting (near) singular Gram matrices, while providing clean mathematical formulations, correctness proofs, and convergence guarantees for the same. Numerical experiments also validate the theoretical claims.\n\nIn terms of clarity, the paper, in general, seems well-organized and technically sound with a clear motivation for each design choice. The pseudocode and figures also help in understanding and are highly appreciated. Some important derivations are delegated to the Appendix, but could benefit from an intuitive explanation in the main text. \n\n**Weakness**:\nI believe the paper can do a better job placing itself in the context of some relevant works. For instance, [this paper](https://proceedings.neurips.cc/paper_files/paper/2024/hash/a20e8451ffb07ad25282c21945ad4f19-Abstract-Conference.html) is quite relevant and adopts a similar approach to solve the context-aware low-rank approximation problem. How does CoTAn compare with the approach presented in this paper? For instance, Lemma 4.2 (and its complete derivation in Appendix B) of this paper, also solves the same problem, without inverting any singular matrix. The rank-constrained regression problem (eq. 5 in this paper ) seems to be the same problem as CoTAn solves (up to some change in variables). The authors seem to have missed discussing this in related works. \n\nMoreover, low-rank decomposition of the weight matrix, by itself, introduces a very large gap in downstream accuracies with respect to the uncompressed baselines. This is true for all methods like ASVD, SVD-LLM and including CoTAn. This often renders low-rank factorizations not-so-useful in practical application where accuracy requirements are paramount. Some discussion needs to be added about how CoTAn is compatible with other methods that can help reduce the accuracy gap between low-rank factorized and uncompressed models.\n\nI would be happy to revise my score if my concerns are addressed.", "was_revised": true, "final_justification": "I have raised my score in light of the author's rebuttal, which is satisfactory.", "rebuttal": "We sincerely thank you for your time and effort. We appreciate your valuable comments and feedback.\n\n\n> (1) How does CoTAn compare with the approach presented in this paper? For instance, Lemma 4.2 (and its complete derivation in Appendix B) of this paper, also solves the same problem, without inverting any singular matrix.\n\nThank for poitining out this interesting work. We will definitely add this study to our literature review. We note that in Lemma 4.2 the authors still invert singular values of the matrix $X$, which can numerically be very close to zeroes (see, for example, Figure 2 in our work): \n$$ Z_\\star \\triangleq \\mathop{\\operatorname*{argmin}}_{\\operatorname{rank}( Z)\\le k} \\|\\|  X Z- Y \\|\\|_F^{2} = ( V  I_m \\boldsymbol \\Sigma^{-1}  {\\acute{U}}  I_k) ( I_k^{\\top} {\\acute{\\Sigma}} {\\acute{V}}^{\\top}),$$\nwhere $\\boldsymbol \\Sigma := {\\tilde \\Sigma } I_m \\in \\mathbb{R}^{m \\times m}$ \nis a diagonal matrix consisting of the non-zero singular values of $X$. At the same time our approach is fully free from any matrix inverse.\n\n> (2) Can the authors comment on how sensitive CoTAN is to the choice of the regularization parameter $\\mu$? Is there a principled way to choose it (for example, via calibration)? Such ablation studies would be helpful in understanding the role of a proximal regularizer that penalizes deviation, such as in (4).\n\n\nWe thank the reviewer for this valuable suggestion. \nThe table below presents the results for different compression ratios. Our current findings suggest that different compression ratios within the same model yield comparable in magnitude optimal values of $\\mu$, with smaller $\\mu$ values tending to correspond to higher compression. As part of our revision, we will expand this analysis with a more comprehensive ablation study on $\\mu$ and carefully integrate these insights into the manuscript. \n\n\n**Table 1.** The dependence of the  _LLaMA-3-8B-Instruct_ model performance on the $\\mu$ hyperparameter at different compression ratio values. The average value of the accuracy on the common reasoning dataset is taken as the model's performance.\n\n\n| $\\boldsymbol \\mu$$\\rightarrow $  Ratio $\\downarrow $   | **0** | **0.001** | **0.01** | **0.1** | **0.2** | **0.5** | **1** | **2** | **5** | **7** | **10** | **20** | **100** |\n|:------:|:--:|:-----:|:----:|:-:|:--:|:---:|:-:|:---:|:--:|:--:|:--:|:--:|:--:|\n| **90\\%** | $59.67_{\\color{gray}\\pm 0.5}$ | $59.59_{\\color{gray}\\pm 0.5}$ | $59.94_{\\color{gray}\\pm 0.5}$ | $60.35_{\\color{gray}\\pm 0.5}$ | $60.56_{\\color{gray}\\pm 0.5}$ | $60.76_{\\color{gray}\\pm 0.5}$| $61.07_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $\\underline{61.59_{\\color{gray}\\pm 0.5}}$ | $61.49_{\\color{gray}\\pm 0.5}$ | $59.80_{\\color{gray}\\pm 0.4}$\n| **70\\%**   | $45.39_{\\color{gray}\\pm0.4}$ | $45.44_{\\color{gray}\\pm0.4}$ | $45.75_{\\color{gray}\\pm0.4}$ | $46.27_{\\color{gray}\\pm0.4}$ | $46.18_{\\color{gray}\\pm0.4}$ | $46.67_{\\color{gray}\\pm0.4}$ | $47.13_{\\color{gray}\\pm0.5}$ | $47.25_{\\color{gray}\\pm0.5}$ | $\\underline{47.54_{\\color{gray}\\pm0.5}}$ | $47.44_{\\color{gray}\\pm0.5}$ | $47.16_{\\color{gray}\\pm0.5}$ | $46.68_{\\color{gray}\\pm0.4}$ | $41.00_{\\color{gray}\\pm0.4}$\n| **60\\%** | $41.09_{\\color{gray}\\pm 0.4}$ | $41.04_{\\color{gray}\\pm 0.4}$ | $41.23_{\\color{gray}\\pm 0.4}$  | $41.48_{\\color{gray}\\pm 0.4}$ | $41.62_{\\color{gray}\\pm 0.4}$ | $41.76_{\\color{gray}\\pm 0.4}$ | $\\underline{41.88_{\\color{gray}\\pm 0.5}}$ | $41.85_{\\color{gray}\\pm 0.5}$ | $41.39_{\\color{gray}\\pm 0.5}$ | $41.18_{\\color{gray}\\pm 0.5}$ | $40.60_{\\color{gray}\\pm 0.5}$ | $39.50_{\\color{gray}\\pm 0.4}$ | $36.25_{\\color{gray}\\pm0.4}$\n| **50\\%** | $37.12_{\\color{gray}\\pm 0.4}$ | $37.2_{\\color{gray}\\pm 0.4}$ | $37.28_{\\color{gray}\\pm 0.4}$ | $37.77_{\\color{gray}\\pm 0.4}$ | $\\underline{37.90_{\\color{gray}\\pm 0.4}}$ | $37.51_{\\color{gray}\\pm 0.4}$ | $37.26_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.22_{\\color{gray}\\pm 0.4}$ | $36.64_{\\color{gray}\\pm 0.4}$ | $36.18_{\\color{gray}\\pm 0.4}$  | $35.29_{\\color{gray}\\pm 0.4}$\n\n\nAs for the principled way to choose it, we naturally observe that for different models the parameter $\\mu$ shows dependence on the scales of $W$ and $X$ (as is also supported by our theoretical estimates). The matrix norms can be computed prior to the compression step and as part of our revision, we will expand this analysis with more comprehensive ablation studies on $\\mu$ and carefully incorporate these findings in the manuscript.\n\n> (3) Any ablation study with the choice of rank?\n\nWe thank the reviewer for this question. Our experiments examining the relationship between compression (for each layer rank is determined by the compression ratio) and optimal $\\mu$ values revealed that lower compression ratios corresponded to smaller optimal $\\mu$ values. This observation aligns with the intuition that reduced parameter counts (from lower ranks) may decrease susceptibility to overfitting on calibration data. Hence, regularization is less necessary in these cases. Please see the table with optimal $\\mu$ below:\n\n**Table 2.** The dependence of the optimal $\\mu$ on compressing ratio.\n\n| **Ratio, \\%** | 90 | 70 | 60 | 50 |\n|:----------------:|:---------:|:--:|:--:|:--:|\n|**Optimal $\\boldsymbol \\mu \\boldsymbol \\approx$**| 10 | 5 | 1 | 0.2 |"}, {"review_id": "RyL7Kj9Uyc", "reviewer": "Reviewer_D85U", "summary": "This paper proposes a numerically stable and efficient framework to perform block-wise context-aware low rank approximation on LLMs. Specifically, the paper proposes a QR decomposition-based inverse free optimization algorithm an a regularization to avoid overfitting. Experiments are conducted on multiple LLMs across different tasks.", "questions": "Please provide experimental results on the effect of using different $\\mu$ in Equation (4)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strength\n1. The paper provides a novel inversion-free regularized optimization framework to solve the low-rank approximation problem of LLMs\n2. The theoretical derivations provided in the paper are clear and solid. The paper has a high writting quality and is easy to follow\n3. Adequate experimental results are included to show the improvements compared with baseline methods.\n\n## Weakness\n1. Low rank approximation generally leads to a worse performance-efficiency tradeoff compared to pruning and quantization. To this end, though the proposed method improves over existing baseline, it has limited significance on pushing the compression state-of-the-art forward. It is interesting to see if the proposed approximation algorithm or derivations can be used on quantization or pruning as well.\n2. The effectiveness of the regularization $\\mu$ is worth analyzing. Though the paper provides an theoretical derivation on the impact of $\\mu$, an experimental study on different choices of $\\mu$ is preferred", "was_revised": true, "final_justification": "I'm satisfied with the author's response. After checking all reviews and rebuttals, I'm confident in my judgement that this paper is solid and should be accepted.", "rebuttal": "We sincerely thank you for your time and effort. We appreciate your valuable comments and feedback.\n\n> (1) It is interesting to see if the proposed approximation algorithm or derivations can be used on quantization or pruning as well.\n\nThank you for this comment. While we acknowledge the important tradeoffs compared to alternative approaches, we believe that our numerically stable framework for weighted low-rank approximation can be useful for developing hybrid compression strategies. For instance, reviewer `FrEc` points out the work [1], which combines quantization and low-rank approximation for compression. This approach also opens up an interesting new research direction for our work.\n\n> (2) Please provide experimental results on the effect of using different $\\mu$ in Equation (4).\n\nThank you, this is indeed an important question to address. The table below presents the results for different compression ratios. Our current findings suggest that different compression ratios within the same model yield comparable in magnitude optimal values of $\\mu$, with smaller $\\mu$ values tending to correspond to higher compression. As part of our revision, we will expand this analysis with a more comprehensive ablation study on $\\mu$ and carefully integrate these insights into the manuscript.\n\n**Table 1.** The dependence of the  _LLaMA-3-8B-Instruct_ model performance on the $\\mu$ hyperparameter at different compression ratio values. The average value of the accuracy on the common reasoning dataset is taken as the model's performance.\n\n\n| $\\boldsymbol \\mu$$\\rightarrow $  Ratio $\\downarrow $   | **0** | **0.001** | **0.01** | **0.1** | **0.2** | **0.5** | **1** | **2** | **5** | **7** | **10** | **20** | **100** |\n|:------:|:--:|:-----:|:----:|:-:|:--:|:---:|:-:|:---:|:--:|:--:|:--:|:--:|:--:|\n| **90\\%** | $59.67_{\\color{gray}\\pm 0.5}$ | $59.59_{\\color{gray}\\pm 0.5}$ | $59.94_{\\color{gray}\\pm 0.5}$ | $60.35_{\\color{gray}\\pm 0.5}$ | $60.56_{\\color{gray}\\pm 0.5}$ | $60.76_{\\color{gray}\\pm 0.5}$| $61.07_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $61.47_{\\color{gray}\\pm 0.5}$ | $\\underline{61.59_{\\color{gray}\\pm 0.5}}$ | $61.49_{\\color{gray}\\pm 0.5}$ | $59.80_{\\color{gray}\\pm 0.4}$\n| **70\\%**   | $45.39_{\\color{gray}\\pm0.4}$ | $45.44_{\\color{gray}\\pm0.4}$ | $45.75_{\\color{gray}\\pm0.4}$ | $46.27_{\\color{gray}\\pm0.4}$ | $46.18_{\\color{gray}\\pm0.4}$ | $46.67_{\\color{gray}\\pm0.4}$ | $47.13_{\\color{gray}\\pm0.5}$ | $47.25_{\\color{gray}\\pm0.5}$ | $\\underline{47.54_{\\color{gray}\\pm0.5}}$ | $47.44_{\\color{gray}\\pm0.5}$ | $47.16_{\\color{gray}\\pm0.5}$ | $46.68_{\\color{gray}\\pm0.4}$ | $41.00_{\\color{gray}\\pm0.4}$\n| **60\\%** | $41.09_{\\color{gray}\\pm 0.4}$ | $41.04_{\\color{gray}\\pm 0.4}$ | $41.23_{\\color{gray}\\pm 0.4}$  | $41.48_{\\color{gray}\\pm 0.4}$ | $41.62_{\\color{gray}\\pm 0.4}$ | $41.76_{\\color{gray}\\pm 0.4}$ | $\\underline{41.88_{\\color{gray}\\pm 0.5}}$ | $41.85_{\\color{gray}\\pm 0.5}$ | $41.39_{\\color{gray}\\pm 0.5}$ | $41.18_{\\color{gray}\\pm 0.5}$ | $40.60_{\\color{gray}\\pm 0.5}$ | $39.50_{\\color{gray}\\pm 0.4}$ | $36.25_{\\color{gray}\\pm0.4}$\n| **50\\%** | $37.12_{\\color{gray}\\pm 0.4}$ | $37.2_{\\color{gray}\\pm 0.4}$ | $37.28_{\\color{gray}\\pm 0.4}$ | $37.77_{\\color{gray}\\pm 0.4}$ | $\\underline{37.90_{\\color{gray}\\pm 0.4}}$ | $37.51_{\\color{gray}\\pm 0.4}$ | $37.26_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.15_{\\color{gray}\\pm 0.4}$ | $37.22_{\\color{gray}\\pm 0.4}$ | $36.64_{\\color{gray}\\pm 0.4}$ | $36.18_{\\color{gray}\\pm 0.4}$  | $35.29_{\\color{gray}\\pm 0.4}$\n\n___\n\n[1] Saha, Rajarshi, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. \"Compressing large language models using low rank and low precision decomposition.\" Advances in Neural Information Processing Systems 37 (2024): 88981-89018."}]

Author rebuttal:

Meta-review: This work addresses the inversion of gram matrices in context aware low rank approximation. It is based on the observation that these matrices are highly ill-conditioned and thus, following conventional wisdom in numerical linear algebra, the formation of these matrices should be avoided to preserve numerical accuracy. The authors propose using classical techniques from numerical linear algebra, such as QR factorization and the TSQR algorithm that instead operate on the square root of the gram matrix. The authors provide extensive numerical evidence that this modification yields moderate but significant improvements in practice. The reviewers commend the clarity of the proposed insight and the thoroughness of the evaluation. While it was pointed out that the improvements could be more striking, the paper seems to bring a valuable novel perspective to the compression and fine-tuning of large language models. Thus, I recommend acceptance
Decision: Accept (poster)