From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Authors: AJAY KUMAR JAISWAL
Yifan Wang
Lu Yin
Shiwei Liu
Runjin Chen
Jiawei Zhao
Ananth Grama
Yuandong Tian
Zhangyang Wang
Venue: ICML 2025 poster (2025)
Keywords: Large language models
Gradient subspace
Memory-efficient training
Optimization
Low Rank Compression
Primary area: deep_learning->foundation_models
TL;DR: Our work provides a theoretical perspective of gradient subspace stabilization and present WeLore as a novel framework for LLM compression and memory-efficient finetuning.

Abstract: Large Language Models (LLMs) matrices can often be expressed in low-rank format with potential to relax memory and compute resource requirements. Unlike previous works which pivot around developing novel matrix decomposition algorithms, in this work we focus to study the emerging non-uniform low-rank properties across weight matrices in LLMs through the lens of stabilizing gradient subspace. \textit{Firstly,} we provide a theoretical framework to understand the stabilization of gradient subspaces through Hessian analysis. \textit{Secondly,} we empirically establish a consequential relationship between the gradient dynamics and low-rank expressiveness of weight matrices. Our findings reveal that different LLM components exhibit varying levels of converged low-rank structure, necessitating a non-uniform rank reduction across them to minimize performance drop due to compression. In view of that, we present \textit{Weight Low-Rank Projection} \textbf{(WeLore)} that unifies weight compression and memory-efficient fine-tuning as ONE, in a data-agnostic and one-shot way. Going beyond only as a compression technique, WeLore categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank Components (N-LRCs) based on their ability to express themselves as low-rank. Our gradient dynamics perspective illustrate that \textit{LRCs tend to have better finetuning capabilities} and their standalone finetuning can closely mimic (sometimes outperform) the training loss trajectory and performance of full-finetuning with notable memory and compute footprint reduction. All codes and checkpoints will be released.

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Ajay Jaiswal" Yifan Wang" 2 Lu Yin
Shiwei Liu Runjin Chen
Jiawei Zhao Ananth Grama
Yuandong Tian 7 Zhangyang Wang'
Abstract
1. Introduction
Large Language Models' (LLMs) weight matri- ces can often be expressed in low-rank form with potential relax memory and compute resource requirements. Unlike prior efforts that focus on developing novel matrix decompositions, in this work we study the non-uniform low-rank prop- erties of weight matrices LLMs through the lens of stabilizing gradient subspace. First, we provide a theoretical framework to understand the stabilization of gradient subspaces through Hes- sian analysis. Second, we empirically establish an important relationship between gradient dynamics and low-rank expressiveness of weight matrices. Our findings reveal that different L.MM compo- nents exhibit varying levels of converged low- rank structures, necessitating variable rank reduc- tion across them minimize drop in performance due to compression. Drawing this result, we present Weight Low-Rank Projection (WeLore) that unifies weight compression and memory- efficient fine-tuning into one, in data-agnostic and one-shot manner. When used as a compres- sion technique, WeLore categorizes weight ma- trices into Low-rank Components (LRCs) and Non-Low-rank Components (N-LRCs) and suit- ably encodes them for minimum performance loss. Our gradient dynamics perspective illustrates that LRCs tend to have better finetuning capabilities and their standalone finetuning can closely mimic and sometimes outperform the training loss tra- jectory and performance of full-finetuning with notable memory and compute footprint reduc- tion. Codes are available at https: // ////ttttu./////////// /t//VVVAAGGoou//// Lolee
Large language models (LLMs) designed with large number of parameters to capture complex patterns and re- lationships in data. However, despite their huge size with billions of parameters, these models' weights often exhibit low-rank structures in practice, offering potential to relax memory and compute resource requirements. While many efforts have focused on developing novel matrix decomposi- tions for efficient training (Hu et al., 2021b; Dettmers et al., 2023; Meng et al., 2024; Biderman et al., 2024b; Lialin et al., 2023a) and model compression (Hsu et al., 2022b; Kaushal et al., 2023b; Li et al., 2023; Jaiswal et al., 2024; Wang et al., 2023b), the underlying mechanisms of how these low-rank structures emerge remain insufficiently understood.
A fundamental question arises: Why do the weights of LLMs tend become low-rank? Prior efforts have proposed vari- ous explanations one perspective focuses the inherent distribution of training data, which may reside on low- dimensional manifold (Timor et al., 2022: Ergen & Pilanci, 2021; Ongie & Willett, 2022). For instance, when data lie on a d-dimensional manifold, the weight matrices of a two-layer ReLU network converge to a rank of most d (Ongie & Willett, 2022). Another explanation examines the optimization dynamics, where certain training procedures (e.g., SGD with weight decay) naturally favor convergence to low-rank regions (Ji & Telgarsky, 2020; Le & Jegelka, 2022; Galanti et al., 2024).
In this work, we approach the low-rank phenomenon through the lens of gradient subspace stabilization. Gra- dient dynamics during pretraining plays a crucial role in shaping the weight subspace, acting like a stream that grad- ually sculpts the weight landscape. Through theoretical analysis of gradient subspace stabilization via Hessian anal- ysis, we establish a fundamental connection between gra- dient dynamics and the emergence of low-rank structures in LLMs.A key insight from our analysis is the presence of distinct Hessian gap, indicating the emergence of low- rank subspace in the weight space. Through accompanying experiments, we identify two significant findings:
'Equal contribution Universiiy Texas at Austin PPurdue Uni- versity "Eindhoven University of Technology Uniiversity of Surrey 2 Uniierssiyy of Oxford Caaiioomia Institute Technology "Meta AL Conrespondence to: Ajay Jaiswal <ajayjaiswale @ udexas.edu>>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Different components of L.MMs exhibit varying degrees

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
of low-rank convergence: I MLP Up/Down Projec- tions and Self-Attention Value Projections demonstrate unclear Hessian gap and slow gradient subspace convergence, resulting in high-rank weights. 2 Self- Attention Query/Key and MIP Gate Projections dis- play clear Hessian gaps, and gradient subspaces settle rapidly, resulting in low-rank weights.
2. Gradient-Hessian Eigenspace Alignment to Low Rank Weights: Theory and Observations
In this section, we present a theoretical framework for un- derstanding the stabilization of gradient subspaces through Hessian analysis. We provide insights into the convergence properties of Hessian eigenspaces, showing that the gradient subspace aligns with the dominant directions in the Hessian. This analysis is crucial for understanding the emergence of low-rank structures in LLMs.
2.
The low-rank convergence patterns vary across differ- ent network depths: ( Middle layers exhibit small Hessian gaps and slow gradient subspace convergence, resulting in high-rank weights. @ Early and late lay- ers have clear Hessian gaps and rapidly evolve into low-rank subspaces, resulting in low-rank weights.
2.1. Preliminaries
We denote the training loss as L(W), where W E R is the set of model parameters. Its gradient is denoted by VL(W) E Rmmx, and the Hessian, as a linear operator, is denoted by H(W)= = (W) E Rmnxm
Based on these observations, we categorize weight matri- ces into two groups: ① Low-rank Components (LRCs): Matrices exhibiting high-quality low-rank structure (charac- terized by heavy-tail in sorted singular values) whose gradi- ents carry rich error signals from data. 2 Non-Low-rank Components (N-LRCs): Matrices with non convernve low- rank structure (missing heavy-tail in distribution of singular values) that cannot be effectively low-rank factorized.
We adopt the standard SGD update
W = W₂ - VL(W))
with learning rate 7j >
These conclusions lead our proposed Weight Low-Rank Projection (WeLore) approach, which unifies weight com- pression and parameter-efficient fine-tuning as one:
The following analysis relies on several assumptions, which summarize here and present in detail in Appendix A.1), We assume the Hessian is Lipschitz continuous, which is justified by the smoothness of common loss like cross-entropy. The loss function satisfies the Kurdyka- Lojasiewicz (KL) condition near optimum, generalizing con- vexity to the nonconvex setting. We also require uniform spectral gap between dominant and non-dominant eigen- values of the Hessian, which naturally emerges from the dominance of task-relevant directions in neural networks. Finally, we assume the model architecture is reversible (Tian et al., 2021), ensuring bounded norms.
functions
1. Compression perspective (WeLore-COMP) : LRCs with stabilized low-rank structure can be factorized by SVD achieve significant compression ratios.
2. Parameter-Effice Efficieet Finetuining perspective (WeLore-PEFT) During fine-tuning, we selectively update only LRCs in their low-rank decomposed format while keeping N-LRCs frozen, leading to effective gradient progress with reduced memory footprint.
2.2. Hessian Eigenspace Stabilization
Our first important theoretical result shows that both the eigenvalues and eigenspaces of the Hessian converge during training.
WeLore operates a one-shot, data-agnostic manner, per- forming layer-wise non- non-uniform rank reduction based on the heavy-tail property of normalized singular values. This approach allows LRCs to support higher rank reduction while preserving N-LRCs at full rank or minimal reduc- tion, ensuring minimal performance degradation. Notably, WeLore achieves remarkable efficiency gains: compared to fillffietunnng a 50% compressed LLaMA-2 7B model, WeLore requires only 35% of trainable parameters, delivers 3x better throughput, and reduces GPU memory usage by 40%. Most importantly, our extensive experiments across both continual finetuning (Figure 5) and downstream tasks (Figure 6) demonstrate that this selective LRC-based fine- tuning can match or even outperform full model finetuning, establishing WeLore as a powerful unified solution for both model compression and efficient adaptation.
Theorem 2.1. Let H; = LWW(W, be the Hessian of the loss L(W) at time t. Under standard assumptions, the eigenvalues and eigenspaces of H, stabilize as t + co. Specifically:
! The eigenvalues converge with step-to-step changes de- caying to zero;
2 The top-r eigenspace converges with vanishing step-to- step changes.
Proof Sketch. The proof follows three key steps. First, we bound the step-to-step changes in the Hessian using Lipschitz continuity: ||Hll S Tu - Will =
2

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Norrulee Cornviti Soacturr
Nortel lead D3 pareul an Sovetrum
Serraling liv Cpenyte Spectrar
Porrakie I serveii Soucturs
Herma lesd Doomata Swetham
So-mall levt Chamr Seactrar centtettir..nr
pille bocted rage 2d .rsce Jo Caw.ttttaaaa.....ooo. L might
by vig Modis prodel arara 1.1...11.....
(ide Laited 5y raggnn pxde Joar 22 www...ttiiiinnno..
Fdle soced zoddl lepen In www. A8g5
1:3c by ring Molll model com T% mid21
100
20
300
340
266 350 366
410
30
250 314
4.0
Figure 1. Hessian gap across layers and components of LLaMA2-7B. 7B. We observe that: 1) exhibit less pronounced Hessian gaps compared to self k. .-pjoj, aelf.attn..-pa p-roj, self.attn.o proj, and mip.gate-pree proj: 2) early and late layers generally display clearer gaps than middle layers; 3) components with pronounced Hessian gap (self attn.k proj. self attn.q proj. self. attm.o proj, and mlp.gate proj) tend to be more low-rank, as shown in our experiments.
mlp.up-pmm proj.ppppp.papoppo..e mlp.down.pro proj. and self attn. p-roj
bL (WW)ll Second, we leverage the KE condition to show that the gradient norm decay follows a power law: [[V)( < C/10/(1-9), This gives us ||H < LLAC/E/(-0) 0)_ Finally, we use Weyl's inequality to bound eigenvalue changes and the Davis-Kahai theorem to con- trol eigenspace shifts, showing both converge as series 11/19(1-0) for 0x > 2. Full proof in Appendix A.2. O
2.4. Hessian Eigenspace Dynamics and Gradient Subspace Stablization
The above analysis gives us a key insight connecting gradi- ent subspace stabilization with Hessian properties. A clear spectral gap, defined the difference between the domi- nant and non-dominant eigenvalues of H,, facilitates rapid gradient alignment with principal subspaces. Components with a clear spectral gap in their Hessian eigenvalues stabilize rapidly, while layers with a flatter spectrum exhibit slower or no stabilization.
2.3. Gradient-Hessian Eigenspace Alignment
Building on the stability results, we show that gradients naturally align with the principal eigenspaces of the Hes- sian, explaining why gradients eventually operate in a low- dimensional subspace.
In Figure 1, we observe the second-order derivative of the loss L(W) with respect to model parameters W for each layer's component. Components such as query, key, and gate projections benefit from this property, as their roles inherently induce low-rank structure in the Hessian. In con- trast, value and down projections lack this gap due to diffuse gradient contributions, compression effects, and weaker sig- nals, resulting in flatter spectra. Additionally, we find that early and late layers tend to develop clear Hessian gaps, whereas middle layers typically show flatter spectrum.
Theorem 2.2. Under the conditions above, the gradient vector G, = Vw L (Wi) asymptotically aligns with the principal eigenspace of the Hessian. Specifically:
[((-) Gell =0
lim 10000
||G/ll
where Un spans the dominant eigenspace of H₁-
The following discussion illustrates how a component's 's role and position in the Transformer architecture affect Hessian stablization:
Proof Sketch. The proof analyzes how gradients evolve in the eigenspace basis of the Hessian. First, we decom- pose the gradient update using Taylor expansion: G++1 = G. - HHHG, + O Next, we project this onto the non-dominant subspace (I - ) and use the spec- tral gap condition >; S A, - 7 for j > r to show the non-dominant component contracts:
Self-Attention Q/K Projections These layers exhibit a low-rank structure, as softmax reweighting in attention only few tokens contribute significantly to the attention scores. These strong Hessian spectral gaps allow gradients to align with principal subspaces and stabilize rapidly.
-U,U,)) This geometric contraction,
<
- (1- - combined with bounded ||Gr]| from reversibility, proves that the gradient asymptotically aligns with the dominant eigenspace. Full details are presented in Appendix A.3. D
Self-Attention V Projection By contrast, the value pro- jection processes token content after attention scores are determined, yielding a denser Hessian without a clear spec- tral gap. This diffuse spectrum hinders gradient alignment
3

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Syye
Taver
Higgh Eieepac Wwr.1
382
1908
5000
1309
13000
205
1509
1989
2000
50
10
Figure 2. (Row 1) Gradients subspace similarity obtained from various checkpoints during pretraining of LLaMA 130M on C4 dataset for 25,000 training steps using Adam Optimizer. (Row 2) Emergence of Low-rank Weight Subspace during pretraining of LLaMA-130M. Each row of individual subplot represents the singular values of weights in a given training step.
and delays stabilization.
sponding emergence of weight low-rank structures through- out pretraining within different layers of LLLMa-130M us- ing C4 daatset. Our findings are summarized as:
MLP Gate Projections Activations like GELU selec- tively amplify dominant features, creating sparse gradients that concentrate in few principal directions. Consequently, the Hessian is low-rank with a strong spectral gap, enabling swift gradient convergence.
We find the emergence of low-rank structure across the weight matrices very early during pretraining, which becomes explicit and notable as pretraining progresses.
Similar to our gradient subspace observations, we find that not all layers can express themselves as low-rank and this property significantly varies subject position (middle layers terminal layers) and role (attention layers or MLP layers).
The down projection compresses nigh-dimensional features back to the model's base dimension, flattening the Hessian spectrum and diluting the strong-ge gaaonggtt directions. This lack of clear spectral gap slows or prevents stabilization.
MLP Up and Down Projections
We empirically find a strong correlation between the gradient subspace stabilization and the low-rank emer- gence across the weight matrices (e.g., the absence of a clear spectral gap that prevents stabilization within model 1 . laeer.5 5mm weight matrix not converging to low-rank')
Layer- Wise Trends Early layers stabilize rapidly due to simpler input representations and the low-rank structure of embeddings and positional encodings. Late layers also stabilize effectively, as gradients become dominated by task- relevant features propagated from the loss function. In con- trast, middle layers exhibit delayed stabilization, as they ag- gregate and mix features from multiple layers, introducing gradient diffusion. Additionally, middle layers are more af- fected by attenuation from softmax and LayerNorm, which further delays stabilization.
mmpp........ p. down proj reflects in the
Next, we investigate whether our low-rank findings are valid for the weight matrices within a publicly available LLM checkpoint. Figure 3 presents the 4096 normalized singular values corresponding to different layers across 32 transformer blocks of LLaMa-2 7B. can be clearly observed that some layers (e.g. se self_attn.g.pt self festn.k.prrt clicit heavy tail behaviour indi- cating better low-rank expressivity compared to others (e.g., up-proj, mlp down.proo).. Another important observation of note is that majority of the layers from the front and tail blocks of the model tend to have better low- rank property, which consistent with our gradient behavior
2.5. Weight Low-rank Dynamics in Pre-trained LLMs
Our aforementioned spectral gap analysis of Hessian eigen- values illustrates the non-uniform stabilization of gradient subspace. Provided the pretrained LLMs weights are essen- tially accumulation of gradients over the course of pretrain- ing, an important question to ask is: How do the low-rank properties of LLM weights correlate with their correspond- ing gradient dynamics stabilization? In this section, we study how gradient subspace stablization observations trans- late to the emergence of -ow-rank structures in the weight matrices of the model. Figure (row 2) presents the corre-
'A sharp bright line across the subplots in Figure 2 (row 2) to the left suggests heavy-tail distribution of singular values. A heavy-tail singular value distribution is favorable property that indicates the matrix be well compressed using few singular vectors without introducing large reconstruction errors.

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
analysis. Heavy tail indicates that only a small fraction of singular values carries maximum information and the corre- sponding matrix can be well approximated using fraction of basis vectors from SVD with small reconstruction error.
multiplication kemels. This is unlike sparsity and quanti- zation, which require specialized kernels, often optimized for each hardware platform for best performance. Recently several efforts (Hsu et al., 2022a: Kaushal et al., 2023a; Yuan et al., 2023; Wang et al., 2023a; Saha et al., 2023; Wang et al., 2024) have explored matrix factorization of LLMs' pretrained weights. These efforts primarily focus on improving SVD using more informative signals like acti- vation, Fisher information and applying it uniformly (same rank reduction ratio) across all the weights. As discussed in previous section, low-rank emergence varies significantly across candidate weights in a pretrained checkpoint. To this end, we pose a relatively under-explored question: How can we carefully curate a layer-adaptive rank reduction ratios for all layers in the pretrained checkpoint?
Layer Name: seif, aton.q proj
Layer Name: self attn.k pra)
1.0
1.0
Blacc 31 2
3.1, 30 31
0.8
0.8
2-22
0.6-
0.6
0.4-
0.4
0.2
tireshiid (k) 0:175
0.2
Orestell 10
0.0
1050
2050
0.0
2000 Rank
1050
Name: mio.uu proj
Name:
Block
0.5
Alore 2-24
0.8
Stock
0.0-
0.6
0.4
0.4
WeLore-COMP (WeLore for compression) is data- agnostic and implemeenaaiion-riienddly normalized singular value thresholding technique² with only one global hyper- parameter (threshold k), as shown as the shaded red and green region in Figure for layer-adaptive rank reduction. Specifically, we preserve normalized singular values greater than the threshold k, shown as shaded green region. For given effective rank reduction ratio of E ERR, the global threshold k can be approximated using linear search" over np. 1 linspace (0, 1, 0. 005) with condition as fol- lows:
02
Hrrash i (k 0.122
02
Ur oid 0.75
0.0
1030
2010
3000
4003
0.0
1000
2005
3000
4036
Ranc
Rank
Figure 3. Normalized singular values of the weight matrices cor- responding different layers of LLaMa-2 7B pretrained check- point from HuggingFace. Each subplot indicates 4096 sorted and normalized singular values corresponding to layer (e.g.. sel selffattn.. weriinnn.ppp. jro from 32 transformer blocks.
Nuno Only LKCa Only
N-LDCa Only LRCS Only
19
Eusum sum (Sw, <k) ~ ERR
(I)
E, len (Sw,)
Here, Wi represents the weight matrix of layer I and Sw; is the array of sorted normalized singular values estimated with torch. svd (W)). Note that k estimation is not com- putationally expensive, since Sw, Vi can be calculated be- fore searching for k. Given weight matrix W-4096 4096 and Sw. {si, 82, ..... $409e J. the compressed rank can be computed r np. sum (Sw, 2 k). In com- pressed format, W-1096 1096 can be represented as a com- position of two small matrices A 4096xr and B x 4096 where r << 4096. As observed in Figure 3, for k = 0.175, which indicates an aggregated 50% rank reduction, majority of the self-attar j-pro from 32 transformer blocks of LLaMa-7B can undergo significant reduction 2 90% (i.e., 400). On the other hand, layers such as mlp up-pro & mlp. jown_projj, which are not low-rank friendly, re- ceive high r.
2000
4000
9200
8000
10000
2002
4000
5030
5000
12030
Iniiiig Steps
raiiing Stops
2000
LACK
Thruppp 10330e
25-
2200
Mww
1.5
10
500
50x Only
IACS Dny
2002
.000
6930
5000
15030
Figure 4. Finetuning statistics and performance comparison Low Rank Components (LRCs) and Non-Low-Rand Components (N- LRCs) layers of 50% compressed LLaMa-2 7B model with C4. Note that all finetuning hyperparameters are kept same in both settings for fair comparison.
3. Adaptive Low-rank Weight Projections: Applications
Given r, for all layers / in the pretrained checkpoint, WeLore categorizes all the layers into two broad categories- Low- rank Components (LRCs) and Non-Low-ramt Components
3.1. WeLore: Low-rank Compression Technique
NNommalization helps us to compare singular value distribution across all layers at the same scale.
Among multiple techniques for LLM compression, low-rank decomposition of pretrained weights as a product of two smaller dense matrices receives special attention because it can leverage highly optimized floating-pone paaing dense matrix
Effecttve Rank Reduction (ERR): Σi rank(W wwonk(WCooppenee Compressed,
Σ, rank(w Original
*Pseudo-code for estimation is provided in Appendix D.
5

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
LLaMa2-7B [PPL: 7.03]
LLaMa2-13B [PPL: 6.53]
Mistral-7B (PPL: 8.141
Rank Reduction
Uniform Reduction
OWL Reduction
WeLore-COMP Reduction
Uniform Reduction
OWL Reduction
WeLore-COMP Reduction
Uniform Reduction
OWL Reduction
WeLore-COMP
Reduction
10%
10.58
12.11
7.13
7.17
7.2
6.55
12.31
11.63
8,76
20%
16.43
14.49
8.28
8.61
8,53
6,96
78.69
NaN
11.90
30%
91.99
NaN
14.41
13.99
11.63
8.66
6746.48
NaN
30.69
40%
NaN
NaN
78.17
1178.03
56.06
24.92
162301.04
NaN
429.08
50%
NaN
NaN
1836.62
4167.79
7984.39
1142.53
248042.97
NaN
1351.32
Table Perplexity (C4 dataset) comparison of LLaMa-2 7B, 13B and Mistral-7B pretrained checkpoints compressed with WeLore-COMP, Uniform, and outlier-sensitive rank reduction ratios.
(N-LRCs). Layers with heavy-tail which can be effectively represented with ri < 0.5x rank (W)) fall in LRCs, while the rest fall in N-LRCs, We replace weight matrices of all LRCs in pretrained checkpoint by composition of two small matrices A & B to achieve notable parameter reduction (e.g., x 0.67 parameters with R = 0.5. saving memory and compute during inference and fine-tuning (low-rank weight representation allows gradients and optimizer states to be in low-rank during finetuning).
~ 3x more trainable parameters, training loss as well as the validation perplexity of finetuning N-LRCss are significantly under-performing in comparison to finetuning LRCs. More- over. it is important note that the throughput achieved by LRCs is 2x in comparison to N-LRCs, which can be attributed to the parameter--fficient low-rank represented weight matrices, gradients, and optimizer state.
4. Experiments and Analysis
3.2. WeLore: Parameter- Efficceett Finetuning Technique
In this section, we first estblish the superiority of WeLore's layer-adaptive rank reduction ratio for effective low-rank compression of pre-trained checkpoints of LLMs. Next, demonstrate the effectiveness of WeLore for LRCs-focused parameter efficient finetuning performance across several downstream tasks. Unlike prior efforts that either focus on low-rank compression or marameter-efficient finetuning, WeLore uniquely differentiates itself by proposing an ef- fective low-rank compression strategy and presents a novel angle of memory and parameter-efficient fine-tuning us- ing LRCs while yielding performance comparable to full- finetuning.
Parameter-Efficieet finetuning techniques (PEFT) which en- able LLMs to perform a new task with minimal updates has received enormous attention to their ability to allow fine- tuned by only updating a small number parameters. Unlike LoRA and its varients which finetune a small added fraction of parameters to original pretrained weight checkpoints not relevant to original pretraining optimization, WeLore pro- vides an alternative approach by capitalizing the gradient perspective to select a small fraction of weights from the pretrained model which can undergo fine-tuning. As dis- cussed above, LRCs exhibits low-rank structure with rich gradient dynamies while N-LRCs can't 't be well-expressed in low-rank format. To this end, WeLore-PEFT make the following proposal:
4.1. WeLore for Compression of Pre-trained LLMs
We evaluate the overall performance of the WeLore-COME non-uniform rank reduction ratio from three aspects: ! perplexity-basee performance evaluation in comparison with uniform rank reduction and outlier-sensitive rank re- duction; 2 performance evaluation on real-world tasks (e. summarization, factoid-QA QA,, and multi-tum conversation), 3 performance comparison of WeL-rr-CCOM when com- bined with activation-guided SVD unlike conventional SVD of LLM weight matrices. Additional empirical GPU mem- ory requirement statistics for varying compression ratios are provided in Appendix E. to highlight WeLore usefulness.
Given a low-rank compressed checkpoint with LRCs and N-LRCs, finetuning with backpropagation only through LRCs (frozen N-LRCs) can closely mimic the performance of full full-finetnning (sometimes better) with considerable memory and compute reduction. Note that LRCs are rep- resented in low-rank format, both gradients and optimizer state will by default in low-rank saving finetuning cost.
Empirical evidence that LRCs are better at learning than N-LRCs: Here, we investigate the relative difference in performance and quantify compute cost related to fine- tuning LLMs. Figure presents a comparison of continual finetuning statistics of I.LaMa-7B pretrained checkpoint with 50% effective rank reduction ratio on C4 dataset for 10,000 training steps. Red color indicates finetuning by back-propagaie propppappggginng only through LRCs (freezing all the N- LRCs) while magenta color indicates finetuning N-LRCS (freezing LRCs). It can be clearly observed that despite
Firstly, low-rank decomposition of LLMs has been primar- ily investigated with unilateral (same rank) reduction across all the weights. In contrast, WeLore-COMM presents non- uniform rank reduction ratio guided by emerged low-rank structures in pretrained checkpoints. Table 1 presents the comparison of perplexity of LLaMa-2 7B, 13B and Mistral- 7B model checkpoints on C4 validation dataset with EER
6

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Tull Fineuunng (&CC NLRC))
Freeuting Galare tricurity
2200
Tiretunts nn
*ull cutore "icsunng (UX. (UKS)) jonly
Gasare MirSUun
1500
Tohing
2003
4020
2000
8:00
10000
2003
4020
5000
50000
10000
PuBl Galoore
Vetere
2000
4032
6200
8000
20020
Triniing Shaps
maining Skaps
Traning Skops
Figure 5. Continual-Fineturn With exactly same hyperparamler configrations. WeLore-PEFT can outperform full-finettning with merely ~35% % of trainable parameters. while providing ~3x better throughput.
statistics and performance comparison of a 50% low-rank compressed LLaMa-2 7B pretrained checkpoint.
30.0
250
*ull Pacuung LOBA Fisturing
27.5
3255
2.5
Wezsre Rrituring
25.0-
300
2.0
22.5
275
2.5
250
27.5
22.5
15.0
260
6.5
22.5
17.3
0.0
20.0
15.0
202
400
920
900
1003
F.4 Fg LAA
Wetore
Fuil Anduring
10%
We.ore
Fill Aintuuig
Lone
Weture
ningiing Steps
Figure 6. Downstream Finetuning statistics and performance comparison of WeLore vs. full-finetuning and LoRA of 50% compressed LLaMa-2 7B model with Strategy QA dataset with max_len 512. All finetuning hyperparameters are kept identical for fair comparison.
Method
Factoid-bused QA
Multi-tum Conversation
In-conteat Summarization
Model
Method
10%
20%
30%
40%
50%
(25% Compression)
LLaMa-7B
No Finetune
7.13
8.28
14.41
78.17
1836.62
Full Model
79.02
7.61
8.15
ore.or-PEFT
7.15
7.40
8.18
9.47
11.87
Uniform Reduction
34.63
4.88
5.01
LLaMa-13B
No Finetune
6.55
6.96
8.66
24.92
1142.53
OWL Reduction
54.42
5.27
5,16
NeLFrr-PEFT
6.55
6.68
7.42
8.69
11.40
SVD-LLM
71.95
6.12
Mistral-BB
No Finetune
8.76
11,90
30.69
429.08
1351,32
WeLore-COMP Reduction
71.89
6.09
6.46
WeLore-PEFT
8.32
8.92
9.71
14.85
21.37
Table 2. Performance comparison of LLaMa-7B compressed (25% ERR) using WeLore-COMP non-uni non-uniform rank reduction ratio on real-world benchmarking tasks.
Table 4. Perplexity (C4 dataset) comparison of LLaMa-7B with
SoTA compression techniques at different reduction ratios.
LLaMa2 7B [1x]
LLaba2
13B [1x]
Method
30%
40%
Wel areM
60%
78%
306
50%
60%
70%
SVD>
10,58
16.43
91.99
NaN
NaN
Compressed Parans
0.85x
0.67x
0.56x
0.45x
0.83x
0.64x
0.53x
0.43x
ASVD
7.24
7.75
8.85
11.33
17.03
+ -LRRA Finetuning
8.1
12.48
21.23
362.2
7.49
21.53
27.90
124.44
SVD-LLM
7.45
7.58
7,98
10.04
16.61
Gallm Finetuning
9.00
18.57
396.05
670.29
8.00
60.07
2454.07
3396 19
WeL.xe-COMP ASVD
7.05
7.21
7,87
9.75
14.76
WeLLor PEET
17,87
47.92
7.42
11.40
19.20
73.59
Table Perplexity (C4 dataset) comparison of LLaMa-7B with SoTA compression techniques at different reduction ratios.
Table 5. Performance (perplexity) comparison of compressed LLaMa-2 7B 13B with WeLore-COMP and continual finetuning with LoRA and GaLore w.r.t. WeLore
of 10% to 50%. It can be clearly observed that as EER in- creases, the perplexity of the baseline compressed model sig- nificantly explodes, but WeLOre-COMP retains the perplex- ity within a reasonable range. For example, WeLore-COMP is ~ 6.4 x better than 30% Uniform EER for LLaMa-2 7B and ~ 47 x better than 40% Uniform EER for LLaMa-2 13B. Note that OWL reduction tends to perform sometimes better than Uniform reduction, but its degradation in perfor- mance with increasing EER is more severe.
baselines. However, it slightly underperforms SVD-LLM for factoid-based question answering task but it is imporatnt to note that SVD-LLM relies caliberation dataset. Our further studies indicate that CeLore-COMP can be achieve significantly higher performance when it is augmented with advanced activation guided SVD techniques.
Thirdly, Activation guidded SVD techniques (Yuan et al., 2023; Wang et al., 2024) have been found more effective than weight-oriented SVD methods by managing activation outliers and adjusting the weight matrix based on the acti- vation distribution. Despite our work focusing on simple weight SVD to enable easy adaptation and minimize sen- sitivity to calibration datasets, we conducted experiments to illustrate that WeLore-COMP can also significantly ben-
Secondly, to further understand the effectivenss of WeLore- COMP in real-world task settings, we consider three popular tasks (factoid-QA, in-context summarization, and multi-turn conversation) following the settings described in Appendix D. For open-ended conversation and in-context summariza- tion tasks, WeLorr-CCMP significantly outperforms all

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Method
CommonsenseQQ
SVAMP
BoolQ
CoinPlip
BigBench"
StrategyQA
Dense Full Finetune
77.052
40.672
88.189
75.000
83.742
69.581
Dense LoRA Finetune
76.414
50.090
70.962
69.333
80,995
68.690
Dense GaLore Finetune
75.339
41.667
68.362
65.667
77.980
67.325
Full Finetune
75.925
40.667
84.005
51.333
83.364
70.783
30%
LoRA
64.537
44,333
81.776
61,333
68.750
65.255
WeLore-COMP
GaLore
64.015
42.667
80.892
55.333
75.735
62.490
+ WeLore-PERL
76.744
53,333
85.040
98,667
81,818
69.648
Full Finetuning
71.908
38.333
83.603
49.000
90.224
68,502
40%
LoRA
54.386
36.667
75.021
54.667
76.002
65.154
Wel.oreCCMM
Gal.ore
52.078
36,333
71.039
50.333
77.910
65.440
PeLore-PEFT
76.003
42,667
81,646
98.666
87.857
67.794
+ Full Finetuning
70.120
25.333
80.113
53,333
89.431
63.411 1
50%
LoRA
35.382
23.667
75,482
50.667
54.022
62.408
WeLore-COMP
+ GaL.re
35.122
21.667
71.552
47.667
58.975
61.336
WeLore-
70.516
30.667
80.377
94.666
87.802
67.290
Table Downstream performance of Dense and WeLore compressed LLaMa-2 7B checkpoint under fin-ttnnntnng along with memory-
efficient finetuning techniques (LoRA and GaLore). All downstream finetuning is performed starting from the same initial checkpoint state for fair comparison.
efit from Activation-SVV Table presents the perplexity comparison of SoTA low-rank compression techniques (in- cluding SVD-LLM and ASVD which rely calibration datasets) wrt. WeLore- COMP non-uniform layer-wise ratios when augmented with calibration dataset.
training token budget (0.7 million) from C4 dataset and keeping all other hyperparameters indetical for fair compar- ison. Table 5 illustrates the superiority of LRCs-focused WeLore-PEFT. which achieves ~8x and ~1.7x at 70% compression ratio for LLaMa-7B and 13B models.
How well WeL.rr-PEFF performs for finetuning on benchmarking downstream tasks? To understand the ef- fectiveness of LRCs-only WeLore-PEFT finetuning, we consider full-parameter finetuning. LoRA, and GaLore for dense pretrained checkpoint, as well as Wel.re compressed checkpoint of LLaMa-7B. We conduct several experiments across various compression ratios on math and common- sense reasoning tasks and report our performance in Table 6. Surprisingly, LRCs-based finetuning of compressed models tends to closely match and sometime outperform even the dense, as well as compressed full full-parameter finetuning of LLaMa-7B pretrained checkpoint. Additionally, the per- formance achieved by WeLore-PEFT is significantly and consistently higher than both LoRA and Gal .ore across all the tasks with memory requirements close to LoRA. Figure shows that unlike LoRA, WeLore-PEFT closely mimics the loss trajectory GPU memory requirements and can achieve throughput greater than LoRA based fine-tuning.
4.2. WeLore for Memory-elficient Finetuning of Pre-trained LLMs
In this section, we investigate the effectiveness of Wel.rre- PEFT, which focuses on selective finetuning of LRCs com- ponents to answer three important questions:
I How well can WeLore-PEFT recover the language mod- eling capabilities of compressed LLMs? We investigate the performance characteristics of LRC--fcused in recovering compressed pretrained checkpoints perplexity of LLaMa-2 7B, 13B and Mistral 7B. More specifically, given a pretrained checkpoint, we first perform rank re- duction using WeL.rr-CCMM with varying ERR between 10-50%. Table presents the C4 (evaluation set) perplexity of SVD weight matrices. Note that with increasing ERR, model perplexity is notably high. Next, perform limited continual finetuning of fraction of LRCs using C4 dataset with sequence length of 1024 on 0.7M tokens. Interestingly, it can be observed from Table 4, that WeLore-PEFT can significantly recover the perplexity of compressed check- points within reasonable range, where benefits increase with higher compression ratios.
folltfinetuning with significantly lower
5. Conclusion
In this work, we focus on the emerging non-uniform low- rank properties across weight matrices LLMs through the lens of stabilizing gradient subspace. We provide a theoretical framework to understand the stabilization of gra- dient subspaces through Hessian analysis. We empirically demonstrate consequential relationship between gradient
② How does WeLore-PEFT compare with conventional LLM finetuning techniques like LoRA and GaLore? To inves- tigate the effectiveness of Wel.ore-PEFFT in comparison with two popular resource-efficient finetuning methods (LoRA and GaLore), we designed controlled experiments with fixed
8

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
dynamics and low-rank expressiveness of weight matrices. We present WeL.oreCCOME adaptive layer-wise low-rank compression strategy for low-rank decomposition, which achieves high compression with minimal drop in perfor- mance. The unique proposition of Wel.ore lies in catego- rizing weight matrices of pretrained models into two broad categories LRCs and N-LRCs based on their low-rank structure. We conduct extensive experiments validate that LRCs yield better trainability than N-LRCs. Given lim- ited compute & memory budget, WeLore-PEFT proposes finetuning of LRCs while keeping N-LRCs frozen with back-propagation for maximal gain (sometimes better than .illtninetni
L. Qlora: Efficient finetuning of quantized Ilms, 2023. URL BtRtttt/:////xxtiii
2200. 14314
Diao, S., Huang, Z., Xu, R., Li, Lin, Y., Zhou, X., and Zhang, T. Black-box prompt learning for pre-trained lan- guage models. arXiv preprint arXiv:2201 1.0831,,,,,, 2022.
Ergen, T. and Pilanci, M. Revealing the structure of deep neural networks via convex duality, 2021. URL https: / wwtpxii.wrggbb///.ee 2002 09773.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference Machine Learning, pp. 10323- 10337. PMLR, 2023.
Impact Statement
Galanti, T., Siegel, Z. S., Gupte, A., and Poggio, T. Sgd and weight decay secretly minimize the rank of your neural network, 2024. URL https: /trpps://axiv.. /awxivivvvvv////iv//////////aavv org/ abs/ 2206.05794.
This paper presents theoretical framework to understand the stabilization of gradient subspaces through Hessian anal- ysis. It proposes WeLore, joint compression and efficient- finetuning strategy grounded on our findings of the exis- tence of varying levels of converged low- rank structures across different LLM components. Our work could lead improved methods for efficient LLMs development, and contribute the "GreenAl" goal.
Hambardzumyan, K., Khachatrian, H., and May, J. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00/21, 2021.
Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv2400.0:3293, 2024.
Acknowledgments
Z. Wang is in part supported by NSF Award 2145346 (CA- REER). This research has been supported by computing support on the Vista GPU Cluster through the Center for Generative AI (CGAT) and the Texas Advanced Computing Center (TACC) at the University of Texas at Austin.
Hayou, S., Ghosh, N., and Yu, Lora+: Efficient low rank adaptation of large models. arXiv preprint wrxiv:240..1235. 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
References
Biderman, D., Ortiz, J. G., Portes, J., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., et al. Lora learns less and forgets less. arXiv preprint wwXiv:2405.09673 2024a.
Hsu, Y.-.,, Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.000 12, 2022a.
Biderman, D., Portes, J.. Ortiz, J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S.. Chiley, V., Frankle, J., Blakeney, C., and Cunningham, J. P. Lora learns less and forgets less, 2024b. URL hww.thttt////..... //tp:iv.xxxv//////// abs/2405. 09673.
Hsu, Y.C.,, Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization, 2022b. URL https: / /awpiviivvvv.o/./////a abs 2220/220000111
Brock, A., Lim, T., Ritchie, J. M., and Weston, N. Freeze- out: Accelerate training by progressively freezing layers. arXiv preprint /70.://770049888 2017.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.. Wang. S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09663 2021a.
Chen, D., Bolton, J., and Manning, C. D. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint 100.v81606.02858, 2016.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models, 2021b. URL https: //axiiv. org/ brg/abs// 2106. 09685.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer,

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
lyyer, M., Boyd-Graber, J. L.,, Claudino, L. M. B., Socher, R., and Daumé, H. A neural network for factoid ques- tion answering over paragraphs. In Conference Empirical Methods in Natural Language Processing, 2014. URL ptttps://ap... org/ Cow.pssI::210344500000 sew.pssseetttisccoo.....
Li, Y., Yu, Y., Zhang, Q. Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large lan- guage models based on low-rank and sparse approxima- tion, 2023. URL https: /awxiv.ivvv./////a//aaa/aa/aaal/ org/ hasp//b///a 2306. 11222.
Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Relora: High-rank training through low-rank up- dates, 2023a. URL https://arxiv/: /bs//
Jain, S., Keshava, V., Sathyendra, S. M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint ar arXiv:2306.012000 2023.
Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Stack more layers differently: High-rank training through low-rank updates. arXiv preprint arXiv:2307.05695 2023b.
Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y. Compressing Ilms: The truth is rarely pure and never simple. arXiv preprint arXiv:2310.01.3
2023.
Liu, S.-Y., Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.C. F. Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 2024.
Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y. Compressing Ilms: The truth is rarely pure and never simple, 2024. URL https: pttttp://aaxiv.orgg
abs abs/2310. 01382.
Liu, Y., Agarwal, S., and Venkataraman, S. Autofreeze: Automatically freezing model blocks to accelerate fine- tuning. arXiv preprint arXiv:2102.01:33 2021.
Ji, Z. and Telgarsky, M. Directional convergence and align- ment in deep learning, 2020. URL https: //arxiv...vv//////////aa org/ /bgg/ab///00 2006.00600000 26657.
Meng, F. Wang. Z.., and Zhang, M. Pissa: Principal sin- gular values and singular vectors adaptation of large lan- guage models, 2024. URL https://arxiv..rrr
Jiang, K., Wu, D., and Jiang, H. Freebaseqa: A new factoid qa data set matching trivia-style question- answer pairs with freebase. In North American Chap- of the Association for Computational Linguistics, 2019. URL https:/ apii www.stismattttneeeeeeee org orc/CoorpusD::1440093
Ongie, G. and Willett R. nonlinear interpolating networks, 2022. URL https:
The role of linear layers in
//arxiv.org////e/.....e...... abs
Renduchintala, A., Konuk, T., and Kuchaiev, O. Tied-lora: Enhacing parameter efficiency of lora with weight tying. arXiv preprint Sati::::31110957777 2023.
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 10X1v:1705.03551,2000 2017.
Saha, R., Srivastava, V., and Pilanci, M. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36, 2023.
Kaushal, A., Vaidhya, T., and Rish, I. Lord: Low rank decomposition of monolingual code 1lms for one-shot compression. arXiv preprint 2:Xiv:2309.1402 2023a.
Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:231.003285, 2023.
Kaushal, A., Vaidhya, T., and Rish, I. Lord: Low rank decomposition of monolingual code lims for one-shot compression, 2023b. URL https: /tttps://arxiv.oot abs/2309. 14021.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. 1695, arXiv preprint arXiv:2306. 11695, 2023.
Le, T. and Jegelka, S. Training invariances and the low- rank phenomenon: beyond linear networks, 2022. URL patps://aaxitn org/ absp://b//22 /2001 11968.
Tian, Y., Yu, I.,, Chen, X., and Ganguli, S. Understanding self-supervised learning with dual deep networks, 2021. URL https: /tt///aavvvv//// org abs / 2210 0.000100000 00578.
Li, S., Yuan, G., Dai, Y., Zhang, Y., Wang, Y., and Tang, X. Smartfiz: An efficient training framework using attention- based layer freezing. arXiv preprint arXiv:2401.16720, 2024.
Timor, N., Vardi, G., and Shamir, O. Implicit regularization towards rank minimization in relu networks, 2022. URL mttps://arx: wtw.iv.org/b/////eeee 12760..
10

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Wang, H., Agarwal, S., Tanaka, Y., Xing, E., Papailiopoulos, D., et al. Cuttlefish: Low-rank model training without all the tuning. Proceedings of Machine Learning and Systems, 2023a.
Wang, H., Agarwal, S., U-chupala, P., Tanaka, Y., Xing, E. P., and Papailiopoulos, D. Cuttlefish: Low-rank model training without all the tuning, 2023b. URL https: //arxiv. orgg abs/ 2305. 02538.
Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd- IIm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint 24Xi::2403.07378, 2024.
Xia, W., Qin, C., and Hazan, Chain of lora: Efficient fine- tuning of language models via residual learning. arXiv preprint ar arXiv:240..0m 401.04411, 2024.
Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Pechenizkiy, M.. Liang. Y.. Wang. Z., and Liu, S. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint 1rX.v:2310.051733 2023.
Yuan, Z., Shang. Y., Song. Y., Wu, Q., Yan, Y.. and Sun, G. Asvd: Activation aware for compressing large language models. arXiv preprint arXiv:2312.058211 2023.
aware singular value decomposition
Zhang, L., Zhang. L., Shi, S., Chu, X., and Li, B. Lora-fa: Memory-effictie low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.033: 033003 2023.
Zhao, J., Zhang. Y., Chen, B., Schäfer, F., and Anandkumar, A. Inrank: Incremental low-rank learning, arXiv preprint arXiv:2306. /250 2023.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient Ilm train- ing by gradient low-rank projection. arXiv preprint arXiv:2403.cm 03507, 2024a.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient 11m training by gradient low-rank projection, 2024b. URL https: //arxiv. org/ abs/2403 03507.
Zheng, L.., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging alm-ae- with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0500 2023.
Zhong, Z., Friedman, D., and Chen, D. Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint 204.022004002440. 2021.

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
A. Proofs
Substituting W W, = -W)))(W.) it follows that:
A.1. Detailed Assumptions
|H & nL (Wlll)
(6)
Assumption A.1. Lipschitz Hessian: There exists Læ >
such that (W") (W)I S Lu IW'-W],
From the Kurdyka-Lojasiente Lojasiewizz (KL) condition, the gradient norm satisfies:
This assumption is justified by the smoothness of common loss functions (e.g., cross-entropy) and regularization terms.
IV (Wall S s (W,) - (WW)))
(7)
Assumption A.2. Kurdeia-L.jjsieviezz (KL.) Condi- tion: The loss satisfies a Kt inequality near W*, implying [| decays at power-law rate.
Using ((s) = sl-0, _, we have (0(s) = (1 - 0)s 8 Substi-
tuting this:
This assumption generalizes convexity to nonconvex.
V & T (L(,)) (W
(8)
Assumption A.3. Spectral Gap: For H; = (WI, with eigenvalues in descending order, there a uniform y > such that minigisr,js (A.HH)) - X, H+)] 2 7.
Let D = (L - L (W*))²-²
representing the scale of
the initial loss gap. Then:
This assumption is reasonable due to the dominance task- relevant directions in the hessian spectrum, reinforced by regularization.
D
L. -W.))
- L(W')
(9)
(T((166-
Assumption A.4. Reversible Structure and Gradient Representation: Assume the model architecture is re- versible (Tian et al., 2021), ensuring unbounded growth in norms. In particular, the gradient can be expressed (introduced in (Zhao et al., 2024b)))
Substituting this into the gradient bound:
C
VL(WWI| & 10/(1-0)*
(10)
Thus, the bound on ||H,|| becomes:
G. = A. - B,WCC
NL#C (0/(100)*
|AHI| S
(11)
where A,, Br, Ct are bounded linear mappings. Such re- versibility implies uniform boundedness Well and ||G,ll over 1, and stable spectral properties of H₃.
Applying Weyl's inequality, the change in eigenvalues be- tween He and He41 is bounded by:
A.2. Proof of Theorem 2.1
(H - (.HH..) S |AH||-
(12)
To show the stabilization of the eigenspace, we start by con- sidering the dynamics of the Hessian updates. The weights are updated using SGD as:
Summing over f. bounds the cumulative eigenvalue changes:
T- nLHC
- -(HH)))A(H)))
(13)
W = W. - LWW,),
(2)
1011100
where y) is the learning rate.
The series converges because (-8/(1-0) decays faster than 1/t, ensuring the cumulative shift remains bounded, thus proving eigenvalue stabilization.
Using a Taylor expansion, the Hessian at W441 can be ex- pressed as:
(3) For eigenspace stabilization, the Davis-Kahan theorem pro- vides a bound on the eigenspace shift:
= =+==v2L(WW) + AHe,
where:
|
|1++1 U.ll <
(14)
AH, V'L(W++1) - V'L(W,).
(4)
7
Substituting the bound for ||H
From the Lipschitz continuity of VWL we have:
|| - Uell S nL#C
(15)
|| |WHel < LH LH+11 Well.
(5)
710((1-0)
12

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Summing over I. bounds the cumulative eigenspace shifts:
Projecting the recurrence onto the non-dominant subspace results in:
T-1 NLHC
|IUT - Uoll -
(16)
(1-U = (1- -1, (I- U Ra.
7t (1-(110)
(24)
The series converges for 0 >
2 ensuring actual convergence of U.. For 0 E (0, $.. this analysis does not guarantee convergence, but it still provides a uniform bound on the step-to-step change in each finite interval.
Since (1- II, ViA, V.A,-"V,
V2 Gr, it follows that:
(I U-- = -1 UU, )G,
- n Σ Au G. + (I- (I-U,,,)
A.3. Proof of Theorem 2.2
s>r
Consider the gradient update W++1 = Wt - GG By Tay- lor's theorem, the gradient the next step given by:
(25)
The reversible structure ensures that |Wel. ||G and || remain uniformly bounded. This boundedness prevents pathological behavior and ensures the spectral gap y remains stable.
G++1 = Ww-LWWet)
wwwLwWW ((W) + H, - W,) + Re,
(17)
where the remainder term R₂ is defined as:
As {W,} is bounded and H(-) is Lipschitz, {H, forms bounded continuous family of matrices. Davis-Kahan perturbation theory guarantees that the eigenspaces depend continuously on The spectral gap Y > 0 ensures mix- ing between the dominant and non-dominant eigenspaces, allowing U2 to vary smoothly with t.
R₁ ( L(W
AW] dr,
AW, = G
(18)
For j > r. A S A - -y. Setting A, - 7 we have:
Substituting AW, into the expression, we have:
Σ d>r
Auuje G. = X(T -,U,)
(26)
G = G. - HH + R.-
(19)
Thus:
Given that H(W) is Lipschitz and ||G,]| is bounded, it fol- lows that |||ll| = O( Thus, the update simplifies
[1 G1t -UU((TU
to:
G++1 = Gt -HH, + O1
(20)
+ +1[(1- -UU,)),
(27)
Since || = (( and ||Gell is bounded, there
For each time step t, let U₂ = [ui..... up] be the orthonor- mal eigenbasis of H, corresponding to the eigenvalues A The orthogonal complement spanned by V, U/4.../ The Hessian can be decomposed
exists > 0 such that [Relll S Kn² Therefore:
[[
s (1 7)))((( - -U,,
U.U.)G. ). +
Kn²
(28)
as:
For sufficiently small n,, say < X
H, --=UUA UUUU + VAA(-VV
(21)
2K we have:
II U,UT s (1- n1
(1- [I U,UT,,
where 115) = ( (r...A)...,,Arr)) and A (-r)
=
(laag(A+1,....)
(29)
Decomposing the gradient G. gives:
This shows a geometric contraction of the non-dominant component. As + 20, [ GG, decays at least ge- ometrically. Since ||G,|| is bounded and does not grow, the non-dominant part shrinks to zero relative to ||Gel]. Hence:
G -UUUUU G. (I-U,
(22)
Applying the Hessian to the gradient, we have:
II(T U.U G
H,G. U.A G. V.A V,T G..
lim 60000
0.
(30)
(23)
GGel
13

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
This proves that Gg asymptotically aligns with the dominant eigenspace of Ht.
point (blue) using SoTA LLM pruning methods. We com- pared it with further compressing dense N-LRCs of WeLore checkpoints with ERR of 10%, 30%, and 50%. Our key observations are: (i) WeLore checkpoints can further enjoy high compression with sparsification of dense N-LRCs with- out signification performance drop to a noticeable sparsity ratio (e.g., WeLore checkpoint with ERR of 50% can be additionally sparsified using Wanda (Sun et al., 2023) with < 2 points increase in perplexity); (ii) ad-hoc sparsifica- tion of LRCs and N-LRCs (dense) suffers higher perfor- mance degradation compared to N-LRCs which demands actively exploring amalgamation of different compression techniques for LLMs to ripe maximum benefits; (iii) devel- opment of better sparsity algorithms (e.g. Wanda (Sun et al., 2023). SparseGPT (Frantar & Alistarh, 2023)) clearly retain their benefits even in mixed compression settings.
B. Additional Experiments
B.1. Visualization of Layer-wise Non-uniform Rank Ratio of WeLorr-COMP
1.0
Rank Reduction Ratio
0.8
0.6
self_ attnntttteee proj
self_ kttn.k proj
self_ a_tnntttteer p_ao
0.2
solr_ ouroo
C. Background Work
mio. galggate__p_ _por
mip. up up_pro)
Cemory-Effiiiien Finetuning: Memory-efficient fine- tuning of LLMs aims to address the significant costs as- sociated with their fine-tuning. This field encompasses sev- eral notable techniques. For instance, Prompt Learning Methods optimize input tokens or embedding while keeping the model's remaining parameters static (Hambardzumyan et al., 2021; Zhong et al., 2021). layer-freezing techniques enhance training efficiency by selectively freezing certain layers (Liu et al., 2021; Brock et al., 2017; Li et al., 2024). Additionally, Adapter Methods introduce a small, update- focused auxiliary module into the model's 's architecture, sig- nificantly reducing the number of trainable parameters, as introduced by (Houlsby et al., 2019; Diao et al., 2022). Among them, one noteworthy technique is Low-Rank Adap- tation (LoRA) (Hu et al., 2021a) and its successors (Ren- duchintala et al., 2023; Sheng et al., 2023; Xia et al., 2024; Zhang et al., 2023; Hayou et al., 2024; Hao et al., 2024; Liu et al., 2024). which introduces a low-rank weight adapter for each layer to reduce the memory footprint by only op- timizing the adapter. These low-rank adapters can then be seamlessly merged back into the original model.
0.0
mlp. dowin_rroj
10
15
20
25
30
Layer Number
Figure 7. I Layer-wise rank ratio of 50% compressed LLaMa-7B.
We investigated the layer-wise rank reduction ratio achieved by .OMP and found it to be highly non-uniform where some layers can be compressed significantly higher than others. In addition, note that layers from the first and last few transformer blocks are cumpression-friendyy Figure 7 illustrates the rank reduction ratios after 50% ef- fective rank reduction of LLaMa-2 7B pretrained check- point using WeLore. Interestingly, it can be noted that belfaattn. q p & self f attn.k k proj layers can be expressed as low-rank with 90% compression. More- over, the majority of layers from transformer blocks at the front and tail end are better at compression aligning with Section 2.5. The green region indicates LRCs while the red region indicates the N-LRCs components.
B.2. Investigating further Compression Opportunity with SOTA LLM Pruning.
Unlike LoRA which performs proxy optimization over ad- ditional parameters while keeping the original parameters frozen, Wel.rre backed by an understanding of gradient dynamics suggests finetuning the original parameters of LRCs in represented in low-rank to mimic full-finetuning. Recently, (Biderman et al., 2024a) found that full finetuning is more accurate and sample-efficient than LoRA across several task categories and WeLore can be an effective al- ternative to achieve the benefits of full-fim filettintunnig within a limited compute and memory budget.
Recently (Yin et al., 2023) investigated the activation outlier- based non- non-uniform sparsity ratios for different transformer blocks within I.LMs. A careful observation of their layer- wise sparsity ratio reveals that the majority of middle trans- former blocks can be subjected to higher pruning ratio which complementary to WeLore low-rank reduction ratio that favours terminal blocks being low-rank friendly. We therefore ask an unexplored question: How does LLM performance changes when we further compress only the dense N-LRCs using SoTA pruming methods?
Low Rank Compression: Large Language Models (LLMs) have succeeded remarkably across various natural language processing tasks, However, the massive scale of these mod-
Figure presents the increase in the perplexity of LLaMa-2 7B on the C4 dataset when we compress dense check-
14

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Magnitude
SparseGPT
Wanda
15.0
Dense (URCs N-LRCs) Prune
Dense (LRCs M-LRCs) Prune
Dense (LRCs N-LRCs) Prune
10% Low Rank LRCS Prune
2.0
10% Low Rank LRRCs Prune
2.0
10% Low Rank + N--RCS Prune
12.5
Rank N-LRCs Prune
low Rank N-RCS Prune
30% Low N-LRES Prune
50% Low Rank N-LRCs Prune
50% Low Rank N-RCCs Prune
50% Low Rank + N-LACs Prune
10.0
1.5
1.5
7.5
1.0
1.0
5.0
0.5
0.5
2.5
0.0
0.0
10%
20%
30%
40%
50%
10%
20%
30%
50%
10%
20%
30%
40%
50%
Figure Perplexity comparison (f) compression of N-LRCs using SoTA LLM pruning methods for LLaMa-2 7B on C4. Here we calculated the increase in perplexity wrt. the initial perplexity of dense and low-rank compressed checkpoints with ERR of r%.
els poses significant challenges in terms of storage efficiency and computational complexity. Among several techniques of LLM compression pruning, quantization, etc.), low- rank decomposition which retains only the top-k compo- nents in the low-rank space have special privilege to lever- age the existing highly efficient kernels over floating point matrices. (Hsu al., 2022a) developed data-aware modi- fication of SVD) that incorporates approximate second-order gradient information. Similarly, (Yuan et al., 2023) pro- posed data-aware decomposition method that minimizes activation error. One primary drawback of these reductions is that they uniformly reduce rank across all weight matrices. In contrast, our work experimentally validates existence of non-uniform low-rank expressiveness across different layers and should be accounted for during low-rank compression. Recently, (Zhao et al., 2023; Wang al., 2023a) found that dynamic rank selection during pretraining can achieve comparable prediction performance as full-rank counterpart.
efficiency. For baselines, we compared with commonly used uniform rank reduction (Hsu et al., 2022a; Kaushal et al., 2023a) and adopted recently proposed outlier-weighed non- uniform ratio (OWL) (Yin et al., 2023). We additionally augmented activation-guidee guided SVD techniques (Yuan et al., 2023) with WeLore's adaptive layer-wise rank reduction ratio to understand how it can benefit them.
D.3. Adaptive Threshold Selection
Algorithm Adaptive Threshold Selection Algorithm
Input: LLM with weights 0, target reduction ratio 8p. cur- rent reduction ratio St, reduction tolerance 88, threshold increment H;;
Output: compressed model 9 satisfying the target reduction ratio Sp, singular threshold H
Initialization: Initialize singular threshold H 0
While not (sp + 84 > > 8p 88) for each MLP layer tensor 6' in do su' +
D. Implementation Details
su + norma zz wiw.eiiigggll...llllleeeseeeeeee
D.I. Network Architectures:
p¹ 0; for each s sol, do
For understanding gradient dynamics and its consequent on the weight space during pretraining, we adopt the LLaMa- 130M architecture following (Lialin et al., 2023b; Zhao et al., 2024a). For our continual and downstream finetuning exper- iments, we adopted the pretrained checkpoint of LLaMa-2 7B and LLaMa-2 13B, and Mistral-7B from HuggingFace.
if H then
p' +1;
P, < Eip²
+ Pr/
if 8p + 88 2 St 2 8p
- 8z then
break;
else
H 11 Hi
D.2. Low Rank Compression:
For low-rank compression using WeLore for LLaMa-2 7B and 13B models, we used torch. svd (Wi) ) to decompose a layer I's weight matrix Wmmxn x Brxm where is decided by the heavy tail distribution of the singular values of W as deseribed in Section 3. If W belongs LRCs, it will be replaced with composition of two linear layers with lowk matrices A & B to improve the computational
D.4. Evaluation Tasks Settings
D.4.1. 1. FACTOID-BASED QUESTION ANSWERING
Task Definition and Rationale. Factoid-based Question Answering (Factoid- (Factoid-QA) (lyyer et al., 2014), which asks pre- cise facts about entities, is a long-standing problem NL.P
15

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
A typical Factoid-QA task aims search for entities or en- tity attributes from knowledge graph, and it is widely used as tool in academia, commercial search engines, and con- versational assistants. Modern LLMs are trained on gigantic text corpora ingesting a large amount of world knowledge about entities and their relationships during pre-training, and have unique abilities to generate factually correct responses to user queries. In this task setting, we aim to investigate how our MeLore-COME based compression impacts LLMSs ability to answer natural language questions using facts, ie., entities or attributes knowledge ingested within them during pre-training?
lurn conversational and instractioonfollooinng ability - two crifical elements for human preference. Evaluating Al chat- bots is a challenging task, as it requires examining lan- guage understanding, reasoning, and context awareness. To compare the performance of compressed LLMs' responses, we closely follow the prompt design setting in MT-Bench (Zheng et al., 2023) using GPT-4 as a judge. We prompt GPT-4 rate the answers generated by compressed LLMs wrt. GPT-3.5 (text-davinci-003) model based on varying metrics (e.g., correctness, helpfulness, logic, accuracy, etc.) on scale of [0-10] with detailed explanations.
Dataset Details. We rely on the 80 high-quality multi-turn questions identified in MT-Bench (Zheng et al., 2023). This setting covers common-use human-centric interaction with LLMs, and focuses on challenging questions to differentiate models. We used 8 common categories of user prompts to guide the prompt construction to interact with compressed LLMs: writing, roleplay, extraction, reasoning, math, cod- ing, etc. For each category, we adopted manually designed 10 multi-turn questions from MT-Bench to evaluate our compressed models.
Dataset Details. We use FreebaseQA (Jiang et al., 2019) which is a dataset for open-domain QA over the Freebase knowledge graph. The QA pairs are collected from various sources, including the TriviaQA dataset (Joshi et al., 2017) and other trivia websites (QuizBalls, QuizZone, KnowQuiz), and are matched against Freebase to generate relevant triples that were further verified by human annotators. TriviaQA dataset shows rich linguis- tic variation and complexity, making it a good testbed for evaluating knowledge ingested within LLMs.
subject-predicate-bbyee
D.6. Continual and Downstream Finetuning:
D.4.2. IN-CONTEXT VARIABLE LENGTH TEXT SUMMARIZATION
For continual finetuning settings, we finetune the WeLore compressed LLLMM-2 7B and 13B models at different com- pression ratios using C4 dataset. The C4 dataset a massive collection of Common Crawl's web crawl corpus, metic- ulously filtered and cleaned to ensure high-quality lan- guage modeling and training. For downstream task fine- tuning of compressed models, we consider good mixture of tasks from commonsense reasoning and math reason- ing, namely Commonsense@A, BoolQ, CoinFlip, SVAMP, BigBench, strategyQA. For comparison, we have used two baselines: (i) LORA: LoRA (Hu et al., 2021a) introduces low-rank adaptors for training the mod- els, W = W. + UV, where Wo is the pretrained weights, which are frozen during training. In our setting, we associate U and V with all the components of the IRRC and N-LRC of the compressed model and fine-tune them while keep- ing Wo frozen. (ii) GaLore (Zhao et al., 2024a): GaLore projects the gradient into low-rank format and updates the optimizer states and projects it back for updating weights. In this setting, we perform finetuning of both LRCs and N-LRCs full-finetuning) with projected low-rank gradients. Note that all our finetuning experiments start from the same checkpoint and hyperparameter settings for fair comparison.
Task Formulation and Details. Modern LLMs have shown astonishing success in summarizing long-context documents in both abstractive and extractive settings. However, it is yet not explored how FFN block skipping impacts LLMSs capability for summarization. In this task setting, we aim to investigate how well WeLore- hold onto consistency, coherence, fluency, and relevance when prompted to summarize textual information of varying length (small, medium, and large) in abstractive setting (Jain et al., 2023). For evaluation, similar to (Zheng et al., 2023), we propose to use GPT-4 as a judge, which compares the compressed LLM generated summaries wrt. GPT-3.5 (text-davinci-0033 generated summaries.
compressed models
Dataset Details and Results We use a popular summa- rization dataset CNN/DailyMaiil (Chen et al., 2016; Jaiswal et al., 2023) for evaluation, which is an dataset containing just over 300k unique news articles writ- ten by journalists at CNN and MailyMail. We created 3 subset categories (small (<470 words), medium (2470 and < 790 words), and large (2 790 words)} of stories, each with 100 articles reflecting word distribution of CNN/Daily- Mail to minimize OpenAl API costs.
E. Inference GPU Memory Statisties of WeLore-COMP
D.5. Multi-turn Conversation and Instruction Following
Task Formulation and Rationale. In this task setting, we investigate how WeLore-COME impacts the LLMs' ability to answer open-ended questions and evaluate their multi-
In this section, we investigate the memory requirement for inference with WeLore compressed models. Table 7 how WeLore allows reducing the memory requirement to load
16

---

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Total Params
Model Memory
Reduction
seglen 512
seqlen 1024
seqlen = 2048
seqler: 4096
0%
6738. 42M
13,579 MB
14,467
17,193 MH
24,519
30%
5794.
11,993
12,565
12,923
14,549
20,853
4543.677 67M
9,501
10,125
12,049 MB
18,377
70%
3072.84M
6,657
7,285 MB
7,625 MB
9,233 MB
15,549
Table Empirical estimate of Inference GPU Memory Requirement (measured with GPUtil library) of LLaMa-2 7B compressed with WeLore with varying context sequence length.
the model parameters by substituting the full-rank weight
matrices in their low-rank format. Given a consumer-grade GPU like GeForce RTX 4090, WeLore can facilitate infer- ence with 4K context length where the original model will flag an OOM error.
17


Reviews:
[{"review_id": "EAp3TKMDxw", "reviewer": "Reviewer_pkHm", "summary": "This paper investigates the low-rank property of LLM weights. The authors identify that low-rank properties vary systematically across components (q/k/v/o/mlp1/mlp2/out) and network depth. Based on this observation, they develop: 1) WeLore-COMP for non-uniform compression across different layers, and 2) WeLore-PEFT for selectively fine-tuning only components with good low-rank properties.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank you for the time to review our work. We greatly appreciate that you have found our theoretical analysis novel and experimental section comprehensive while identifying the unique proposition of WeLore as a cohesive method to handle compression and fine-tuning in a unified way. We would now like to address the weaknesses pointed by you one by one as follows:\n\n**1. Issues with Figure and Visualization Clarity:** We appreciate your concerns regarding about some mislabel and missing texts in our figures in the submitted draft. We promise you that we will address all these issues in the camera-ready version of our submission.\n\n**2. Hyperparameter sensitivity (k):** Thank you for raising this point. We would like to highlight that our hyperparameter k is uniquely determined/searched based on the effective rank reduction ratio (ERR) required. We use a linear search with np.linspace(0, 1, 0.005) using the algorithm described in Appendix D3 (Algorithm 1) to preserve normalized singular values greater than the threshold k so that we can achieve the ERR. In our experiments,  we have found that the precision 0.005 is sufficient to select optimal threshold K for a given ERR without notable variation in results. We will provide this additional clarification in the final draft of our submission. For additional information, we have included the pre-estimated singular value thresholds (k) for LLaMa-2 7B and 13B as follows:\n\n\n|Model| 10% | 20% | 30% |  40% | 50% | 60% | 70%|\n| ------------- |:-------------:|:-------------:|:-------------:|:-------------:|-------------:|-------------:|-------------:|\n|LLaMa-2 7B| 0.065 | 0.085|  0.115|  0.145 | 0.175 | 0.215 | 0.260| \n|LLaMa-2 13B| 0.065  | 0.085 |  0.115 |  0.140 |  0.180 |  0.225 |  0.270 | \n\n**3. WeLore and other compression techniques:** We appreciate your interest about WeLore comparison with other compression techniques like pruning and quantization. WeLore is a low-rank compression method with its own unique benefit of hardware-friendly acceleration and it is unclear what compression ratio of low-rank compression will equate to what compression ratio of pruning and quantization. However, we have indeed conducted experiments to study WeLore in conjunction with pruning. A careful observation of popular *non-uniform layerwise pruning algorithms (https://arxiv.org/pdf/2310.05175) reveals that the majority of middle transformer blocks can be subjected to a higher pruning ratio* which is **complementary** to *WeLore low-rank reduction ratio that favours terminal blocks being low-rank friendly* (Appendix B.1). Our experimental results in Appendix B.2 reveals that WeLore can be used in conjunction with SoTA pruning techniques like SparseGPT, Wanda etc. due to existing orthogonal properties in mixed compression settings with minimal performance degradations.", "claims_and_evidence": "Yes.\n\n1. Novel theoretical analysis. The gradient subspace analysis provides valuable insights into why low-rank structures emerge in LLMs, connecting optimization dynamics to model structure.\n\n2. Non-uniform approach. The recognition that different components have inherently different low-rank properties is insightful and leads to significant performance gains over uniform compression methods.", "methods_and_evaluation_criteria": "Yes.\n\nUnified framework. Unlike prior work focusing separately on compression or fine-tuning, WeLore provides a cohesive approach addressing both challenges simultaneously.", "theoretical_claims": "Not yet.", "experimental_designs_or_analyses": "Yes, this paper has a comprehensive evaluation. The work includes extensive experiments across different models (LLaMA-2 7B/13B, Mistral-7B), compression ratios (10-70%), and downstream tasks.", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "weaknesses:\n\n1. Figure issues and visualization clarity. Several figure problems: (1) Row 1, column 4 in Figure 2 is mislabeled as \"k_proj\" instead of \"q_proj\"; (2) Figure 1 lacks logical organization for comparing eigenvalue gaps among MLP components, attention components, and different layer depths; (3) Almost all the texts are missing in Figures 3-6, so it is impossible to interpret their results properly.\n\n2. Hyperparameter sensitivity. The threshold $k$ used for determining rank reduction lacks rigorous justification. The paper doesn't analyze how sensitive the results are to this parameter or provide a principled method for selecting optimal values across different models and tasks.\n\n3. Subjective LRC/N-LRC definition. The distinction between Low-Rank Components (LRCs) and Non-Low-Rank Components (N-LRCs) relies on empirical thresholds and visual interpretation of \"heavy-tail\" distributions (In figures 1-2). A quantitative metric would be better.\nLimited comparison with quantization methods. The method lacks thorough comparison with compression methods (quantization, pruning).", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jq7Tq8jOdH", "reviewer": "Reviewer_XxZn", "summary": "This paper studies the emergence of low-rank structures in Large Language Models (LLMs) through gradient subspace stabilization, revealing that as training progresses, gradients increasingly align with dominant Hessian eigenspaces, driving weight matrices toward low-rank factorization. The authors support this phenomenon with theoretical analysis—incorporating Hessian smoothness, KL conditions, and reversibility assumptions—and empirical evidence from LLaMA-based models (7B, 13B) and Mistral-7B. Building on these insights, they propose WeLore-COMP, a data-agnostic compression method that applies an adaptive global threshold on singular values to selectively compress \"low-rank components\" (LRCs) while preserving \"non-low-rank components\" (N-LRCs), and WeLore-PEFT, a parameter-efficient fine-tuning approach that optimizes only LRCs while freezing N-LRCs. Empirically, WeLore-PEFT matches or surpasses full fine-tuning performance while significantly reducing memory and computational costs, demonstrating the effectiveness of leveraging LLMs' intrinsic low-rank structure for efficient model compression and adaptation.\n\n## update after rebuttal\n\nThanks for the response. I will keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank you for the time to review our work. We would first like to thank you for finding our work to provide extensive experiments to establish robustness and have practical significance. We would now address your weakness point by point as follows: \n\n**1. Computational overhead of SVD on large matrices:** Thank you for raising this point. We would like to highlight that WeLore-COMP is a *one-shot data-agnostic* compression technique. WeLore-COMP requires **one-time SVD estimation** of all weight matrices of the pretrained checkpoint which can be saved and various compression ratios (e.g., 10%, 20%, 30%, etc.) can be achieved by a simple linear search of threshold k (Algorithm 1) without any need for re-estimation of SVD. \n\n1.  For empirical estimates of **LLaMa2-7B** using a Nvidia A6000 RTX GPU, a 4096 * 4096 matrix SVD decomposition takes ~2.6895 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~19 seconds. With 32 layers, the total time taken will be approximately ~10 minutes. *Note that SVD estimation of each layer can be executed independently and parallel*. Therefore, with 8 GPUs, we can reduce the time 10/8 = ~1.25 minutes in total.\n\n2.  For empirical estimates of **LLaMa2-70B** using a Nvidia A6000 RTX GPU, a 8192 * 8192 matrix SVD decomposition takes ~2.799 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~20 seconds. With 80 layers, the total time taken will be approximately ~26 minutes. *Note that SVD estimation of each layer can be executed independently and parallel*. Therefore, with 8 GPUs, we can reduce the time 26/8 = ~3.25 minutes in total.\n\nWe promise to include additional computational cost discussion on SVD in our final draft. \n\n**2. Generalizability to other architectures such as GPT-style models:** We would like to have additional clarification on this weakness.  Our experiments across various tasks,compression ratios on LLaMa & Mistral illustrate that WeLore generalizes to GPT-style decoder models. Decoder architectures are most popular and scaled up architecture, that’s why they are our experimental focus.\n\n\n**3. Theoretical framework relies on a reversibility assumption:** The current proof employs the reversible network structure primarily as a sufficient condition to ensure the boundedness and stability of both gradients and Hessians. In particular, the reversibility assumption is invoked to prevent unbounded growth of norms and to guarantee stable spectral properties of the Hessian sequence $\\{H_t\\}$. Notably, if boundedness and stability can be established through alternative means, strict reversibility may not be necessary.\n\nFrom a purely mathematical standpoint, the key requirements of the proof are as follows:\n1. **Boundedness of $||G||$, $||W_t||$, and $||H_t||$:**  \n   It is imperative to ensure that the norms of gradients, weights, and Hessians do not diverge. When these quantities remain bounded, the spectral gap property and the Lipschitz continuity of the Hessian ensure stable eigen-decomposition and validate the application of Davis–Kahan perturbation theory.\n2. **Stable Spectral Gap:**  \n   For $H_t=\\nabla^2 L(W_t)$ with eigenvalues in descending order, there is a uniform $\\gamma>0$ such that $\\min_{1 \\le i \\le r,\\, j>r}|\\lambda_i(H_t)-\\lambda_j(H_t)|\\ge \\gamma$.\n\nIt is important to emphasize that these conditions do not inherently require reversibility. Many standard optimization settings—such as strongly convex problems or well-conditioned neural architectures—employ mechanisms that naturally ensure boundedness and prevent degenerate Hessians. Consequently, if uniform boundedness of parameters, gradients, and Hessians can be secured by other means (for example, weight regularization, strong convexity near minima, or alternative architectural constraints), then the strict assumption of reversibility is not essential.\n\n**4. Benefit from other compression techniques on N-LRCs layers:** Thank you for raising this point and we agree that exploration of mixed compression strategy can be highly interesting with WeLore. We have indeed conducted experiments to study WeLore in conjunction with pruning. A careful observation of popular *non-uniform layerwise pruning algorithms (https://arxiv.org/pdf/2310.05175) reveals that the majority of middle transformer blocks can be subjected to a higher pruning ratio* which is **complementary** to *WeLore low-rank reduction ratio that favours terminal blocks being low-rank friendly* (Appendix B.1). Our experimental results in Appendix B.2 reveals that **WeLore can be used in conjunction with SoTA pruning techniques** like SparseGPT, Wanda etc. due to existing orthogonal properties in mixed compression settings with minimal performance degradations.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The main theoretical insights (Theorems 2.1 and 2.2) revolve around bounding changes in Hessian eigenvalues/eigenvectors over training steps under certain smoothness and KL assumptions. They demonstrate that: (1) The top-r eigenvectors converge quickly in orientation (Davis–Kahan, implying small angles); (2) The gradient eventually resides in the span of these dominant eigenvectors, pushing the solution toward a lower-rank subspace.\nWhile the authors do not present a fully rigorous line-by-line proof for every constant, they cite standard results (Lipschitz continuity, KL-based gradient decay) and supply sketches in Appendix A. This level of detail is acceptable for a conference paper.", "experimental_designs_or_analyses": "Yes - All experiments.", "relation_to_broader_scientific_literature": "This paper extends prior works on:\n1. LLM compression (e.g., SVD-based approaches, low-rank factorization, pruning, quantization).\n2. PEFT (LoRA, QLoRA, and other adapter-based methods).\n3. Hessian/gradient-based analyses linking spectral properties to optimization geometry.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths:\n\nThe paper provides a novel theoretical explanation by linking the emergence of low-rank structures to Hessian eigenspace alignment, offering a more principled understanding beyond heuristic-based approaches. The proposed adaptive rank thresholding achieves a better balance between compression and accuracy compared to uniform baseline methods. Extensive experiments across different model sizes, fine-tuning settings, and tasks demonstrate the robustness of the approach. Additionally, WeLore-PEFT has practical significance, as it can be directly integrated into training pipelines, reducing memory and compute costs while maintaining strong performance.\n\n## Weaknesses:\n\n1. The study focuses primarily on LLaMA- and Mistral-based models, limiting its generalizability to other architectures such as GPT-style models.\n2. The computational overhead of performing singular value decomposition (SVD) on large matrices is not thoroughly analyzed, particularly for very large models (13B+ parameters), where approximate SVD methods may be necessary.\n3. The theoretical framework relies on a reversibility assumption that may not always hold in standard Transformer layers, and the practical implications of this assumption are not fully discussed.\n4. Some layers are classified as non-low-rank components (N-LRCs) with minimal rank collapse, but it remains unclear whether these could still benefit from advanced compression techniques or partial rank factorization.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kXQlAO0F3m", "reviewer": "Reviewer_A6uE", "summary": "This paper proposes that repeated gradient alignment on the leading Hessian directions gradually drives large transformer models toward low-rank weight configurations. The authors formalize this tendency as a “rank collapse” that can be exploited both for compressing pretrained networks and for selectively finetuning only the most significant parameter subspaces. \nTheir method, WeLore, implements a global thresholding rule to prune weaker singular values layer by layer. Empirically, they show that WeLore not only yields strong compression with minor performance impact, but also supports a novel partial finetuning routine that updates only the high-importance (low-rank) components.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank you for the time to review our work. We would now address your weakness point by point as follows: \n\n**1. Computational overhead of SVD on large matrices:** Thank you for raising this point. We would like to highlight that WeLore-COMP is a *one-shot data-agnostic* compression technique. WeLore-COMP requires **one-time SVD estimation** of all weight matrices of the pretrained checkpoint which can be saved and various compression ratios (e.g., 10%, 20%, 30%, etc.) can be achieved by a simple linear search of threshold k (Algorithm 1) without any need for re-estimation of SVD. \n\n1.  For empirical estimates of **LLaMa2-7B** using a Nvidia A6000 RTX GPU, a 4096 * 4096 matrix SVD decomposition takes ~2.6895 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~19 seconds. With 32 layers, the total time taken will be approximately ~10 minutes. *Note that SVD estimation of each layer can be executed independently and parallel*. Therefore, with 8 GPUs, we can reduce the time 10/8 = ~1.25 minutes in total.\n\n2.  For empirical estimates of **LLaMa2-70B** using a Nvidia A6000 RTX GPU, a 8192 * 8192 matrix SVD decomposition takes ~2.799 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~20 seconds. With 80 layers, the total time taken will be approximately ~26 minutes. *Note that SVD estimation of each layer can be executed independently and parallel*. Therefore, with 8 GPUs, we can reduce the time 26/8 = ~3.25 minutes in total.\n\nWe promise to include additional computational cost discussion on SVD in our final draft. \n\n**2. Low-rank patterns and Datasets:** We would like to clarify that low-rank patterns emerging in model checkpoints are subjected to their pre-training. WeLore exploits these existing patterns for data-agnostic compression and PEFT. Experimentally through a wide-range of tasks in our experiments ranging from commonsense tasks to open-ended tasks (MT-Bench with code-centric questions - Table 2), we found that WeLore consistently perform well for the existing low-rank patterns in pre-trained checkpoints.\n\n**3. Freezing Weights and Underfitting:** Thank you for raising this point. Across all the experiments with WeLore-PEFT with different compression ratios, we didn’t observe the underfitting issue and WeLore-PEFT can closely mimic the training trajectory of full-finetuning. For example, as it can be seen from Table 6, with 30% compression, which approximately freeze 70% weights of the model during PEFT, WeLore outperform full-finetuning across several tasks which indicate that task adaptations can be adopted by only a few components in the model.\n\n**4. Extension for encoder-decoder architecture:** Thank you for this suggestion. Decoder-only architectures like LLaMa and Mistral family are most popular and scaled up architecture, that’s why we choose them as our experimental focus. We would like to highlight that while deeper cross-attention layers or an encoder-decoder structure might exhibit differences in rank collapse behavior compared to decoder-only models, these differences do not fundamentally invalidate the mechanism underlying WeLore’s compression or PEFT strategy. The low-rank collapse is driven by the stabilization of the gradient subspace and the emergence of a clear Hessian spectral gap—a phenomenon that is intrinsic to the training dynamics rather than being solely an artifact of a particular architecture. Even if cross-attention layers show a less pronounced rank collapse (due to their role in integrating encoder context, which can diffuse gradient signals), WeLore’s design is inherently adaptive. In other words:\n\n• Even if some layers (such as deep cross-attention ones) deviate from the typical rank collapse pattern, the thresholding in WeLore is computed on a per-layer basis, automatically preserving higher ranks where needed and applying more aggressive compression where the low-rank structure is clear.\n\n• The core mechanism—gradient subspace alignment with dominant Hessian directions—remains relevant across different architectures. Thus, while encoder-decoder models might introduce variations, they do not “disrupt” the phenomenon in a way that would render adaptive thresholding ineffective.\n\nDue to time limit of rebuttal, we couldn't complete our experiments but we are working towards it and we will include in our final version.", "claims_and_evidence": "The key claim is that top Hessian eigenvectors dominate gradient trajectories in large transformers, effectively concentrating the updates within a small subspace. Data from LLaMA-family checkpoints supports this assertion, revealing that the ratio of retained singular values decreases in deeper layers without degrading perplexity or downstream accuracy. The authors also show that once low-rank behavior emerges, subsequent training maintains it or even accentuates it, implying a persistent structural bias that WeLore can exploit.", "methods_and_evaluation_criteria": "WeLore applies a global singular-value threshold to each layer’s weight matrices, creating an adaptive truncation that balances compression and fidelity. The authors assess the resulting models on perplexity (using the C4 dataset) and multiple downstream tasks (QA, summarization, conversation). \nThey also measure speedups from partial finetuning, focusing on the memory and throughput gains by only updating the “highly ranked” fraction of parameters.", "theoretical_claims": "The analysis builds on classic Lipschitz continuity arguments and eigenvector perturbation bounds, positing that second-order curvature narrows the gradient search space over training iterations. \nBy showing that the Hessian’s top eigenvalues shift slowly, the authors infer that meaningful weight updates stay within a lower-dimensional manifold, effectively turning the model’s parameters into a near-rank-deficient collection. \nWhile they provide partial proofs for these curvature-based claims, the full set of constants and domain-specific constraints (like attention mechanisms) remain only informally addressed.", "experimental_designs_or_analyses": "In addition to testing different compression ratios on LLaMA-7B, 13B, and Mistral-7B, the authors compare WeLore’s adaptive thresholding with uniform SVD, outlier-based factorizations, and competing methods for parameter-efficient finetuning. They track how reduced-rank layers retain strong performance across open-domain QA and abstractive summarization. \nAlthough the evaluations are thorough within the LLaMA family, no experiments on non-decoder architectures (e.g., T5) or specialized tasks (like code completion) appear. This leaves questions about broader applicability.", "relation_to_broader_scientific_literature": "This work situates itself at the pretty unique intersection of Hessian-based analyses of network optimization and low-rank compression strategies. \nUnlike prior studies that introduce side modules (e.g., LoRA) or uniform rank constraints, WeLore leverages a single threshold that reflects global curvature trends. It also resonates with recent theoretical models linking gradient flow to emergent structure in overparameterized networks, though the paper’s emphasis on multi-layer transformers marks a specialized application.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "A notable strength is the demonstration that partial finetuning on just the “most relevant” components can match or exceed full finetuning in some tasks, potentially reducing hardware requirements in practical scenarios. Another strength is how their single threshold mechanism adjusts itself to each layer’s singular values. \n\nHowever, a potential weakness is that the approach has only been tested on decoder-only or Mistral-like models, so it is unclear if encoder-decoder architectures with cross-attention layers would exhibit comparable rank collapse. Also, the reliance on repeated SVD across many layers might be resource-intensive, and the discussion of approximate factorization is brief.", "other_comments_or_suggestions": "It would be valuable to investigate whether these low-rank patterns hold in tasks that deviate strongly from the original training domain, such as specialized code-generation or multimodal inputs.", "questions_for_authors": "Please see several questions in my above comments. Two additional questions:\n1. Can deeper cross-attention layers or encoder-decoder styles disrupt the rank collapse pattern observed in decoder-only frameworks, and if so, how might that influence WeLore’s thresholding strategy?\n2. Does “freezing” the bulk of weights ever lead to underfitting for tasks that place heavy demands on representational flexibility, or does the initial training’s rank deficiency remain adequate for adaptation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9cE1cT4EP", "reviewer": "Reviewer_bRJ5", "summary": "The authors deal with the task of studying low-rank compressions within LLM approaches studying the low-rankness of the weight matrices of the LLM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time to review our work. Next, we will try to address the weakness pointed by you one-by-one as follows:\n\n**1. Discussion of Sparsity in Section 2.4 and typoes:** Thank you for identifying this mistake. In Section 2.4, we mean low-rank representation and we promise to correct this in our camera-ready version of our submission. We will also correct additional typoes including threshold.\n\n**2. Rescaling Figure 1:** We would like to highlight that we intentionally have not rescaled all subfigures in Figure 1 to effectively capture/illustrate the variation in eigenvalues across different component types (MLP vs attention) of the model.\n\n**3. Clarification regarding Equation 1:** $S_{W_l}$ is an array of singular values of weight matrix $W$ of layer $l$. $(S_{W_l} < k)$ will be a $0/1$ array indicating which singular values are less  than k which will be truncated or compressed. $sum(S_{W_l} < k)$ is the sum of 1’s in the array illustrating the reduction ratio of layer l. Summing across all the layers (i.e., $\\sum_l sum(S_{W_l} < k)$) will give the total rank reduction in the entire model. $len(S_{W_l})$ is the indicator of the number of singular values in the weight matrix of layer $l$ and it may vary depending upon the model architecture choices.", "claims_and_evidence": "The authors discuss the low-rankness of the matrices within the LLM model. The illustrate this numerically, which is nicely shown in Figure 1.", "methods_and_evaluation_criteria": "Yes this is appropriately chosen.", "theoretical_claims": "The authors prove something about the properties of the Hessian during training where the full proof is given in the appendix and a sketch in the main paper. Also the authors prove some eigenspace alignment.", "experimental_designs_or_analyses": "The tests seemed appropriate to me.", "relation_to_broader_scientific_literature": "The findings are presented in comparison to other low-rank techniques for LLMs and this seems to the best of my knowledge appropriate.", "essential_references_not_discussed": "I did not miss any key references.", "other_strengths_and_weaknesses": "**Strength**\n**Weakness** The authors in Section 2.4 discuss sparsity but it is not clear to me what the mean here. If they refer to a low-rank representation this is quite something else than sparsity. If you mean low-rankness please be specific if you mean sparsity what is inducing this sparsity, i.e. in the coefficients, in the matrices? Again when talking about Q/K projections it is unclear to me whether sparsity is confused with low-rankness. Same comment regarding the MLP Gate Projections.", "other_comments_or_suggestions": "Threshold is misspelled throughout the figures", "questions_for_authors": "Would it be possible to rescale all subfigures in Figure 1 to have the largest value at 1. This makes the rank decay more comparable. Is equation (1) really meant to be the sum of sums? Isnt the length of $S_{W_l}$ not the same everywhere? Then the numerator sums up the singular values, right? How is the activate SVD defined? This is not explained but heavily used.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a unified framework for memory-efficient fine-tuning and compression of LLMs, grounded in a theoretical analysis of gradient subspace stabilization and low-rank weight matrix properties.
This submission received strong support from multiple reviewers:
- Reviewer pkHm highlighted the originality of the theoretical analysis and the unified design of WeLore 
- Reviewer XxZn appreciated the theoretical grounding, strong empirical evaluation, and practicality of the method.
- Reviewer A6uE commended the demonstration that partial fine-tuning can outperform full fine-tuning under significant compression.
- Reviewer bRJ5 acknowledged the soundness of the theoretical analysis.

Reviewers acknowledged and appreciated the authors’ responses.  Before the rebuttal, Reviewers raised concerns regarding figure clarity and typographical issues, the empirical justification and sensitivity of the threshold hyperparameter, the scalability of SVD in large models, the limited exploration beyond decoder-only architectures. These issues were comprehensively addressed in the rebuttal with clear clarification, additional empirical details, and plans for revisions. 

Therefore, AC recommends acceptance of the paper.
Decision: Accept (poster)