CLOVER: Cross-Layer Orthogonal Vectors Pruning
Authors: Fanxu Meng
Pingzhi Tang
Fan Jiang
Muhan Zhang
Venue: ICML 2025 poster (2025)
Keywords: SVD
PEFT
Pruning
Primary area: deep_learning->large_language_models
TL;DR: A post-training orthogonal initialization method for better pruning and fine-tuning.

Abstract: Decoder-only models generate tokens autoregressively by caching key/value vectors, but as the cache grows, inference becomes memory-bounded. To address this challenge, we introduce CLOVER (Cross-Layer Orthogonal Vectors) pruning, a novel approach that treats pairs of components of the attention mechanism as low-rank decompositions. CLOVER applies Singular Value Decomposition (SVD) to the Q-K and V-O pairs within each attention head. The resulting singular values, in turn, guide pruning and further serve as trainable parameters for efficient fine-tuning, ultimately enabling the model to recover its performance to the level before pruning.After pruning and fine-tuning, these values are reintegrated into the model without increasing its parameter count. Visualizations across various models show that CLOVER effectively removes linear redundancies within attention heads, greatly improving pruning efficiency. For example, pruning 70\% of the Q-K head dimension in GPT-2 XL results in a perplexity comparable to that of pruning just 8\% using vanilla pruning. The combination of CLOVER and TransMLA achieves a speedup of up to 11.1$\times$ over LLaMA-2-7B.

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Fanxu Meng' Pingzhi Tang'
Fan Jiang'
Muhan Zhang 12
Abstract
y
y
Decoder-only models generate tokens autore- gressively by caching key/value vectors, but as the cache grows, inference becomes memory- bounded. To address this challenge, intro- duce CLOVER (Cross-Layer Orthogonal Vectors) pruning, a novel approach that treats pairs of com- ponents of the attention mechanism as low-rank decompositions. CLOVER applies Singular Value Decomposition (SVD) to the Q-K and V-O pairs within each attention head. The resulting singular values, in turn, guide pruning and further serve trainable parameters for efficient fine-tuning, ultimately enabling the model to recover its per- formance to the level before pruning.Afer prun- ing and fine-tuning, these values are reintegrated into the model without increasing its parameter count. Visualizations across various models show that CLOVER effectively removes linear redun- dancies within attention heads, greatly improving pruning efficiency. For example, pruning 70% of the Q-K head dimension in GPT-2 XL results in a perplexity comparable to that of pruning just 8% using vanilla pruning. The combination of CLOVER and TransMI.A achieves a speedup of up to 11.1x over LLaMA-2-7B. Our code avail- able at:
Wo
Vvo
Wの
W
VQK
Uvo
Sux
W K
Uok
X
X
(a) One Attention Head
(b) CLOVER
Vanilla CLOVER
CLOVER
19g of Perplexity
25
0.08
0.7
10
Pruning Ratio
(c) Pruning without Training
(d) Fine-Tuning Pruned Model
Figure I. (a) We treat the Query-Key and Value-Output layers within a single attention head unified structure. (b) Apply SVD obtain two sets of singular vectors for initializing the Q-K and V-O layers, along with singular values that guide pruning or enable efficient full-rank fine-tuning (c) This cross-layer orthogo- nalization strategy allows for higher pruning rates. (d) The pruned model maintains strong performance after fine-tuning.
1. Introduction
In recent years, Large Language Models (LLMs) have rapidly evolved into essential tools for productivity (OpenAI, 2024; Anthropic, 2024; Team et al., 2024a). Open-source models (AI@ Meta, 2024; Mistral, 2024; Qwen, 2024; Liu al., 2024b; Team al., 2024b; Abdin al., 2024) have also narrowed the performance gap with closed-soure mod- els. The success of L.Mss is largely attributed to Next Token
Prediction (Radford et al., 2018; Brown et al., 2020), where tokens are predicted sequentially, with attention computed between each token and preceding ones. avoid redun- dant computations, key-value features are cached. However, as model size grows, the overhead of caching becomes sub- stantial, leading to memory and communication bottlenecks. For instance, in the case of 65B parameter model (Tou- vron et al., 2023) with 8-bit key-value quantization, storing 512K tokens requires over 86GB GPU memory, which surpasses the capacity single 80GB GPU (Sun et al., 2024).
"Institute for Artificial Intelligence, Peking University SState Key Laboratory of General Artificial Intelligence, BIGAL Corre- spondence to: Muhan Zhang <muhan www.cdu.u.......
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
To enable efficient training and inference, we introduce CLOVER (Cross-Layer Orthogonal Vectors) pruning, novel method that orthogonalizes the Query, Key, Value, and Output vectors without generating additional transfor- mation matrices. As shown in Figure la, we treat the Q-K and V-O pairs in each attention head as a low-rank decom- position of Wox and Wvo. By crossing these layers and performing SVD on WoK and Wyo, the Query, Key, Value, and Output vectors become orthogonal within each atten- tion head. Figure bb illustrates how the resulting singular values can guide pruning or serve as trainable parameters for efficient fine-tuning, After pruning or fine-tuning, these values can be reintegrated into the model without increas- ing its parameter count. Notably, previous methods, such as SVFT (Lingam et al., 2024), obtain orthogonal vectors by directly performing orthogonal decomposition on each projection matrix, which can lead to the introduction of large number of additional parameters during fine-tuning, In contrast, CLOVER jointly decomposes Q-K and V-O pair- sas transformation matrices for each other. CLOVER only generates a small set of singular values to guide pruning and fine-tuning, which can be merged back into the model without increasing inference costs.
2. Related Work
LLM Compression To alleviate the memory pressure of KV caches in long-context models, researchers have ex- plored several complementary directions: dynamic token pruning keeps only the past tokens whose attention scores materially influence future predictions, discarding the rest at inference time (Fu et al., 2024; Jo & Shin, 2024; Li et al., 2024b); rank compression of keys and values factorizes or groups the K/V tensors so that a lower-rank set of basis vectors plus coefficients can replicate the original attention, cutting the cache size almost linearly with the reduced rank (Shazeer, 2019; Ainslie et al., 2023; Liu et al., 2024a; Yu et al., 2024); head- dimension-level pruning statistically identifies and removes low-impact attention heads or sub- dimensions, slimming each token's 's stored representation (Ashkboos et al., 2024; Xia et al., 2023; Sun et al., 2023); cross-layer KV sharing reuses one KV table across multiple transformer layers, turning layer- wise memory growth into constant factor (Sun al., 2024; Brandon al., 2024; Liu et al., 2024e; Zuhri et al., 2024); and quantization encodes KV weights and activations in INT4-INT8 (or fewer bits), shrinking the cache size without altering the computational graph (Frantar et al., 2022; Dettmers et al., 2022; Xiao et al., 2023; Liu et al., 2024e; Hooper et al., 2024). Although these compression methods can reduce the model size and may ultimately achieve inference speedup, dropping ac- tive dimensions inevitably hurts accuracy (Ma et al., 2023); fortunately, parameter-efficien fine-tuning (PEFT) meth- eds. e.g. LoRA or adapters inserted after pruning-restie most of the lost quality with just 0. 0.1-1 % extra trainable parameters (Guo et al., 2023).
By orthogonalizing the vectors, we eliminate linear re- dundancy. Attention heads contain numerous non-zero norm vectors. Directly pruning these vectors would degrade performance, but orthogonalizing them allows us to rep- resent the entire attention head's space using a small set of orthonormal basis. The remaining vectors are nearly zero, making them safe to prune. As shown in Figure 1c, pruning 70% of the total budget in the query-key pair using ChovER-where the pruning ratio can vary across differ- ent attention layers-yielde a perplexity comparable to that of vanilla pruning, which removes only 8% of the vectors. We summarize the contribution of our paper as follows:
Parameter Efficient Fine-Tuning Low Rank Adaption (LoRA) is widely used due to its simplicity and effective- ness, with recent works enhancing it further (Zhang et al., 2023; Zi et al., 2023; Liu et al., 2024d; Zhao et al., 2024; Jiang et al., 2024). PiSSA (Meng et al., 2024) improves convergence speed by initializing adapters with principal singular values and vectors, also reducing quantization error (Wang et al., 2024a;b; Li et al., 2024a). However, PISSA is limited by its use of a fixed set of orthogonal bases. SVFT (Lingam et al., 2024) directly applies Singular Value Decom- position (SVD) to the original matrix. but this increases the number of parameters, raising computational overhead and reducing efficiency. The CLOVER method addresses these issues by treating the Query-Key pairs in each attention head as low-rank matrices. Using orthogonal decomposi- tion, CLOVER eliminates the need for additional transfor- mation matrices. Instead, it leverages small set of singular values to linearly combine orthogonal vectors, making the approach more parameter-efin effficient... After fine-tuning, the adapter can be smoothly reintegrated into the original matrix structure.
We treat the Q-K and V-O pairs in each attention head as low-rank approximations of WoK and Wvo. By performing SVD, we orthogonalize the attention head without adding extra transformation matrices.
This orthogonalization not only reduces linear redun- dancy, but also is compatible with any other pruning method, thereby allowing for higher pruning ratios. Pruning 46.42% of the vectors Whisper's attention head- OpenAl's speech-to-text model(Radford et al., presyese performance, without the need for additional training.
CLOVER enables parameter-efficient fine-tuning, sur- passing SOTA methods such as LoRA, DoRA, HiRA, and PiSSA on eight commonsense reasoning tasks across LLaMA-2-7B. and LLaMA-3-8B. Additional analyses further highlight its advantages.
2

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
3. CLOVER: Cross-Layer Orthogonal Vectors
singular values in S Se most d. We truncate the
SVD to retain only the top-r singular values without any loss of information:
Below a step-by-step breakdown the CLOVER method, illustrating how it performs orthogonalization of the Query, Key, Value, and Output layers in Multi-Head Attention, how orthogonal initialization helps improve pruning rates, and how the singular value matrices obtained from orthogo- nal decomposition can be used for efficient parameter fine- tuning.
Wak = U U@xl:: r] S s r, r| Vokl r.l
where S d.
The process can be easily applied to Wy and Wo, as de- tailed in Appendix D.4.
We begin by using the computation of the Q-K pair as a representative example, which is then generalized to the V-O pair.
CLOVER for Pruning After performing SVD, we can rewrite the weight matrix W QK as follows:
Multi-Head Self-Attention In multi-head self-attention mechanism with H heads, each head h E {1...., H} com- putes an attention score as:
WOk Ug r] S brarl
(vh [okl.:r)) r))"
Q.KT vd
attn(Qn, Kn. softmax
W
W2
where d is the dimension of each head, Qn, Kv E IRd are the query and key representations for head h.
Instead of storing the full matrices Wh and W E DRd we store the smaller factors W and W E Dxr, which are significantly smaller than the original matrix since S d < D. This leads to reduction in both memory usage and computational cost. Additionally, we can further prune small nonzero singular values (and their correspond- ing singular vectors) that fall below a chosen threshold, further reducing the parameter count and computational overhead.
Specifically, the queries and keys for head h are obtained by multiplying the input matrix X E (n the sequence length, D the hidden dimension) with the correspond- ing "slice" of projection matrices Wo, WK E Dxxxx respectively:
Qn = - KA = X whhd.
Cross-Layer Merging Substituting the expressions for Qn and K2 into the product Qn we have:
CLOVER for Fine-Tuning CLOVER can be used not only for pruning, but also for parameter-efffcient fine-tuning, We freeze the matrices U2 :,; r] and V2 I r). and only fine-tune the singular values l rr.
Qu = wwh (wkkhi)) X T
Notice that the original weights Whhhh and whbbd are each in Dxd When multiplied together, the resulting ma-
In contrast to SVFT, which factorizes the original weight matrices Wo, WK, Wv, Wo E HDDHxx individually, CLOVER factorizes the merged weights Wax and & within each attention head. As result, the tunable matrix SQx has size bounded by x (considering all heads). In comparison, SVFT requires factorizing large matrices each into three components S, V e IR DxD), leading to significant increase in parameter count and computational overhead, even with sparse updates for the singular values S.
trix QK whihh (wl has dimensions D x D. Since d < D, directly using W in computations or storing it as trainable parameters would be highly ineffi- cient, limiting the applicability of such parameter merging.
Cross-Layer Orthogonal Decomposition To mitigate the large size of W we factorize via SVD:
For example, consider the LLaMA 2-7B model with 32 attention heads and a head dimension of d 128. By factorizing each head separately, the largest size for SQx is O(32 x 128 x 128), which is significantly smaller than factorizing a IR 4096 matrix. This makes CLOVER's parameter efficiency comparable to that a LoRA config- uration with rank 32, as shown in Appendix B, but with additional potential for pruning.
W
UÍj Sāx Vlk
where U/yx and Vdx are D x D orthogonal matrices, St is a D x D diagonal matrix of singular values.
Since w- liih and wh A each have dimensions R Dxd the rank of Wh 2x is at most d. Thus, the number of nonzero
3

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Table Comparison of CLOVER with SliceGPT and AL. on pruning DeepSeek- V2-L.te and I.LaMA---BB separately, and
evaluation of their fine-tuned performance across six benchmarks.
Model
Hidden Size
Head Dim
Avg.
MMLU
ARC
PIQA
HS
OBQA
WG
DeepSeek V2 Lite
61.54
43.29
60.39
79.92
74.51
45.40
65.75
-6.25%
57.30
38.40
55.95
77.20
68.67
41.20
62.35
SliceGPT
-12500%
53,51
35.24
51.97
74.27
62.08
37.80
59.67
-25%
59.84
41.16
57.56
79.27
72.61
44.60
63.85
.CLOVER
-50%
57.25
38.96
55.27
78.02
69.63
41.40
60.22
Model
KV Cache
Head Dim
Avg.
MMLU
ARC
PIQA
HS
OBQA
WG
LLaMA-2-7B
59.85
41.43
59.24
78.40
73.29
41.80
64.96
-68.75%
59.82
40.87
59.18
77.91
71.82
45.20
63.93
-LranMLA
-87.50%
59.36
40.77
58.84
78.18
71.28
43.60
63.46
-92.97%
58.68
40.82
59.72
76.55
69.97
43.60
61.40
-68.75%
-50%
59.40
40.91
58.97
78.35
71.32
43.40
63.46
CLOVER
-87.50%
-50%
59.28
40.46
59.12
77.48
70.62
44.60
63.38
-92.97%
-50%
59.13
40.69
60.03
77.09
69.65
45.20
62.12
tations in the absorb phase. To compare the effectiveness of CLOVER with other pruning methods, we adapted the SliceGPT (Ashkboos et al., 2024) codebase to support the DeepSeek model architecture. And we applied CLOVER to orthogonally initialize the attention heads and pruned the attention head dimensions based the magnitude of singular values.
Table 2. Comparison of latency metrics for different method.
Model
Prefilling (ms)
Generation (ms/token)
DeepSeek 195.12
40.11
SliceGPT
191.91
40.32
CLOVER 177.02
31.00
Additionally, we compared pruning for LLaMA-2-78 with TransMLL (Meng et al., 2025), which converts models us- ing Multi-Head Attention (MHA) or Grouped Query Atten- tion (GQA) into MI.A-based models, effectively compress- ing the KV cache. TransMLA can be further combined with CLOVER to prune the dimensionality of attention heads more efficiently. We pruned the K NoPE and V head di- mensions in the LLaMA-2-7B model released in their paper, to evaluate CLOVER'S ' effectivvness. For fine-tuning, we followed the TransMLA procedure on a mixed pretraining dataset, as shown in Table
4. Experiments
In Section 4. we compare CLOVER with SliceGPT (Ashk- boos et al., 2024) and TransMLA (Meng et al., 2025), which respectively prune DeepSeek-v2-L.ite (DeepSeek-AA, 2024) and LLaMA-2-7B (AI@Meta. 2023). Section 4.2, we vi- sualize how CLOVER removes linear redundancy between vectors, facilitating more efficient pruning. In Section 4.3, evaluate the acceleration performance of CLOVER. In Section 4.4, we demonstrate CLOVER's s ability to perform significant pruning In Section 4.5, we apply CLOVER to orthogonalize the attention heads of the GPT-2-XL model (Radford et al., 2019), to explore the role of CLOVER in both pruning and fine-tuning. In Section 4.6, we conduct fine-tuning experiments on eight commonsense tasks, com- paring CLOVER with SOTA PEFT methods.
All models were evaluated on six benchmarks: MMLU (Hendrycks et al., 2021), ARC (easy and challenge) (Clark et al., 201 8a), PIQA (Bisk et al., 2020a), HellaSwag (HS) (Zellers et al., 2019a), and Winogrande (WG) (Sakaguchi et al., 2021a). These evaluations serve to validate the effec- tiveness of different pruning strategics.
4.1. Comparing CLOVER with Other Methods
As shown in Table CLOVER achieves performance com- parable SliceGPT while pruning 50% of the head dimen- sion, compared to SliceGPT's 6.25% pruning. However, as demonstrated in Table 2, CLOVER delivers 1.25 speedup, whereas SliceGPT provides no acceleration. Fur- thermore, building on TransMLA, an additional 50% prun- ing of the head dimension still allows the model to recover its performance with only a small amount of retraining.
Currently, pruming efforts for DeepSeek models are lim- ited. The few existing approaches mainly focus on reduc- ing the number of experts in the MoE module (Gu al., 2025). However, by orthogonalizing the attention heads in DeepSeek, we observe that significant redundancy also exists within MLA (Figure 2a). Removing this redundancy can substantially reduce the computational overhead during training, pre-filling, and the computation of query represen-

---

CLOVER: Cross-Lay Orthogonal Vectors Pruning
1.50
0.08
Vanilla CLOVER
Vanilla CLOVER
1.25
CLOVER
Vanilla
Varilla CLOVER
Vanilla CLOVER
0.06
IWollwa
1.00
Pmlliw
1.0
[10,
0.93)
11, 2.66)
0.75
0.04
1. 0.04)
2
0.50
(29. 1.251
0.5
0.02
0.25
0.0
50
100
0
25
50
75
0.00
0
20
40
60
0
20
40
60
0.00
50
100
0.6-
Vanilla CLOVER
0.6
Vanilla CLOVER
0.6
0.8
Vanila CLOVER
0.6
0.8
Varilla CLOVER
0.08
0.06
Vanilla CLOVER
0.4-
0.4
(27. 0.29)
0.4
(26. 0.46)
0.04
0.4
(44. 0.02)
0.2
172. 0.13)
0.2
(35. 0.10)
0.02
0.2
0.0
0.00
100
0
50
20
60
40 Sorted Dimensions
50
100
Sorted Dimensions
Sorted Dimensions
Sorted Dimensions
Sorted Dimensions
(a) DeepSeek--22eee V2-Lite
(b) Llama-3.2 3.2-Vnsionn
(c)
Whisper-Large--2?
(d) SDXL
(e) CLIP-
Figure 2. CLOVER (orange) uses fewer orthogonal basis vectors than Vanilla Pruning (bluc) to span the attention head space. The first row shows the importance of Q-K dimensions, and the second row shows V-O dimensions. After the red dot, CLOVER's importance lower, and pruning these vectors results in less performance loss.
4.2. CLOVER Removal Redundant Vectors
vectors have norms that approach zero, indicating that they are already represented by the dominant singular vectors and can be pruned without loss of performance. Beyond the red intersection point, CLOVER's s remaining vectors exhibit consistently lower importance than those in Vanilla Pruning, meaning pruning these vectors results in less per- formance degradation. This demonstrates why CLOVER enables a higher pruning ratio. A similar trend is observed for the V-O pair, although the model's inherent sparsity is less pronounced than in the Q-K pair, making the effect less noticeable. Still, in most models, pruning half of the vectors has a smaller impact on performance compared to Vanilla Pruning. Notably, in CLIP-ViT-bige (Figure 2e), a proportion of the vectors already have a norm of zero, allowing for safe pruning.
CLOVER achieves higher pruning ratio due to the sig- nificant linear redundancy present in the model. By repre- senting the entire attention head with only small number of orthogonal vectors, CLOVER effectively removes this redundancy. To illustrate the advantages of CLOVER in eliminating linear redundancy, we apply it to a variety range of models, including the large language model DeepSeek- V2-Lite (DeepSeek-AA, 2024), the multimodal automatic speech recognition and speech translation model Whisper- Large-v3 (Radford et al., 2023), the multimodal instruction- tuned image reasoning generative models LLaMA-3.2- IB- Vision (AI@ Meta, 2024), the image encoder CLIP-ViT- bigG (Cherti et al., 2022), and the image generation model Stable Diffusion XL (Podell et al., 2023). We compute the L2 norm for each dimension (equal to singular values) in both the Q-K pair and the V-O pair, sorting the values in descending order within each attention head for better visu- also perform Vanilla Pruning,
Beyond the red intersection point, CLOVER's remaining vectors exhibit consistently lower importance than those in Vanilla Pruning, meaning pruning these vectors results in less performance degradation. This demonstrates why CLOVER enables a higher pruning ratio. A similar trend is observed for the V-O pair, although the model's inherent sparsity less pronounced than in the Q-K pair, making the effect less noticeable. Still, in most models, pruning half of the vectors has smaller impact on performance compared to Vanilla Pruning. Notably, in CLIP-ViT-bige (Figure 2e), proportion of the vectors already have a norm of zero, allowing for safe pruning.
alization. For comparison, which does not utilize CLOVER initialization but instead sorts directly based on the Lg norm.
Figure 2 showcases the first attention head from the first layer of each model. In the first column of the figure, depict- ing the Q-K norm, we observe that in the original model, the importance of each dimension is relatively balanced (e.g. Figure 2c). This balanced distribution is result of the linear redundancy, where different directions are inter- twined, making it challenging to prune individual directions without negatively affeeting the model's performance. How- ever, after applying CLOVER's 's orthogonal decomposition, only a small number of orthogonal bases on the left side exhibit significantly large norms. These vectors span al- most the entire attention head's 's space, and the remaining
Beyond the red intersection point, CLOVER's remaining vectors exhibit consistently lower importance than those in Vanilla Pruning, meaning pruning these vectors results in less performance degradation. This demonstrates why CLOVER enables higher pruning ratio. A similar trend is observed for the V-O pair
5

---

CLOVER: Cross-Lage Orthogonal Vectors Pruning
Low Rank 512
11.1x
Low Rank
512
6.7x
Low Rank Q
512
4.8k
Spooduup
10
Full
Seeedup
Full Rank Q
Speedup
Full Rank Q
8
3
2-
1
1k
4k
6k
1k
2k
4K
8k
16k
32k
1k
2k
8k
16k
32k
Context Length
Context Lengthh
Context Length
(a) 165.2 TFLOPS 24GB
(b) 312 TFLOPS 40GB
(c) 320 TFLOPS 64GB
Figure 3. Inference speedups with CLOVER comparing to the original LLaMA2 7B model on three platform. Low-rank Q and Full-rank indicate whether the query projections also compressed. Context length represents the total sequence length.
4.3. Inference Speedup with CLOVER
markably, the model's output remains nearly unchanged, with only one error, which has been highlighted in the text using strikethrough and red for elarity:
In Figure 3, we benchmark the inference performance of CLOVER foaturrfa 92.97% reduction in the KV cache and 50% reduction in the Q nope, K nope, and V head di- mensions-uuing the vLLM framework across three GPUs with varying compute capabilities and memory sizes: 165.2 TFLOPS with 24GB memory, 312 TFLOPS with 40GB memory, and 320 TFLOPS with 64GB memory. The fig- ure illustrates the inference speedup of the pruned model relative the original LLaMA-2-7B. "Low-rank Q" and "Full-rank Q" indicate whether the query projections were also compressed. The context length refers the total sequence length, which includes both the prompt and gener- ated tokens (with equal lengths for each).
0.5
Amoittate
0.0
-0.5
285
425
625
Be5
10e5
Samples
Figure 4. An audio waveform from the librispeech dataset.
Mr. Quilter is the apostle of the middle classes, and glad to welcome his gospel. Nor is Mr. Quilter's manner less interesting than his matter. He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and can discover in it but little of rocky Ithaca. Linnell's pictures are sort of Up Guards and Adam paintings, and Mason's 's exquisite idles are as national as jingo poem. Mr. Birkett Foster's landscapes smile at one much in the same way that Mr. Carker used to flash his teeth. And, and Mr. John Collier gives his sitter cheerful slap on the back before he says, like shampooer in a Turkish bath, next man.
Our experiments demonstrate that CLOVER's ' inference speedup increases with longer context lengths. As long sequences typically lead to both compute and memory bot- tlenecks, compressing the KV cache and attention head dimensions helps alleviate these issues, thereby enabling higher speedups. Notably, for an 8K context window on the first hardware platform, the CLOVER-pruned model achieves an impressive 11.1x inference acceleration.
4.4. CLOVER for Training- Free Pruning
As demonstrated by the prominent low-rank properties in Figure 2c, we applied pruning to the Whisper-large- v3 model (Radford et al., 2023). We use the offi- cial Whisper-large-v3 example (LibriSpeech Long dataset (Gandhi et al., 2023)") intuitively highlight the effective- ness of CLOVER pruming. For reference, the waveform of this input is shown in Figure 4, and the corresponding target translation script is provided in Appendix C.
In contrast, vanilla pruning-which forgoes orthogonal ini- tialization and prunes head vectors solely based on their norm femm-results in the model completely failing to generate valid outputs at the same pruning ratio.
After applying CLOVER to orthogonalize the vec- tors, we pruned vectors with magnitudes close to zero ([Walll]Wx < 1 3 and ||Wi|||WT]
This example validates our earlier claim that straightfor- ward pruning of non-zero dimensions can lead to accumu- lated loss. In contrast, CLOVER effectively eliminates lin- ear redundancy, enabling a significantly higher pruning ra- tio. When the linear redundancy is sufficiently pronounced, CLOVER can even achieve a high pruning ratio without the need for fine-tuning recover performance.
10 10-3). This pruning achieved ratios of 56.01% % and 36.82% for the parameters in K Pair and V-O Pair, respectively. Re-
6

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Table Pruning the attention layers of GPT-2-XI. using CLOVER and vanilla pruning various sparsity levels. We report perplexity on WikiText-2 (lower is better) and evaluate tuninn performance on OpenWeb Text under different token budgets. The base model's s perplexity is 14.78. CLOVER" and Vanilla fine- line-tune tune the pruned attention layers, while CLOVER PEFT fine-tunes the singular value matrices obtained from the decomposition of the QK and VO projections.
w/o Training ())plexity()
66M Tokens Perplexity (!)
131M Tokens Perplexity (L) CLOVER#T CLOVERPEFT
Pruning Ratio
Vanilla
CLOVER
Vanilla
CLOVER*T
CLOVERREFT
Vanilla
12.5%
33.76
15.89
16.04
15,45
15.67
16.38
15.77
15,42
25.0%
78.36
17.45
16.93
15.70
15.89
17.07
16.05
15.75
37.5%
159.4
20.95
18.17
16.17
16.60
18.14
16.48
16.41
50.0%
338.9
35.12
20.45
17.22
17.63
19.02
17.13
17.71
62.5%
538.5
85.25
24.65
19.32
20.64
21.44
18.40
20.39
75.0%
708.8
187.4
36.04
24.65
29.28
27.22
20.99
28.44
4.5. Pruning and Fine-Tuning with CLOVER
merged into the U and V matrices. Instead, they are re- tained for parameter-efficien fine-tuning, where only these singular values are updated, and merging is deferred until post-training. PEFT typically converges more slowly than pallmparreeter finetuning. To accelerate convergence, we increase the learning rate from 6 10 to x 10-3 and re- move weight decay, while keeping all other hyperparameters consistent with those used in Vanilla and CLOVER
Model pruning often necessitates fine-tuning to recover from performance degradation. CLOVER supports both pruning and fine-tuning within a unified framework. In this section, we evaluate CLOVER's 's effectiveness in both aspects. We initialize GPT-2-XL with CLOVER and prune the model by removing vectors corresponding to the singular values with the smallest magnitudes.
As shown in Table 3, CLOVER induces significantly less performance degradation than Vanilla pruning by concen- trating functionality into fewer orthogonal bases. For in- stance, pruning 50% of the parameters without fine-tuning increases CLOVER perplexity by only 1.38x, com- pared to 21.9x for Vanilla. After fine-tuning, CLOVER substantially outperforms Vanilla; for example, CLOVER with a 75% pruning rate achieves comparable performance to Vanilla pruning at only 62.5%. Owing to its reduced model disruption, CLOVER also requires fewer training tokens to restore performance (e.g., perplexity with 66M tokens closely matches that with 131M), whereas Vanilla pruning demands more data, increasing both computational cost and the risk of degradation out-of-domain tasks.
To achieve better performance, Figure Ic allows each layer to adopt a different pruning rate. In this case, fixed propor- tion of parameters is pruned across the entire model based on the global ranking of singular values (for CLOVER) or Ly- norms (for Vanilla pruning). Remarkably, pruning 70% of the parameters using CLOVER yields performance compa- rable to pruning only 8% with Vanilla pruning, highlighting the effectiveness of CLOVER's orthogonal initialization in facilitating structured pruning.
However, using a uniform pruning rate across all lay- ers ers-where the same percentage of the smallest singular vectors is pruned per layer-is beneficial for consistent train- ing and inference speed. Therefore, unless otherwise noted, apply uniform pruning across layers. After fine-tuning the singular values S are merged into their corresponding U and V matrices. For comparison, we also apply Vanilla prun- ing, which lacks CLOVER's s orthogonalization and instead uses an L2- Lg-norm-based criterion.
Moreover, CLOVER which fine-tunes only the singu- lar values from the SVD decomposition and the attention layer biases, enables performance recovery with minimal resource consumption and parameter updates. At lower pruning rates, CLOVER even surpasses full attention- layer training (CLOVER However, at higher pruning rates, performance declines significantly due to the limited number of remaining tunable parameters (e.g.. only 0. 1.5 of the original attention-layer parameters are updated).
Following pruning, we evaluate model perplexity on the WikiText-2 dataset (Merity et al., 2016). We then fine-tune the pruned models the Web dataset (Gokaslan & Cohen, 2019), using the nanoGPT framework?. To mini- mize disruption to the pretrained model, only the pruned at- tention layers are fine-tuned, while the MIP,, embedding lay- ers, and LM head remain fixed. This setup is referred to as CLOVER and Vanilla, respectively. In the CLOVER PEFT configuration, the singular values S are not immediately
These results empirically validate the benefits discussed earlier: CLOVER's orthogonal initialization of attention heads enables the representation of the entire attention space using compact set of orthogonal bases, which is highly advantageous for pruning. Furthermore, the singular value matrix can be seamlessly merged back into the attention head.
7

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Table Accuracy comparison of 1.AaMA-77,, and L.MMM3-8BB with various PEFT methods eight commonsense reasoning datasets.
Results of LoRA and DoRA are taken from (Liu et al., 2024d). Results of HiRA are taken from (Huang et al., 2025).
Hella
Wino Grande
Model
Method
Params
BoolQ
PIQA
SIQA
ARC-e ARC-c
OBQA
Avg.
Swag
LoRA
0.83%
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
DoRA
0.84%
71.8
83.7
76.0
89.1
82.6
83.7
68.2
82.4
79.7
LLaMA2-7B
HiRA
0.83%
71.2
83.4
79.5
88.1
84.0
86.7
73.8
84.6
81.4
PiSSA
0.83%
75.0
87.0
81.6
95.0
86.5
88.5
75.9
86.4
84.5
CLOVER
0.83%
75.0
86.4
82.0
95.1
87.5
89.6
76.6
89.4
85.2
LoRA 0.70%
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
DoRA 0.71%
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
LLaMA3-8B
HiRA 0.70%
75.4
89.7
81.2
95.4
87.7
93.3
82.9
88.3
86.7
PiSSA 0.70%
77.2
90.0
82.9
96.6
88.4
93.6
82.4
87.4
87.3
CLOVER 0.47%
76.4
89.3
82.1
96.9
89.9
93.6
84.5
90.6
87.9
The comparison of memory consumption and runtime in Table 5 demonstrate that CLOVER consumes less GPU memory and exhibits shorter training runtime compared to LoRA. We attribute this to CLOVER being applied between two layers, whereas LoRA operates in parallel with the main branch. This enables sequential computation, eliminating the need to retain the input features of the main branch.
Table 5. Comparison of training costs between LoRA and CLOVER on LLaMA-2-7B. We trained on a common- sense dataset for 3 epochs with model_max_length steps num gpus 4, executed 4 x 312 TFLOPS- 80G GPUs.
1024,
per.device wweeevieeerrainn..ee batih_bbach_siz = 2.
Method Params
Max Memory
Runtime
LoRA 0.83%
110.84 GB
2:42:37
LoRA and DoRA results are taken from the DoRA paper, while HiRA results are sourced directly from its original publication. Since PiSSA has not conducted experiments on commonsense reasoning datasets, we include its perfor- mance by reproducing the experiments ourselves. For a fair comparison, we adopt the hyperparameters from DoRA and adjust the learning rates accordingly. As shown in Ta- ble 8, CLOVER achieves the best performance with a learn- ing rate of 1c-4, which we apply consistently across both LLaMA-2-78 and LLaMA-3-8B PiSSA performs best with learning rate of 2e-5, 5, as reported in its original paper; all other hyperparameters remain unchanged. Due to the stable training behavior observed both PiSSA and CLOVER, omit the validation procedure used in DoRA-whert where the best pesst-erforming model is selected every 80 iterations based on the validation set. Instead, we train for the full 3 epochs and use the final model checkpoint for testing.
CLOVER
0.83%
104.75 GB
2:22:47
4.6. Comparison with PEFT Methods
In this section, we conduct an ablation study to com- pare the fine-tuning capability of CLOVER against several parameter-efficient fine-tuning (PEFT) methods, including LoRA (Hu al., 2021), DoRA (Liu al., 2024d), HiRA (Huang et al., 2025), and PiSSA (Meng et al., 2024). We ex- clude SVFT (Lingam et al., 2024) from this comparison due to its significant computational overhead. The evaluation spans eight sub-tasks, as detailed in Table 7. All models are fine-tuned on the Commonsense- 148k dataset and evaluated on the respective test sets of each sub-task.
For CLOVER, apply orthogonal decomposition to the Value-Output projection and fine-tune the resulting singular value matrix. Due to the non- lonelr RoPE (Su et al., 2024) operation between the query and key, we instead decompose the Key layer and fine-tune its transition matrix. Likewise, in the m1 p. wup.ppprr proj layer, we treat every consecutive dimensions as a head, apply orthogonal decomposition, and update the corresponding transition matrix.
Table 4 demonstrates that CLOVER consistently outper- forms all other methods across all models and tasks. Specif- ically, on LLaMA-2-7B. CLOVER surpasses LoRA, DoRA, HiRA, and PiSSA by 7.6%, 5.5%, 3.8%, and 0.7%. Even on LLaMA-3-8B, with fewer trainable parameters, CLOVER outperforms by 7.1%, 2.7%, 1.2%, and 0.6%. CLOVER leads most sub-tasks and ranks second in few.
The number of trainable parameters in LLaMA-2-7B matches those used in LoRA, DoRA, HiRA, and PiSSA, all employing rank-32 updates. For LLaMA-3-8B, we re- duce the number of trainable parameters to two-thirds of the amount used in the other models.
These experiments demonstrate that CLOVER possesses strong fine-tuning capabilities, making effective for recov- ering performance degradation caused by pruning. Addi- tional analysis is provided in Appendix D.
8

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
5. Conclusion and Limitations
Acknowledgement
In this paper, we introduce Cross-Layer Orthogonal Vectors (CLOVER), a method that orthogonalizes vectors within attention heads without requiring additional transformation matrices. This orthogonalization process condenses effec- tive parameters into fewer vectors, improving the pruning ratio. By fine-tuning the singular values obtained through orthogonalization, CLOVER learns linear combinations of orthogonal bases, enabling full-rank updates. When applied to prune 50% of the attention head parameters in GPT-2XI,, CLOVER results in perplexity that is just one-tenth of that achieved by standard pruning methods. For Whisper- Large-v3, CLOVER removes 46.42% of the parameters without fine-tuning, while preserving model performance. Furthermore, when used for fine-tuning, CLOVER outper- forms state-of-the-att methods such as LoRA, DoRA, HiRA, and PiSSA, achieving superior results with equal or fewer trainable parameters. We also demonstrate how CLOVER removes linear redundancy to facilitate pruning and discuss the necessity of fine-tuning across all orthogonal bases. Vi- sual comparisons of models fine-tuned with different meth- ods further illustrate its effectiveness.
This work is supported by the National Key R&D Program of China (20222D0160300), National Natural Science Foun- dation of China (62276003), and Kunpeng&Ascend Center of Excellence, Peking University.
Impact Statement
This paper proposes a cross-layer orthogonal initialization method to guide model pruning and efficient fine-tuning, of- fering valuable insights for the application and development of large models. Both application directions aim to reduce training and inference costs, lower computational overhead, decrease power consumption, and minimize carbon emis- sions.
References
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
Despite its advantages, CLOVER has some limitations. When nonlinear operations are present between Q-K or V-O pairs (such as with the uidely-used RoPE (Su al., 2024)), cross-layer orthogonalization is not feasible. In these cases, we instead perform head-wise orthogonalization within the Key layer during fine-tuning. Fortunately, CLOVER Fine-Tuning can apply intra-layer attention head orthogo- nalization, while CLOVER Pruning remains applicable to many popular models, including DeepSeek (DeepSeek-AI. 2024; Liu et al., 20244) uses Decoupled RoPE), ViT and SDXL (which use absolute positional encoding), and BLOOM (Workshop et al., 2022) (which employs Alibi rel- ative positional encoding (Press et al., 2021)). Additionally, as a newly proposed method, our current evaluation fo- cuses primarily on basic pruning tasks and does not include comparisons with other state-of-the-are pruning techniques. However, because CLOVER does not alter the model struc- ture and only updates the initialization method, can be combined with existing pruning methods to further enhance their effectiveness.
AI @ Meta. Llama 2: Open foundation and fine-tuned
chat models. CoRR, abs/22377.09288 2023. doi: 10. 483500arXiiv22000092288,, URL https: /tt.//doorrgoooo 10. 10.48550/axxiv. 230. 09288.
Meta. https://github.c wwt.:/ua...ooooottooaaaaaa..aa blob Btbb:maannMMDEL__CO [A_____ARDD___
Llama 3 model card, 2024. URL
1lama3/
Ainslie, J., Lee-Thorp. J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.1322 13245, 2023.
Anthropic. Claude 3.5 sonnet, 2024. URL https:/ //ww. anthropic. com/ www.//ww/////cccccccc/ccc/www.we www./ 3 5 sonnet.
Ashkboos, S., Croci, M. L., Nascimento, M. G. d.. Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401, 1224401..5222 15024, 2024.
As a novel technique, CLOVER holds considerable promise for future applications. For instance, it could be combined with quantization methods to eliminate outliers, guide prun- ing and fine-tuning based on data feature directions, or even inspire new model architectures.
Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. Smollm-corpur 2024. URL https: / http:///uggggnggggggeoe co datasets/ ctt.gaaaaacccotaattt/////// HuggingFaceTBB http://///moommmmmmmmmmee sm/l smtp//mmmmmmmuuu......m
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and
Choi, Y. PIQA: reasoning about physical common- sense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The
9

---

CLOVER: Cross-Laye Orthogonal Vectors Pruning
Tenth AAAT Symposium on Educational Advances in Artificial Intelligence, EAAT 2020, New York, NY, USA, February 7-12. 2020, pp. 7432-7439. AAAI Press, 2020a. doi: 10. 100099/AAAL1344055.000000 doi wwg.orggg/ooo 609 609//a0i...... v33.aa................ 05. 6239.
Fu, Q.. Cho, M., Merth, T., Mehta, S., Rastegari, M., and Najibi, M. Lazyllm: Dynamic token pruning for efficient long context Ilm inference. arXiv preprint arXiv:240.14055, 2024.
URL https://
Gandhi, S., von Platen, P., and Rush, A. M. Distil-whisper: Robust knowledge distillation via large-scale pseudo la- belling. arXiv preprint ar arXiv:231 110222311100000 .00430, 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020b.
Gokaslan, A. and Cohen, V. pus.
Openwebtext cor-
http://Skyl becicyllonnnn ion00o.o0oooo www.ooo..tttb.ii..... OpenWebleettCorpass 2019. 2019.
Brandon, W. Mishra, M. Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405. 12981, 2024.
Gu, H., Li, W., Li, L.,, Zhu, Q. Lee, M., Sun, S., Xue, W., and Guo, Y. Delta decompression for moe-based Ilms compression. arXiv preprint arXiv:2502.17298, 2025.
Brown, T., Mann, B., Ryder, N., Subbiah, M. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Guo, S., Xu, J., Zhang, L. L., and Yang, M. Com- presso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310000015 2023.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.071433 2022.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: (d=d7KB-jIII /ttp:///ppeeeeeeeeeeeeeeee / net forum?
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions. arXiv preprint 2019.
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length Ilm in- ference with kv cache quantization. arXiv preprint uXiv:2401.18079 2024.
Clark, P., Cowhey, I., Etzioni, O.,, Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/1803.05457, 2018a. URL http:///axxxii
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106.0968.. 2021.
Clark, P., Cowhey, I., Etzioni, O.,, Khot, T.. Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 803.05457, 2018b.
Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, HiRA: Parameter-efficient hadamard high-rank adap- tation for large language models. In The Thirtcenth International Conference Learning Representations, 2025. URL i id=TwJrTz9cRS.
DeepSeek-Al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.00 .44344, 2024. URL /tttps://doi.rrgo 08550/arXiv. 2405. 04434.
net ntt.:/uunettttrt forum?
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 O: 8-bit matrix multiplication for transform- ers at scale. Advances in Neural Information Processing Systems, 35:30318-30332 2022.
Jiang, T., Huang, S., Luo. S., Zhang, Z., Huang, H., Wei, F. Deng, W., Sun, F., Zhang, Q., Wang, D., et al. Mora: High-rank updating for parameter-efficiet fine-tuning. arXiv preprint Xiv:24055.12100 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Jo, H.-r. and Shin, D. A2sf: Accumulative attention scoring with forgetting factor for token pruning in transformer decoder. arXiv preprint arXiv:2407. 20485, 2024.
10

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Li, M., Lin, Y., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models. aiv preprint Siiv:2411.055000 2024a.
Chapados, N., Patwary, M., Tajbakhsh, N., Jernite, Y.,
Ferrandis, C. M., Zhang, L.,, Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H. Starcoder 2 and the stack v2: The next generation, 2024b.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye. H., Cai, T., Lewis, P., and Chen, D. Snapkv: Lim knows what you are looking for before generation. arXiv preprint arXiv:2404. 14469, 2024b.
Ma, X., Fang. G., and Wang, X. Llm-prunce pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Lingam, V., Tejaswi, A., Vavre, A., Shetty, A., Gudur, G. K., Ghosh, J., Dimakis, A., Choi, E., Bojehevski, A., and fine-tuning with singular vectors, arXiv preprint arXiv:2405.1995 19597, 2024.
Meng. F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint ar arXiv:2404.02948, 2024.
Sanghavi, S. Svft: Parameter-efficten
Meng, F., Yao, Z., and Zhang, M. Transmla: Multi- head latent attention is all you need. arXiv preprint arXiv:250.00864 2025.
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04p: 04434, 2024a.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024b.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arxiv: wtt.1088990279.. 2018.
Liu, A., Liu, J., Pan, Z., He, Y. Haffari, G., and Zhuang, B. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405. 2024c.
Mistral. Cheaper, better, faster, stronger: Continuing to push the frontier of ai and making it accessible to all, 2024. URL httts::/missteet /ttp//mmstaaaaeeeeeeeeeeeeaeeele wwt.strallaaaaasssss.saaaaa mixtral-8x22
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F. Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposeed low-rank adaptation. arXiv preprint arXiv:2402.09355. 2024d.
OpenAL. Hello GPT-4o, 2024. URL https: //penai.. hwmm/index/eello-eee
Paster, K., Santos, M. D., Azerbayev, Z., and Ba, Open- webmath: An open dataset of high-quality mathematical web text, 2023.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asym- metric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750. 2024c.
Podell, D.,. English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arviv preprint arXiv:2307.01 019952, 2023.
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu: the finest collection of educational /o/ content, 2024a. URL hitps: fineweb-edd.. datasets / Huggg
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint 108.::2208..24000 2021.
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang, Z., Liu, Q. Abulkhanov, D., Paul, I., Li, Z., Li. W.-D., Risdal, M., Li, J., Zhu, J., Zhuo, T. Y., Zheltonozhskii, E., Dade, N. O. O., Yu, W., Krauß, L., Jain, N., Su, Y., He, X., Dey, M., Abati, E., Chai, Y., Muennighoff, N., Tang, X., Oblokulov, M., Akiki, C., Marone, M., Mou, C., Mishra, M., Gu, A., Hui, B., Dao, T., Zebaze, A., Dehaene, O., Patry, N., Xu, C., McAuley, J., Hu, H., Scholak, T., Paquet, S., Robinson, J., Anderson, C. J.,
Qwen. Qwen2.5: A party of foundation models, 2024. URL https:/ /twp::///ww/mmmmmmmmm/m github. httpb//////g/go/// http///ogggg////////////m /wwn2..ee2222..mm///ee//..../... 5.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training, San Francisco, CA. USA, 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In International conference machine learning. pp. 28492-2883 PMLR, 2023.
Wang, S., Yu, L.., and Li, J. Lora-ga: Low-rank adap-
tation with gradient approximation. arXiv preprint mrXiv:2407.55000 2024a.
Wang. Z., Liang, J., He, R., Wang. Z., and Tan, T. Lora- pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242 2024b.
Sakaguchi, K., Bras, R. L.. Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema chal- lenge at scale. Commun. ACM, 64(9):99-106, 2021a. doi: 10.1145/3474381, Utthtsss: / /doi wtw.ddi..rrgg 10./
Workshop, B., Scao, T. L.., Fan, A., Akiki, C., Pavlick, E., Iliè, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., et al. Bloom: A 176b-parameter open-access multilin- gual language model. arXiv preprint arXiv:2211.05 2022.
1145/3474381
Sakaguchi, K., Bras, R. I.., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-10) 2021b.
Wu, X., Huang, S., and Wei, F. Mixture of lora experts. arXiv preprint arXiv:2404.13628, 2024.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.
Socialiqa: Commonsense reasoning about social interac- tions. arXiv preprint
Xia, M., Gao, T., Zeng. Z., and Chen, D. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.0661 2023.
2019.
Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint 191 1.02150, 2019.
Xiao, G., Lin, J.. Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228. 2024.
Stack Overflow. Stack overflow, 2025. URL https:// stackoverflow com. Accessed: 2025-05-21.
Yu, H., Yang, Z., S., Li, Y., and Wu, J. Effectively com- press kv heads for Ilm. arXiv preprint atXiv:2406.07066 2024.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 12763,,063, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D. R., and Märquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4791-4800, Association for Computational Linguistics, 2019a. doi: 10.116653V1/P199-4477 URL https: //opi. http:///dioo.oo11.1111111 http:///.......////////////11111
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Sun, Y., Dong, L., Zhu, Y., Huang, S., Wang, W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.0055 05254,, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.007 2019b.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a.
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N.,
He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for efffiettrrefffccent fine- tuning. arXiv preprint arXiv:2303.10512. 2023.
Team, G., Riviere M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024b.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient Ilm train- ing by gradient low-rank projection. arXiv preprint ar arXiv:2403.0550) 033007 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient founda- tion language models. arXiv preprint 1rX1v:2302.13971, 2023.
Zi, B., Qi, X., Wang, L., Wang, J., Wong, K.FF,, and Zhang, L. Delta-lora: Fine-tuning high-rank parame- ters with the delta of low-rank matrices. arXiv preprint arXiv:2309.022. 022411 2023.
12

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Zuhri, Z. M. K., Adilazuarda, M. E. Purwarianti, A., and Aji, A. F. Mikv: Multi-layer key-value heads for memory efficient transformer decoding. arXiv preprint xxXiv2240.00227,,0000 2024.
13

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
A. Dataset and Hlyper-Paraeeto for Table 1
Following the experimental setups of TransMLA (Meng et al., 2025), we fine-tune our models using the prtraining corpus from SmolLM (Ben Allal al., 2024). The dataset comprises SineWeb-Edu Edu-Dedu (Lozhkov al., 2024a). Cosmopedia- v2 synthetic dataset generated by Mixtral (Wu et al., 2024), Python-Edu from StarCoder (Lozhkov al., 2024b), Open- Web-Math (Paster et al., 2023), and data from StackOverflow (Stack Overflow, 2025).
Table 6. Composition of the training dataset.
Dataset
Sampling Weight
fineweb-edu-deduup
0.70
cosmopedia-v2
0.15
python-edu
0,06
open-web-math
0.08
stackoverflow
0.01
To ensure fair comparison, we replicate the dataset mixing ratios used in the TransMLA setup, as shown in Table 6, to maintain experimental consistency. DeepSeek-V2-- V2-Lite is trained on 1B tokens for both SliceGPT and CLOVER using SliceGPT's hyperparameters, For LLaMA-2-7B, we apply CLOVER top of the checkpoint released by TransMLA, which pruned 93% of the KV cache and was trained on 6B tokens. Additionally, we further train the model with 1.5B tokens for various pruning ratios using TransMLA's hyperparameters.
B. Dataset and HlyperPaaramedt
for Table 4
The commonsense reasoning tasks consist of 8 subtasks, each with predefined training and testing sets, described by LLM-Adapters (Hu et al., 2023). The following table lists the details of each sub-dataset.
Table 7. Details of datasets for commonsense reasoning tasks.
Dataset
Train
Test
About
BoolQ (Clark al., 2019)
9,427
3,270
Naturally occurring yes/no questions from unconstrained settings.
PIQA (Bisk al. 2020b)
16.113
1,838
Questions with two solutions requiring physical commonsense.
SIQA (Sap et al., 2019)
33,410
1,954
Reasoning about actions and social implications.
HellaSwag (Zellers et al., 2019b)
39,905
10,042
Commonsense NLI questions with context and endings.
WinoGrrande (Sakaguchi et al., 2021b)
40,398
1,267
Fill-in-the-blank task with hinary options.
ARC-e (Clark et al., 2018h)
2,251
2,376
Grade-school multiple-choice science questions in Easy sets.
ARC-c (Clark et al., 2018b)
1,119
1,172
Grade-school multiple-choice science questions in Challenge sets.
OBQA (Mihaylov et al., 2018)
4,957
500
Questions requiring multi-step reasoning and commonsense knowledge.
For WinoGrande, the original dataset includes multiple partitions: [xs, s, m, I, xl, debiased]. While LLM-Adapters simply concatenated all these partitions, note that the "xl" partition actually includes all others, leading extensive data duplication. After removing duplicates, the training data is reduced from 63.2K to 40.4K instances.
Additionally, in the LLM-Adapters paper, the training set sizes of AhC.Challenge and ARC_Easy were reversed by mistake; here, we correct that error.
The results for LoRA and DoRA presented in Table are directly taken from the original DoRA paper, where the hyperparameters are carefully tuned. Similarly, the results for HiRA are cited from its original publication. In contrast, we introduce new experimental results for PiSSA and CLOVER, both of which are optimized with the best learning rates (Table 8) and aligned hyperparameters. Specifically, PiSSA achieves optimal performance at learning rate of 2e-5, while CLOVER performs best at le-4, as shown in the table below:
Table presents a comparison of hyperparameters for different fine-tuning methods on commonsense tasks. The target model remains the same for LoRA, DoRA, HiRA, and PiSSA. However, DoRA introduces an additional magnitude module, leading to a slightly higher parameter count. In single layer of LoRA, the trainable parameters are as follows:
14

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Method
Learning Rate
Acc
le-4
80.1
5e-5
82.9
PISSA
3e-5
84.1
2e-5
84.5
le-5
83.6
5e-4
79.0
2e-4
83.9
CLOVER
le-4
85.2
5e-5
84.3
2e-5
82.8
Table 8. Learning rate searching.
In LoRA, the trainable parameters are:
Q 4096 x 32 + 4096 32
K 4096 x + 4096 32
V 4096 x 32 4096 32
Up 4096 x 32 + 11008 x 32
Down 4096 x 32 + 11008 x 32
The total sum is 1,753,088. In CLOVER, the trainable parameters are:
QK = 32 x 128 x 128
VO 32 x 128 128
UD 172 64 64
The total sum is also 1,753,088.
Since CLOVER inserts trainable parameters across layers, we use the Q-K pair notation to represent its target model. When CLOVER updates parameters within attention head, the number of trainable parameters matches exactly that of LoRA rank 32. To adjust the number of learnable parameters, CLOVER multiple blocks. Both PiSSA and CLOVER exhibit stable training performance. Therefore, instead of validating every 80 steps, we omit frequent validation, improving training efficiency.
either span multiple heads split single head into
Table 9. Detailed Training Hyperparameters. Q-K.V-O, means CLOVER update pair of orthogonal vectors.
Evaluation steps
Batch size
Warmup Steps
Method
Target
LR
Scheduler
Epochs
LoRA DoRA Q.K.V.U.D
Q.K.V.UDD
80
3e-4
Linear
16
100
3
80
2e-4
Linear
16
100
3
HiRA Q.K.V.UD
80
1e-4/2e-4
Linear
32
100
3
PiSSA Q.K.V.UDD
2c-5
Linear
16
100
3
CLOVER
Q-K,V-O, U-D
le-4
Linear
16
100
3
C. LibriSpeech Long dataset target transcript
Below the reference of the LibriSpeech Long dataset for comparison.
15

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. Nor is Mr. Quilter's manner less interesting than his matter. He tells us that this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. He has grave doubts whether Sir Frederick Layton's 's work is really Greek after all, and can discover in it but little of rocky Ithaca. Linnell's s pictures are sort of Up Guards and Adam paintings, and Mason's 's exquisite idles are as national as a jingo poem. Mr. Birket Foster's s landscapes smile at one much in the same way that Mr. Carker used to flash his teeth, and Mr. John Collier gives his sitter a cheerful slap on the back before he says, like a shampooer in a Turkish bath, next man.
In fact, with Vanilla Pruning ratios of just 22.31% % and 6.69% for Wo-Wx and Wy-Wo, respectively, the model's output already significantly degraded.
Mr. Colter is the personal of the classes, and we are glad to welcome his gospel. Nor is Mr. Colter's manner less interesting than his manner. He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similarty he is drawn from eating and its results occur most readily to the mind. He is very dull, so very frequently, and is very Greek after all, and can discover in it but littl of Rocky Ithuca. The Nell's pictures are sort of up-guard to Adam's paintings, and Mason's 's exquisite idylls are as national as a jingle poem. Mr. Burke and Foster's landscapes smile at one much in the same way as Mr. Parker, Mr. Flash is tits. And Mr. John Collier gives his sitter cheerful slap on the back before he says like shampoo and Turkish bath, Next man.
D. Further Analysis of CLOVER's Fine-Tunii Capability
D.1. Necessity of Full-Direction Fine-Tuniee -unniT
Besides pruning with a large ratio, CLOVER is capable of learning linear combinations of all orthogonal vectors within each attention head. This capability allows CLOVVRR to resemble full- full-parameter fine-tuning more closely. To highlight the advantages of updating all orthogonal bases, we randomly sampled 16 instances from the Commonsense dataset, fed them into the model, and performed SVD to the model. We then recorded the projection magnitudes of input features along all orthogonal directions. Figure 5 visualizes the results for the middle layer, revealing the following insights:
Top 256
Bottom 256
Top 256
Bottom
Next
Bettem
Next 256
6%
10% 7%
7%
7%
10%
76%
81%
100%
Bemaining
Remaining
Remaining
(a) PiSSA w/o Singular Value
(b) LoRA
(c) PiSSA
(d) CLOVER
Figure 5. Proportion of data projections across different components in random directions (LoRA) versus orthogonal directions (PiSSA). as well as all orthogonal directions (CLOVER).
1) Without accounting for the scaling effect of singular values, the projection magnitude along the principal singular vector consistently exceeds that in other directions. This observation supports PiSSA's approach, which updates based on the principal singular values and vectors, leading to improved training performance. In contrast, LoRA projects in random directions, resulting in uniform projection magnitudes across all directions.
2) The singular values in the original model reflect the importance of each direction in the pretraining task. The model amplifies the components along directions with larger singular values and suppresses those along smaller singular values. Therefore, it is crucial to consider the scaling effect of singular values. As shown in Figure 5c, the projection magnitude along the principal singular vector direction increases to 18%.
3) While more data projections align with the principal singular vector at higher ranks, 82% of the feature components are still projected onto other directions. In extreme cases, if a task is entirely orthogonal to the vectors used by PiSSA, training
16

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
on such task may result in zero gradients, thereby limiting its learning capacity. Under the same rank constraint, 94% of the feature components in LoRA projected outside the LoRA adapter, making it more susceptible the zero-gradient problem.
Since CLOVER updates across all orthogonal directions, as shown in Figure 5d it effectively mitigates this issue. Conse- quently, CLOVER outperforms both LoRA and PiSSA multi-task learning, even when using the same or fewer learnable parameters (Section 4.6).
D.2. Visualizing Rank Updates
To demonstrate CLOVER achieves full-rank updates, we multiply the updated singular values with their corresponding singular vectors and perform SVD on the base model (Sor applied to the Key layer, Svo to the Value layer, and SuD to the Up layer). We take LoRA, and Full Fine-tuning for comparing. Figure shows the singular value of the middle layer in LLaMA-2-7B. revealing that CLOVER and Full Fine-tuning achieve full-rank updates, while LoRA is constrained by its low-rank design.
2.0
0.8
6-
1.5
0.6
1.0
0.4
2:
0.5
0.2
0.0
0.0
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
(a) Full Fine- Tunnng
(b) LoRA
(c) CLOVER
Figure 6. AW low rank in LoRA, while full rank for Full-Fine-Tuning and CLOVER.
D.3. CLOVER Avoids Intrusive Dimensions
Recent research (Shuttleworth et al., 2024) has highlighted an issue with LoRA, referred to as the "intrusive dimensions" phenomenon. As illustrated in Figure 7b, LoRA introduces new random directions into the model, which possess large magnitudes and thus precede all the original singular vectors. The study suggests that these "intrusive dimensions" can degrade the model's performance, exacerbating catastrophic forgetting during continual learning with LoRA. contrast, CLOVER addresses this issue by fixing all orthogonal bases and updating only the vector combinations. As result, the changes introduced by CLOVER fine-tuning closely resemble those generated by full parameter fine-tuning, shown in Figure 7a and Figure 7c
D.4. Cross Layer Orthogonal Vectors in Value and Output layers
In the main text, only presented the orthogonalization process for the Q-K pair. Here, provide the method for orthogonalizing the V-O pair. Additionally, for up-down layers, the output dimension of the Up layer can be reshaped into block number x block size, followed by performing orthogonal decomposition within each block.
Y (Qn, Kn)VWo, V XWv E bbxxxxxd
(1)
= (Qt, KW) WvWo = Wvo = USV E DxD
(2)
attn(Q KU)XUSV Spaevorradd Svo E ERAXrooXPes = 0,r S d.
(3)
attn(Q,, kun(Q@n,/k)XUUvvsvvooooa Uvo BR Dxhiruo Vvo E x
(4)
17

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
0
100-
100
100
200
200
200
300
300
300
400
400
400
100
200
300
100
200
300
400
100
200
300
400
(a) Full Fine-Tuning
(b) LoRA
(c) CLOVER
Figure 7. Intruder dimensions phenomenal in LoRA, which does not exist in Full Fine- Fine-Tuning and CLOVER.
Through this series of transformations, Wv and Wo be equivalently replaced by orthogonal vectors Uvo and Vvo. along with the diagonal matrix Svo. Since Two S d. the singular zero values and their corresponding singular vectors can be safely pruned. After guided pruning, Svo can be merged into Uvo and Vvo, resulting in no additional computational overhead.
E. Visualizing more attention heads
In Section 4.2, only presented the first attention head in the first layer. Here, provide broader view by showcasing more attention heads. Figure 8 illustrates the La norm of all Q-K heads in the first, middle, and last layers of Whisper-Large- v3. Figure 9 shows the Le norm of all Q-K heads in the first, middle, and last layers of ViT-bigG.
From these figures, we can observe that CLOVER consistently represents the entire attention head with fewer orthogonal bases across all layers and all attention heads. This property forms the foundation of CLOVER's 's effectiveness enhancing pruning.
18

---

CLOVER: Cross-Layer Orthogonal Vectors Pruning
Layero.ck
20
Decompose
230
4.0
620
000
1000
1200
Layer: qk
600
Layer.31. ak
22.5
10.0
5.0
2.5
0.0
250
620
1000
1200
Figure Lo-norm 0-th, 15-th, 31-st attention layers in the Whisper-lare--3 encoder. The blue represents results after redundancy removal using the CLOVER method, while the orange line depicts the L2-norm directly computed for each dimension.
Layer:
Decompose
250
500
730
1000
1250
1503
Layen gk
2.0
1.5-
1.0
0.5
0.0
Layer qk
1.0
0.8-
0.5-
0.4
02-
0.0
503
750
1000
1250
3500
Figure The norm 0-th, 15-th, and 31-st attention layers in ViT- The blue line represents the results after redundancy removal using the CLOVER method, while the orange line depicts the Lu-norm directly computed for each dimension.
19


Reviews:
[{"review_id": "Er7zncjPlk", "reviewer": "Reviewer_DDfo", "summary": "Decoder - only models face memory issues during inference as the key/value cache grows. This paper introduces CLOVER to address this by treating attention layers as low - rank decompositions. CLOVER applies SVD to the Q−K and V−O pairs in each attention head. The resulting singular values can guide pruning or be used for efficient fine - tuning without increasing the model's parameter count.\nExperiments on models like GPT - 2 XL, Whisper - Largev3, and LLaMA - 3 show that CLOVER improves pruning efficiency. For example, it can prune more vectors in GPT - 2 XL with less performance degradation. In fine - tuning, CLOVER outperforms state - of - the - art methods such as LoRA, DoRA, HiRA, and PiSSA on commonsense reasoning tasks. It can achieve full - rank updates and better performance with equal or fewer trainable parameters.\nAlthough CLOVER has limitations when dealing with certain non - linear operations, it can be combined with other pruning methods and has potential for future applications, like being combined with quantization methods.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Clarification of Domain**\n\n**Q1.1: Whether CLOVER is a PEFT method.**\n\n**A1.1:** CLOVER is a straightforward yet effective re-initialization method that benefits **both pruning and PEFT**. Pruning and PEFT are closely interconnected, as both aim to achieve efficient training and inference under resource constraints. Typically, pruning requires a recovery process, and PEFT methods are commonly applied to minimize training overhead and enhance efficiency, particularly in data-scarce scenarios [1]. While many previous pruning methods primarily focus on the pruning phase and directly apply LoRA in the recovery phase [1, 2, 3, 4], **CLOVER uniquely addresses both tasks simultaneously**. It not only enhances pruning rates (comparing with vanilla pruning and SliceGPT, refer to **A3 for reviewer jpx5**) but also improves PEFT training effectiveness. We believe this dual approach provides valuable insights for efficient model deployment.\n\n**Q1.2: Whether CLOVER can only be used for LLMs.**\n\n**A1.2:** Indeed, our proposed CLOVER method targets on the attention component. But since **attention mechanisms are widely used across many domains beyond LLMs** (such as Vision Transformers (ViT) for image classification, detection, segmentation, DiT for image generation, and Automatic Speech Recognition, as well as Text-to-Speech), CLOVER has potential and value in a wide range of applications. Additionally, in Section 4.4, we provide experimental evidence using the **speech recognition** model Whisper, where CLOVER achieves parameter reductions of 56.01% in the Q-K pair and 36.82% in the V-O pair without the need of further fine-tuning. CLOVER’s flexibility allows it to integrate with other techniques, extending its applicability to other models.\n\n**Q2. GPU Memory Consumption and Runtime Comparison with LoRA**\n\n**A2:** We conducted experiments on the LLaMA-2-7B model using commonsense data, trained for 3 epochs with the following hyperparameters: model_max_length = 1024, per_device_train_batch_size = 2, gradient_accumulation_steps = 2, num_gpus = 4, executed on 4 × A800-80G GPUs, with seed = 42. The comparison of memory consumption and runtime is as follows:\n\n| **Method** | **Trainable Parameters** | **Total Parameters** | **Max Allocated Memory** | **Runtime** |\n| --- | --- | --- | --- | --- |\n| LoRA | 56,098,816 | 6,794,514,432 | 110.84 GB | 2:42:37 |\n| CLOVER | 56,098,816 | 6,794,514,432 | 104.75 GB | 2:22:47 |\n\nthe results demonstrate that CLOVER consumes less GPU memory and exhibits shorter training runtime compared to LoRA. We attribute this to CLOVER being applied between two layers, whereas LoRA operates in parallel with the main branch. This enables sequential computation, eliminating the need to retain the input features of the main branch.\n\n**Q3. Rigor in Demonstrating Full-Rank Adaptation’s Advantages and Information Redundancy**\n\n**A3:** In Section 4.5, we emphasize the importance of full-rank adaptation by demonstrating that only 6% of the orthogonal components from randomly selected commonsense samples project onto low-rank vectors, leading to inefficient data utilization. Full-rank adaptation resolves this issue by fully utilizing the data’s orthogonal projection components. Sections 4.6 and 4.7 visualize the matrices fine-tuned by various methods, showing that CLOVER’s effects closely resemble full-parameter fine-tuning, whereas LoRA, constrained to low-rank updates, introduces intrusive dimensions. Several studies [5, 6, 7, 8] have rigorously compared full-rank and low-rank adaptation, consistently concluding that full-rank updates outperform low-rank methods. Regarding the concern about information redundancy, we believe this issue is more closely related to the number of trainable parameters rather than the direction of adaptation itself. Since CLOVER maintains parameter efficiency, it mitigates redundancy effectively while offering greater flexibility and learning capacity.\n\n**Q4. Same as Q2**\n\nWe hope that these responses address your concerns regarding CLOVER, and we look forward to further discussing any additional points you may have.\n\n### Reference:\n\n[1] LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS 2023.\n\n[2] SlimGPT: Layer-wise Structured Pruning for Large Language Models, NeurIPS 2024.\n\n[3] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LargeLanguage Models, ICML2024.\n\n[4] SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR24.\n\n[5] HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models, ICLR 2025.\n\n[6] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024.\n\n[7] QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation, NeurIPS 2024.\n\n[8] Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices, arxiv preprint.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. Mainly for applications, no proof included.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Not all. See the weakness 1.", "essential_references_not_discussed": "Yes.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. This paper is well organized, motivations and method details can be clearly understood.\n\n2. The proposed method can achieve better accuracy with similar or fewer parameters.\n\n3. The paper provides sufficient technical details and implementation parameters, and fine-tunes tasks for as many domains as possible, e.g., NLP, CV.\n\n**Weaknesses:**\n\n1. The domain of this paper is unclear. From my understanding, the proposed method can only be used for models that contain attention mechanisms (please correct me if I am wrong), not all linear-layer models. Although experiments on stable diffusion are included, this is most likely the attention part between the text encoder and u-net, rather than the core component of SD. So, first of all, should the paper even consider including a claim like, for example, a xxx method for large language models? Second, this paper does not seem to explicitly state that this is a PEFT method, while the baselines are almost all PEFT methods. This is important in my opinion, because pruning and efficient fine-tuning are distinct, at least, they should consider the unique characteristics of training and fine-tuning, respectively. Therefore, in summary, two points are unclear: a. whether it can only be used for LLMs; b. whether CLOVER is a PEFT method.\n\n2. I will use the PEFT one to understand CLOVER. As for the PEFT method, the authors show the performance with similar or fewer parameters, which is certainly correct. However, should also consider to show the GPU memory consumption and running time compared to LoRA. For example, the time to run one epoch with the same batch size and max length.\n\n3. Full rank adaptation may be intuitively reasonable, but, to my knowledge, few papers have rigorously demonstrated this. So I do not recommend authors to make such STRONG claims. Moreover, full rank may also lead to information redundancy. The author can consider whether this is one of the main arguments of CLOVER. If so, please elaborate on it. If not, please hide it. \n\n4. Could the authors please provide a comparison of GPU memory consumption between CLOVER and LoRA? No need for complete fine-tuning, just provide the GPU memory data, for example, you can use the smallest batch size to perform several steps.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKGXhmsKrZ", "reviewer": "Reviewer_jpx5", "summary": "The manuscript introduces CLOVER which orthogonalizes the Query, Key, Value, and Output vectors in the attention layers, aiming to reduce the computational overhead and thus guiding pruning and serving for effective fine-tuning. Specifically, it is based on treating pairs of attention layers as low-rank decompositions using Singular Value Decomposition (SVD). CLOVER applies SVD to Query-Key (Q-K) and Value-Output (V-O) pairs, allowing the extracted singular values to guide pruning or act as trainable parameters for efficient fine-tuning. Experiments demonstrate the superior performances compared with other state-of-the-art methods such as LoRA, DoRA and PiSSA in PEFT tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful and constructive feedback. We have carefully addressed each of your concerns and will incorporate your valuable suggestions into the camera-ready version of the paper.\n\n**Q1: Clarify how the proposed method reduces memory load during inference.**\n\n**A1:** Autoregressive token generation in LLMs involves frequent access to the previous key-value (KV) cache, which can significantly hinder inference speed [1]. CLOVER addresses this issue by reducing the dimensionality of attention heads, thereby reducing the KV cache size and effectively lowering memory usage during inference. As noted in **A1 to Reviewer AvA1**, this reduction in KV cache size contributes to more efficient processing during inference.\n\n**Q2: Clarify the confusing descriptions of CLOVER and CLOVER†.**\n\n**A2:** Thank you for pointing out the confusion between CLOVER and CLOVER†. We acknowledge this naming conflict, especially in pruning experiments where both techniques are used together. To resolve this, we will clearly redefine the terms in the camera-ready version:\n\n•\t**CLOVER:** After pruning the less important components based on singular value magnitudes, **only the singular values are fine-tuned**, which are then merged back into singular vectors (this corresponds to the original **CLOVER†**).\n\n•\t**CLOVER†:** After pruning the less important components, they are directly merged into singular vectors, followed by **full-parameter fine-tuning** (this corresponds to the original **CLOVER**).\n\nThis adjustment will ensure clarity and consistency across both the methodology and experimental sections. In this context, CLOVER† serves as an ablation study, validating the effectiveness of the orthogonal initialization method in comparison to vanilla pruning.\n\n**Q3: Setting of Vanilla Pruning and comparison with SoTA methods.**\n\nVanilla pruning differs from CLOVER primarily in that it does not perform orthogonalization. Instead, we directly prune the head dimension based on the vector norm.  (Lines 182-183 right, 275-293 left). CLOVER is orthogonal to existing pruning methods and can be effectively combined with techniques like SliceGPT [2]. In response to your suggestion, we compared the combination of CLOVER and SliceGPT with SliceGPT alone on both OPT and Deepseek models, reporting the perplexity results on WikiText2, as shown in the table below:\n\n| **Method** | **Attention Pruning Ratio** | **MLP Pruning Ratio** | **OPT 6.7B** | **Deepseek V2-Lite** |\n| --- | --- | --- | --- | --- |\n| Baseline | 0% | 0% | 10.85 | 6.31 |\n| SliceGPT | 25% | 25% | 11.90 | 8.65 |\n| CLOVER | **30%** | 25% | **11.89** | **8.53** |\n\nThe baseline represents the original model with no pruning. CLOVER achieves performance comparable to SliceGPT, despite pruning 30% of attention parameters, compared to SliceGPT’s 25%. This demonstrates the simplicity and effectiveness of CLOVER.\n\nIt is important to note that most current pruning techniques for Deepseek primarily focus on reducing the number of MoE experts [3] and token selection [4]. There is limited research on pruning attention heads, attention head dimensions, or latent dimensions. CLOVER facilitates pruning these components in Deepseek’s Multi-Layer Attention (MLA), significantly reducing the KV cache overhead. Additionally, CLOVER utilizes a small number of dominant singular values from the Query-Key matrices to dynamically identify important tokens for full computation, enhancing token selection accuracy, especially in long-context tasks when combined with methods like MoBA [4]. Moreover, by rotating Q-K and V-O pairs in attention heads, CLOVER minimizes outliers in weights and activations [5], which is advantageous for reducing quantization errors. We are excited to continue exploring CLOVER to optimize the efficient deployment of Deepseek models.\n\n**Q4: Address the inconsistency in the abstract between the submitted PDF and the ICML submission system.**\n\n**A4:** Thank you for bringing this to our attention. We will promptly update the abstract in the ICML submission system to ensure that it is consistent with the content in the submitted PDF.\n\nWe hope this addresses your comments and concerns effectively. We look forward to your feedback and the opportunity to further improve our work.\n\n**References:**\n\n[1] Keep the Cost Down: A Review on Methods to Optimize LLM’s KV Cache Consumption, COLM 2024.\n\n[2] SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR 2024.\n\n[3] MoE-Pruner: Pruning Mixture-of-Experts Large Language Models Using Hints from Its Router, arXiv preprint.\n\n[4] MoBA: Mixture of Block Attention for Long-Context LLMs, arXiv preprint.\n\n[5] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, arXiv preprint.", "claims_and_evidence": "This paper overclaims the contributions of the CLOVER framework to cache optimization during the inference phase. The authors attribute the challenges in the development of existing models to \"memory and communication bottlenecks\" [L051-054 (Left)]. However, it is unclear how the proposed method reduces memory load in the inference phase.", "methods_and_evaluation_criteria": "The descriptions of CLOVER and CLOVER† are somewhat confusing. \nIn [L141-143 (Right)], the authors demonstrate the CLOVER fine-tuning as “We freeze the matrices UQK h [:, : r] and VQK h [:, : r], and only fine-tune the singular values SQK h [: r, : r].” Besides, in [L190-193(Right)], the authors clarify the CLOVER† as “In the CLOVER† case, after pruning, S is not immediately merged into the U and V matrices but is used for parameter-efficient fine-tuning, with the merging occurring afterward.” \nThe two descriptions appear to be the same, which raises questions about what is the difference between CLOVER and CLOVER†.", "theoretical_claims": "I have checked the correctness of proofs for theoretical claims in Multi-Head Self-Attention Setup, Cross Layers Merging and Cross Layers Orthogonal Decomposition. All the formulations are correct.", "experimental_designs_or_analyses": "The authors only compare the performance of CLOVER with Vanilla method in the context of pruning, without conducting comparisons with other state-of-the-art methods.\nFirst, they do not explain the specific setting of “Vanilla” method. Second, it is insufficient to demonstrate the effectiveness of CLOVER by merely comparing with Vanilla method. It is recommended that the authors supplement their findings with recent baselines.", "relation_to_broader_scientific_literature": "This paper shows a promising method in model pruning and parameter-efficient fine-tuning by orthogonalizing the Query, Key, Value, and Output vectors in the attention layers with SVD.", "essential_references_not_discussed": "There are not any related works that are essential to understanding the key contributions of this paper but not currently cited in this paper.", "other_strengths_and_weaknesses": "Strengths:\n1.\tEfficient Pruning: From theoretic perspective, the method of matrix compression using SVD is novel and effective. It removes redundant vectors while keeping critical features intact, making pruning more effective.\n2.\tParameter Efficient Fine-tuning\na)\tFrom performance perspective, CLOVER surpasses LoRA, DoRA, HiRA, and PiSSA on commonsense tasks, achieving up to 7.6% higher accuracy with equal or fewer trainable parameters.\nb)\tCompared to other PEFT methods, CLOVER still has some advantages. For example, CLOVER alleviates the intrusive dimensions problem identified in LoRA.\n\nWeaknesses:\nSee “Claims And Evidence”, “Methods And Evaluation Criteria” and “Experimental Designs Or Analyses”.", "other_comments_or_suggestions": "The abstract of this paper in submitted PDF is inconsistent with that submitted to the ICML system.", "questions_for_authors": "Please clarify the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIVUR17RzU", "reviewer": "Reviewer_grik", "summary": "The paper addresses the memory increase in large language models due to kv caching and applies SVD to the pairs of KV and QO matrices. After an SVD decomposition, the new representation can be used for efficient pruning or for fine-tuning. The authors include experiments on both these fronts, showing that the proposed method allows for higher pruning ratios for the same performance compared to Vanilla pruning improvements over various PEFT methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Is the comparison in Section 4.5 fair, considering that LoRA is designed for low-rank?**\n\n**A1:** As noted in **A2 for Reviewer DDfo**, CLOVER and LoRA have an identical number of trainable parameters. CLOVER offers slight improvements in both training time and GPU memory consumption when compared to LoRA.The original paper also present comparisons with HiRA, a full-rank update method.\n\n**Q2.1:** Are the experiments conducted with a single learning rate per method and a single seed?\n\n**A2.1:** The results for LoRA and DoRA presented in Table 2 are directly taken from the original DoRA paper, where the hyperparameters are carefully tuned. Similarly, the results for HiRA are cited from its original publication. In contrast, we introduce new experimental results for PiSSA and CLOVER, both of which are optimized with the best learning rates and aligned hyperparameters. Specifically, PiSSA achieves optimal performance at a learning rate of 2e-5, while CLOVER performs best at 1e-4, as shown in the table below:\n\n| **Method** | **Learning Rate** | **Acc** |\n| --- | --- | --- |\n| PiSSA | 1e-4 | 80.1 |\n|  | 5e-5 | 82.9 |\n|  | 3e-5 | 84.1 |\n|  | 2e-5 | **84.5** |\n|  | 1e-5 | 83.6 |\n| CLOVER | 5e-4 | 79.0 |\n|  | 2e-4 | 83.9 |\n|  | 1e-4 | **85.2** |\n|  | 5e-5 | 84.3 |\n|  | 2e-5 | 82.8 |\n\nIn the original paper, PiSSA and CLOVER were trained using the default seed (42). Following your suggestion, we tested multiple seeds (40, 41, and 42). We will conduct these experiments on additional models and include the updates in the camera-ready version.\n\n| **Model** | **Method** | **Avg.** |\n| --- | --- | --- |\n| LLaMA-7B | PiSSA | 82.7±0.06 |\n|  | CLOVER | 83.3±0.35 |\n| LLaMA2-7B | PiSSA | 84.3±0.26 |\n|  | CLOVER | 84.9±0.23 |\n\n**Q2.2:** Why was the learning rate of CLOVER† adjusted from 6e-4 to 6e-3?\n\n**A2.2:** Both Vanilla Pruning and CLOVER employ full-parameter fine-tuning and same learning rate, while CLOVER† fine-tunes only singular values. Generally, PEFT converge more slowly than full-parameter methods. To speed up convergence, we increased the learning rate and removed weight decay for CLOVER†.\n\n**Q3: How does CLOVER compare to SOTA methods?**\n\n**A3:** Please refer to **A3 for Reviewer jpx5**.\n\n**Q4: What does the y-axis represent in Figure 2?**\n\n**A4:**  After applying CLOVER, we multiply the square root of the singular values by each singular vector and calculate their L2 norms. This result is equivalent to directly using the singular values. Therefore, we uniform this representation for better comparison.\n\n\n**Q5: Section 4.4 presents a single example, which might seem like cherry-picking.**\n\n**A5:** The example is not cherry-picked; it is an official sample provided with the Whisper-large-v3 model [1]. Due to significant linear redundancies in the attention heads of the Whisper model, CLOVER demonstrates substantial effectiveness. We aimed to highlight this clearly in the paper. We plan to systematically evaluate the Whisper model and, as suggested, move this case study to supplementary materials.\n\n**Q6: What are the “series” and “parallel” baselines?**\n\n**A6:** “Series” and “parallel” are two standard baselines commonly used in parameter-efficient fine-tuning for NLP tasks. The series adapter [2] inserts a trainable module sequentially after the Attention and MLP layers, while the parallel adapter [3] integrates a trainable module parallel to the Attention and MLP layers.\n\n**Q7: Why is D compared with a single head in the experiments?**\n\n**A7:** The comparison is fair because we are comparing the reduced dimensions within each attention head, not the entire set of heads against a single head. This process reduces the dimension of each original attention head from $D \\times d$ to $D \\times r$, where $D$ is the hidden size, $d$ is the attention head dimension, and $r$ represents the attention head rank, with $r \\leq d \\ll D$.\n\n**Q8: What is CLOVER†?**\n\n**A8:** Please refer to A2 for Reviewer jpx5.\n\n**Q9: Are LoRA and DoRA properly tuned?**\n\n**A9:** Please refer to A2.1.\n\n**Q10: What does L262 mean by “Due to the non-linear…matrix”?**\n\n**A10:** Matrix merging and decomposition are linear operations, which require linearity to preserve equivariance during orthogonalization. However, RoPE encodings vary depending on token positions, introducing non-linearity. This prevents cross-layer merging and decomposition. To address this, we perform orthogonal decomposition within a single layer rather than across layers when fine-tuning Q-K Vectors with RoPE.\n\n**Q11: What does the x-axis represent in Figure 1d?**\n\n**A11:** It represents the pruning ratio.\n\nThank you for your thoughtful and constructive feedback, which has significantly improved our paper. Please feel free to reach out if you have any further questions or concerns.\n\n[1] https://huggingface.co/openai/whisper-large-v3\n\n[2] Parameter-Efficient Transfer Learning for NLP, ICML 2019.\n\n[3] Towards a Unified View of Parameter-Efficient Transfer Learning, ICLR 2022.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the fine-tuning part. For pruning, the only baseline is vanilla pruning.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experiments are in general sound. An issue is with section 4.4 where the results are not convincing. The more important aspect is the lack of baselines form the pruning literature in Section 4.1 and Table 1.", "relation_to_broader_scientific_literature": "The paper discusses connections with the PEFT and LLM compression literature and uses standard (foundation) models for evaluation.", "essential_references_not_discussed": "I believe that all essential references are included.", "other_strengths_and_weaknesses": "## Strengths\n\n1. The paper addresses two important issues in the literature  \n2. The idea to treat the KV and VO matrices as pairs is interesting. The writing on the method is clear and the method itself intuitive. (some other parts are not clear \\-\\> see weaknesses)  \n3. The experimental validation contains multiple different settings to test the method’s effectiveness..\n\n## Weaknesses\n\n1. The paper is not clear at some points.   \n   1. How is section 4.5 a fair comparison given that LoRA is is by design low-rank?  \n   2. See questions for other instances  \n2. The hyperparameter selection seems to be not thorough: only one learning rate per method and one seed (?). Another example can be “We adjust the learning rate from 6e-4 to 6e-3 and remove weight decay, while keeping other hyperparameters consistent with the other two methods.”  \n3. The pruning literature is vast but only vanilla pruning is used for comparison", "other_comments_or_suggestions": "1. Abstract: “these values are reintegrated…count” can be rephrased for clarity  \n2. Figure 2 y axis can be the singular value itself with x axis the index. The current presentation is confusing.  \n3. Section 4.4 presents a single example (e.g. Figure 3 adds no value) and it seems like cherry picking. Imo should be moved to appendix or removed altogether.  \n4. Conclusion: CLVOER instead of CLOVER", "questions_for_authors": "1. What are the “series” and “parallel” baselines?  \n2. L128 (2nd column): “Instead..overhead”. Is the comparison here fair? $D$ corresponds to all heads but is compared with a single head?  \n3. What is CLOVER with the cross superscript? It has not been introduced in the manuscript.  \n4. Can you explain the vast performance difference compared to LoRA and DoRA? Are these methods properly tuned?  \n5. L262 “Due to the non-linear…matrix”: can you clarify this sentence?  \n6. Xaxis of Figure 1d?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWLgWJerLN", "reviewer": "Reviewer_AvA1", "summary": "This paper proposes a method dubbed CLOVER to address the memory-bound in large language models during inference. Specifically, CLOVER performs singular value decomposition (SVD) on the Query-Key and Value-Output parameter matrices in the attention layer, thereby orthogonalizing the vectors within attention heads to reduce linear redundancy. This enables efficient pruning and parameter efficient fine-tuning. Experimental results demonstrate that CLOVER can remove more redundancy than the vanilla pruning method and achieves performance improvements with parameter efficient but full-rank fine-tuning while avoiding the issue of intrusive dimensions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Benchmark the effectiveness of the proposed method.**\n\n**A1:** For the pruning experiments, we evaluated the effectiveness of CLOVER using the Wikitext2 eval-dataset with the following settings: batch size of 32, max sequence length of 1600, and tested on an A100-PCIE-40GB GPU with different pruning ratios for GPT-2-XL inference speed. The results are summarized in the table below:\n\n| **Pruning Ratio** | **Vanilla Perplexity** | **CLOVER Perplexity** | **Median Time per Batch (s/batch)** | **Throughput (token/s)** | **Latency (ms/token)** |\n| --- | --- | --- | --- | --- | --- |\n| 0% | 14.78 | 14.78 | 0.0635 | 503.67 | 1.99 |\n| 12.5% | 16.38 | 15.77 | 0.0572 | 559.14 | 1.79 |\n| 25.0% | 17.07 | 16.05 | 0.0504 | 634.83 | 1.58 |\n| 37.5% | 18.14 | 16.48 | 0.0505 | 633.06 | 1.58 |\n| 50.0% | 19.02 | 17.13 | 0.0412 | 777.09 | 1.29 |\n| 62.5% | 21.44 | 18.40 | 0.0463 | 690.89 | 1.45 |\n| 75.0% | 27.22 | 20.99 | 0.0390 | 820.40 | 1.22 |\n\nThe results clearly demonstrate that pruning the head dimension significantly accelerates model inference. Notably, CLOVER pruning 50% of the head dimensions outperforms vanilla pruning at 37.5% by generating 144 more tokens per second while maintaining a lower perplexity.\n\nFor the PEFT benchmarking experiment, please refer to **A2 for Reviewer DDfo**.\n\n**Q2: Conduct pruning on models other than GPT-2-XL and combine CLOVER with existing SOTA pruning techniques.**\n\n**A2:** Thank you for providing a list of some state-of-the-art pruning research. In Section 4.4, we also applied pruning to Whisper-large-v3 in addition to GPT-2-XL. Due to time constraints, we only compared CLOVER with SliceGPT during the rebuttal period. **In the camera-ready version, we will cite them all and compare CLOVER with those methods.** As mentioned in **A3 for Reviewer jpx5**, we expanded our experiments by using CLOVER pruning on Deepseek and OPT models. Early results indicate that CLOVER achieves a 5% pruning ratio improvement on these models, and we plan to refine these experiments further. Additionally, CLOVER supports pruning in models such as ViT and DiT, and we will continue to compare CLOVER with pruning methods in these domains.\n\n**Q3: L2-norm-based pruning typically utilizes the norm of either W_q or W_k individually, or their sum, rather than their product.**\n\n**A3:** You are correct that many L2-norm-based pruning methods use the norm of either W_q or W_k individually or their sum. However, there are cases where the product of both norms is used. For example:\n\n1.\tWhen pruning the output of one layer affects the input of the next, considering both layers together may yield better results, and in such cases, the product of the two layers is used [1].\n\n2.\tIn the attention layer, the attention score is influenced by both Q and K, which is why their product is considered in pruning [2].\n\n**Q4: Does the experiment in Section 4.4 maintain the parallelizability of multi-head attention? Specifically, does it remove the same number of dimensions for each head?**\n\n**A4:** Yes, all pruning in this paper maintains the parallelizability of multi-head attention. In Section 4.1, the pruning ratio is consistent across all layers, with each attention head having the same dimension. In Section 4.4, to achieve higher pruning rates while maintaining a training-free approach, we allow for different pruning ratios for each layer while ensuring that the number of dimensions removed is the same for each attention head within the same layer. This design ensures that inference can proceed in parallel.\n\n**Q5: Figure annotations and typo corrections.**\n\nThank you for pointing out these issues. In the camera-ready version, we will revise the figure captions for clarity and make necessary corrections to improve the overall expression.\n\n**Q5.1: In Figure 4, what values are being ranked in “Top,” “Next,” and “Bottom”?**\n\n**A5.1:** As explained in [Lines 375-377, left] and [Lines 362-364, right], after performing SVD on the model, According to the magnitude of the singular values, the singular vectors are denoted as: “Top-256,” “Next-256,” and “Bottom-256.” We found that only 10% of the feature components are projected onto the singular vectors corresponding to the Top-256 singular values.\n\n**Q5.2: In Figure 5, what do the horizontal and vertical axes represent?**\n\n**A5.2:** In Figure 5, the X-axis represents the index of the feature values of $\\Delta W$, while the Y-axis represents the magnitude of these feature values.\n\nWe hope this clarifies your questions and concerns. We greatly appreciate your effort and look forward to your further feedback on our response.\n\n**References:**\n\n[1] Fluctuation-based Adaptive Structured Pruning for Large Language Models, AAAI 2024.\n\n[2] Round and Round We Go! What makes Rotary Positional Encodings useful?, ICLR 2025.", "claims_and_evidence": "Yes. The claim in the paper about cross-layer decomposition is mathematically sound and has experimental validation.", "methods_and_evaluation_criteria": "Yes. The proposed method facilitates the aggregation of the principal components in the parameter space, exposing redundant components. Additionally, it enables full-rank parameter updates, demonstrating practical significance.", "theoretical_claims": "Yes. I checked SVD, intrusive dimensions problem, etc. and the paper discusses them correctly.", "experimental_designs_or_analyses": "Yes. The experimental design of the paper is generally well-founded. Specifically, the paper validates the effectiveness of CLOVER for pruning on GPT-2-XL, demonstrates its efficiency for PEFT across the LLAMA series models and multiple inference tasks. Furthermore, more fundamental analysis is provided by observing the parameter and feature space rank, projection distribution, spectral variations, etc.\nHowever, some settings in the experiment are insufficient or may be confusing, please refer to Other Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "The proposed CLOVER demonstrates applicability in both pruning and PEFT. In the context of pruning, existing structured pruning methods suffer from significant performance degradation. CLOVER effectively reduces redundant dimensions by concentrating the principal components in the parameter space. However, the paper claims to only introduce a novel perspective and does not benchmark its performance against SOTA approaches, which **cannot make me fully convinced**. Regarding PEFT, most existing methods operate within individual layers and incur substantial cost. In contrast, CLOVER employs cross-layer low-rank decomposition to achieve full-rank fine-tuning with reduced cost and mitigates the issue of intrusive dimensions.", "essential_references_not_discussed": "The paper provides a comprehensive enough discussion of related work.", "other_strengths_and_weaknesses": "This paper provides a new perspective on discovering redundancy and improving PEFT performance, which demonstrates application value. The writing is clear and fluent, confirming to the fundamental academic writing standards. However, there are still some weaknesses.\n1. The paper claims to address the issue of memory-bound during inference. However, the experiments section does not present any memory-related results to validate the effectiveness of the proposed method. Furthermore, the method is not compared with any novel pruning approaches (such as LLM-Pruner[1], SliceGPT[2], FLAP[3], UKMP[4]) in the targeted inference stage. While a study offering a novel perspective may be tolerated even if it does not achieve optimal performance, it should at least present comparative results in the experiments or demonstrate the effectiveness of combining this method with existing SOTA pruning techniques.\n2. The figures in the paper are somewhat difficult to interpret. For example, in Figure 4, what values are being ranked in \"Top\" \"Next\" and \"Bottom\"? In Figure 5, what do the horizontal and vertical axes represent?\n3. The quantitative validation of CLOVER on pruning is only conducted on GPT-2-XL. Since Figure 2 already demonstrates that the method effectively exposes redundancy across various models, reporting the pruning results on these models as well would better showcase the effectiveness of the proposed method.\n[1] Llm-pruner: On the structural pruning of large language models. NeurIPS 2023.\n[2] Slicegpt: Compress large language models by deleting rows and columns. ICLR 2024.\n[3] Fluctuation-Based Adaptive Structured Pruning for Large Language Models. AAAI 2024.\n[4] Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models. AAAI 2025.", "other_comments_or_suggestions": "There are some typos such as  in line 115 “without need introduce” and  in line 181 “We presents”.", "questions_for_authors": "1. In Figure 2, why is the product of the norms of $W_Q$ and $W_K$ used as the vertical axis? I understand that this is consistent with the CLOVER method, but is this metric also used as an importance criterion in the vanilla pruning method? Based on my understanding of the pruning field, l2-norm-based pruning typically utilizes the norm of either $W_Q$ or $W_K$ individually or their sum, rather than their product. Therefore, I have concerns regarding the correctness of the vanilla pruning method used for comparison.\n2. Does the experiment in Section 4.4 maintain the parallelizability of multi-head attention? Specifically, is does the $W_{QK}$ remove the same number of dimensions for each head?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an approach that improves memory efficiency in LLMs by performing an SVD on some matrices in the attention layer, resulting in orthogonal vectors within the attention heads that reduces linear redundancy.  The authors show the resulting model has benefits for token pruning and parameter-efficient finetuning.  The paper was reviewed by four experts, where a majority recommend acceptance.  The lone dissenter noted their primary concern was that the writing changes they recommended are extensive enough to warrant another round of reviews.  The other reviewers echoed the concern with the clarify of the paper in its current form, but have provided a detailed plan on how to address reviewer concerns.  While the ACs finds the writing concern justified, they do not find it warrants overturning the majority of reviewers as the authors plan for addressing these was accepted by reviewers.  The authors are strongly encouraged to consider the reviewer recommendations when revising their paper.
Decision: Accept (poster)