Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression
Authors: Jingcun Wang
Yu-Guang Chen
Ing-Chao Lin
Bing Li
Grace Li Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Cross Layer Parameter Sharing in LLMs
LLM Compression with SVD
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We compress LLMs by cross-layer parameter sharing with SVD.

Abstract: Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which prevents their practical deployment in many applications. To reduce memory storage of LLMs, singular value decomposition (SVD) provides a promising solution to approximate weight matrices for compressing LLMs. In this paper, we take a step further to explore parameter sharing across different layers with SVD to achieve more effective compression for LLMs. Specifically, weight matrices in different layers are decomposed and represented with a linear combination of a set of shared basis vectors and unique coefficients. The types of weight matrices and the layer selection for basis sharing are examined when compressing LLMs to maintain the performance. Comprehensive experiments demonstrate that Basis-Sharing outperforms state-of-the-art SVD-based compression approaches, especially at large compression ratios.

Published as a conference paper at ICLR 2025
BASIS SHARING: CROSS-LAYER PARAMETER SHARING FOR LARGE LANGUAGE MODEL COMPRESSION
Jingeun Wang
Yu-Guang Chen
Technical University of Darmstadt jingcun. wangêtu-des rmstadt..dtdd
National Central University andyygchen ncu edu. tw
Ing-Chao Lin
Bing Li
National Cheng Kung University iclinecsie ncku. ed
University of Siegen
Bing. Li@uni siegen.dde de
Grace Li Zhang
Technical University of Darmstadt grace. www.saadt..aaeaeaoooo de
ABSTRACT
Large Language Models (LLMs) have achieved remarkable breakthroughs. How- ever, the huge number parameters in L.MS require significant amount memory storage in inference, which prevents their practical deployment in many applica- tions. To reduce memory storage of LLMss, singular value decomposition (SVD) provides a promising solution to approximate weight matrices for compressing LLMs. this paper, we take a step further to explore parameter sharing across different layers with SVD achieve more effective compression for LLMs. Specif- ically, weight matrices in different layers are decomposed and represented as a linear combination of a of shared basis vectors and unique coefficients. The types of weight matrices and the layer selection for basis sharing are examined when compressing LLMS maintain the performance. Comprehensive exper- iments demonstrate that Basis Sharing outperforms state-of-the-ar SVD-based compression approaches and parameter sharing techniques, especially under large compression ratios.
INTRODUCTION
Large Language Models (LLMs) have revolutionized natural language processing by enabling machines understand human language capabilities, they are computation- and memory-intensive, making their deployment on resource- constrained devices challenging. To address this challenge, model compression has become widely adopted technique to reduce model size and complexity.
accurately. Although these models have remarkable
Common compression techniques, such model distillation (Gu al., 2024; Magister al., 2023; Jiang et al., 2023b; Huang et al., 2022; Qiu et al., 2024), pruning (Frantar & Alistarh, 2023; 2022; Ma al., 2023; Sun al., 2024; Jiang et al., 2024; Petri al., 2023), and quantization (Lin al., 2024; Zhao et al., 2024; Ashkboos et al.. 2024; Xiao et al., 2023; Sun et al., 2023), early-exit (Chen et al., 2024; Wang et al., 2024a), etc. have been extensively studied. While such techniques are effective in many scenarios, these methods often require hardware modification and expensive retraining. Compression techniques based on low-rank approximation with, e.g., Singular Value Decomposition (SVD) (Yuan et al., 2023; Hsu et al., 2022; Wang al., 2024b), provide promising alternative since they are not restricted by such constraints. In SVD-based weight compression, a weight matrix in a layer is processed individually by decomposing it into three matrices. By removing small singular values in the decomposed diagonal matrix, the original weight matrix can be approximated with fewer number of weight values.

---

Published as a conference paper at ICLR 2025
Despite the benefits of SVD-based weight compression, the potential of grouping layers for weight approximation and compression has not been explored thoroughly. Since weight matrices in different layers of an LLM might share similarity, parameter sharing across layers can be exploited to further compress weight matrices for L.Mss. In sharing parameters across layers, Hay & Wolf (2024) trained small language model by restricting weight matrices in some layers to be the same. On the one hand, this brute-force method leads to significant performance degradation since weight matrices in different layers should vary to maintain their functionalities. On the other hand, it impractical to train LLMs from scratch due to limited training data or high training costs.
Contrary to previous work, in this paper, we use pretrained LLMs to enable weight matrices across layers to share a common set of basis vectors but still retain their different functionalities with unique coefficients. Our method, called Basis Sharing, can compress L.LMs effectively. In summary, our contributions are as follows:
1. We propose to represent weight matrices across different layers in a pretrained L.M with a linear combination of a set of shared basis vectors and coefficients unique to specific layers. This basis sharing can effectively reduce the number of parameters in LLMs while only affecting the performance of LLMs slightly.
2. We examine cross-layer basis sharing for different types of weight matrices in LLMs according to the incurred compression errors. The types of weight matrices whose sharing across layers does not incur significant compression error are selected for compressing LLMs.
3. For the selected types of weight matrices, we also develop criterion to group layers to share a set of basis vectors but have individual coefficients to preserve the performance of LLMs.
4. We conduct extensive experiments on a variety of LLMs, including the LLaMA family (Touvron et al., 2023a:b), OPT-6.7B (Zhang et al., 2022), Mistral-7B (Jiang et al., 2023a), and GPT-2 (Radford et al.. 2019). Our Basis Sharing can surpasses the state-of-the-art SVD-based methods both generation tasks and downstream reasoning tasks without any fine-tuning under compression ratios from 20% to 50%. Specifically, compared with state-of- the-art SVD-based compression approaches, Basis Sharing can further reduce the perplexity by up to 25% generation tasks and improve accuracy by up to 4% on downstream reasoning tasks under the same compression ratio.
2 RELATED WORK
Large Language Model Compression LLM compression techniques include model distillation, pruning and quantization, etc. Gu et al. (2024); Huang et al. (2022); Magister et al. (2023); Jiang et al. (2023b) successfully applied model distillation to LLM by retraining, which incurs high computational cost. Frantar & Alistarh (2023; 2022); Sun et al. (2024); Ma et al. (2023) pruned weights that are less sensitive to outliers. However, the resulting unstructured weight matrices do not provide meaningful compression benefits real hardware. Structured pruning techniques, such as 2:4 or 4:8 pruning, can achieve effective compression but restrict a fixed 50% pruning ratio, which limits flexibility in balaneing performance and compression ratio. Zhao al. (2024); Ashkboos et al. (2024); Lin et al. (2024); Xiao et al. (2023) allocated higher quantization bits to weights with larger influence on outliers, but it does reduce the number of parameters, limiting its impact on overall compression.
SVD-based Weight Compression SVD-based weight compression has flexible compression ratio to maintain performance without retraining. Golub et al. (1987) were the first to apply SVD for neural network compression, and Lv al. (2023); Wu et al. (2023) extended this approach to shallow transformer models (Vaswani, 2017). However, in LLM compression, these methods incur significant errors since they do not consider outliers in activations. FWSVD (Hsu et al., 2022) addresses this issue by incorporating the impact of outliers through the Fisher information analysis of weight matrices. However, this method requires gradient information during training process, which is computationally prohibitive for LLMs. ASVD (Yuan et al., 2023) alleviates this problem by selecting key channels in the weight matrix based on their sensitivity to outliers and minimizing compression error in these channels. While it avoids the need for gradients, ASVD still lacks direct connection
2

---

Published as a conference paper at ICLR 2025
between SVD truncation enror and the overall model compression emror. SVD-LLM (Wang et al., 2024b) improves this by introducing a whitening matrix that captures outlier information, effectively reducing compression error. However, all of these methods focus only on compressing individual weight matrices within single layer, missing the opportunity to exploit weight compression across multiple layers.
Parameter Sharing Parameter sharing reduces model size by reusing weight matrices across different layers. Inspired by recurrent neural networks, Dehghani et al. (2019) explored this concept within transformers by restricting all layers in the encoder and decoder share the same weights. Similarly, Reid et al. (2021) divided transformer parameters into two groups (attention-related and beedforward-reltter and compressed the model by sharing weights within each group. Takase & Kiyono (2021) applied selective weight sharing. where specific layers shared the same weights rather than all layers. Beyond direct weight sharing, Xiao al. (2019); Bhojanapalli et al. (2021) introduced the idea of sharing attention scores between layers. By reusing attention scores, some weight matrices for attention computation could be discarded. Dynamic Tying (Hay & Wolf, 2024) determines layer-wer wise weight sharing during training using reinforcement learning, which is still time- mons-onnnnnng for large LLMs. All of these approaches have been tested only on smaller transformer models and rypically require training from scratch or full parameter fine-tuning, which makes them impractical for LLMs.
METHODOLOGY
Shared Basis
Sume Param.
Contrary the previous techniques that require training from scratch and weights in some layers
B
X
w
W
are restricted to be the same during training, we
C₁
C
adopt pretrained LLM to explore representing weights across different layers with combinations of set of shared basis vectors and individual co- efficients. Since the set of basis vectors can be shared across several layers, the number of param- eters in the LLM can thus be reduced effectively. The difference between the previous weight shar- ing method and our Basis Sharing is illustrated Figure 1.
L
L
Lee
Unique Coef.
(a)
(b)
Figure 1: (a) Two layers share the same weight matrix in previous work. (b) Two layers share the same basis matrix but have their individual coefficients in our work.
To exploit the cross-layer parameter sharing to compress LLMs, the subsequent subsections address the following challenges: 1) What methodologies can be used to process the weight matrices across layers in an LLM to determine a set of shared basis vectors and individual coefficients? 2) Which types of weight matrices across layers in LLM can take advantage of parameter sharing without affecting its performance significantly? 3) Which layers can share set of basis vectors in an LLM without affecting its performance significantly?
3.1 REPRESENTING WEIGHT MATRICES ACROSS LAYERS WITH COMBINATIONS OF BASIS VECTORS AND CORFFICIENTS
Suppose that we have weight matrices across layers, denoted as (11), ("),, W(( xdd To derive a set of shared basis vectors and coefficients to represent such weight matrices, intuitively, such matrices can be horizontally concatenated into one matrix, denoted as W E Rd agdixnd2 and singular value decomposition (SVD) can be applied to decompose this matrix into three matrices: U. Σ, VT. Σ is a d1 X ndg diagonal matrix consisting of singular values of W.
By selecting the top singular values in Σ, W can be approximated as W where the dimensions of Uk. Σg and V are di x ki, k k. and k x nd2, respectively. The value of k should be determined to balance the compression ratio and the performance of the compressed LLM (Appendix A.2 shows the evaluation of l under a given compression ratio). WA be rewritten as W: = BVTT where B is the multiplication result of Un and Σg. We call B a basis matrix and column of B is basis vector, denoted as B : V can be considered as coefficient matrix, i.e.,
U V.T
VT C. Accordingly, the jth column of the original weight matrix W(( in the 7th layer can be
3

---

Published as a conference paper at ICLR 2025
AW1
X
W 2id3
1/2
k
nid
1.0
0.5
0.01
|| 111
E
12
0.2
0.8
0.8
10
||AW|
10.8
1.1
0.7
0.8
0.0
B
d2
d2
ndg
d5
AW₂
X
W/i
k basis vectors
(C0
C(s)
1.0
0.01
1.0
0.01
||WW, | = 9.8
0.4
0.02
2.0
10
||AWW, 0.8
W
Ollx
5.0
0.03
8.0
Figure 2: Weight matrices across layers are concatenated horizontally into a weight matrix, which is processed by SVD. The jth column of the original weight matrix in a layer can be represented as a linear combination of k: shared basis vectors and coefficients.
Figure 3: AW, and AW2 are dif- ferences with respect to the orig- inal weight matrix after compres- sion. ||Will is smaller than | F. but |XAWi]l is larger than AAWll....
approximated liner combination of & basis vectors and individual coefficients as follows.
Bim Cin.. m.d"
(1)
mm
where COO is the coefficient matrix in 5th layer. The process of weight matrix approximation and representation is illustrated in Figure 2.
In the weight matrix approximation with SVD above, input data, denoted as X, are not considered. In fact, the result of XW instead of W is used in inference. Accordingly, applying SVD directly onto weight matrices without incorporating input data might lead to significant computation loss and potentially affect the performance of the LLM. Figure 3 illustrates an example, where a weight matrix approximated with SVD leads to a large compression loss in the form of Frobenius loss, denoted as || KAW]rr. Since the second element in the input data affects the computation accuracy significantly. the second column of the weight matrix should be approximated more accurately compared with other columns reduce the overall computation loss. Yuan al. (2023); Wang et al. (2024b) also pointed out similar results.
To incorporate the effect of input data into the weight approximation with SVD maintain the performance of the LLM, we will scale the concatenated weight matrix W with matrix S E Rd xd: as follows
(2) The matrix S should be evaluated to represent the impact of input data on the weights, so that can adjust W accordingly to reflect the significance of different input data. To obtain appropriate S, we will adapt the techniques developed in Wang et al. (2024b), where S can be evaluated with S(S)T = (booeskk((()) X). However, X in their technique refers to input data of a layer instead of several layers our method. To evaluate S considering several layers, we will vertically concatenate the input matrices in such layers, denoted as, the concatenated X. In our experiments, we use 256 samples from WikiText-2 (Merity et al., 2016) with each 2048 tokens evaluate X, similar that in Wang et al. (2024b).
W S----S (SW).
(*), and compute the S with
Instead of applying SVD directly on the concatenated weight matrix W, we will decompose SW with SVD and approximate this scaled weight matrix SW RJ U V B'C', where B' and C' are the revised basis matrix and coefficient matrix, respectively. To recover the approximated weight matrix for computation in inference, S will be multiplied with B', the result of which will be the final adjusted basis matrix, i.e.,
w S U S'C'
(3)
where B" is the final adjusted basis matrix in our paper.
4

---

Published as a conference paper at ICLR 2025
3.2 SELECTION OF WEIGHT MATRICES IN LLMS FOR CROSS-LAYER PARAMETER SHARING
Modern L.MS constructed based on the decoder-only transformer architecture. A layer in such an architecture includes several types of weight matrices, which have different functions. WK. Wo and Wv are three types of projection matrices, which are used to generate the key, the query and the value matrices. Wo. another type of weight matrices, further transforms the attention result to build a new representation for an input embedding. Wúp and Wate (used in LLaMA and LLaMA2), further types of weight matrices, represent this transformation result into a high-ding diheemieeninn embedding. Afterwards, W Down, the last type of weight matrices, projects the high dimension embedding back to the low dimension embedding. The types of weight matrices above have different functions, so that we need to determine which type of weight matrices can take advantage of cross-layer basis sharing with SVD described in Section 3.1 without affecting the performance of the LLM significantly.
First of all, the type of matrices whose function are to project a high-dimension embedding into a low-dimension embedding such as WDwn. cannot take advantage of the cross-layer parameter sharing. The reason is that after the horizontal concatenation of such matrices, the rank of the concatenated matrix will be larger than that of individual matrix. Under the same compression ratio, compressing the concatenated matrix with SVD incurs a larger Frobenius loss than the original weight matrix.
For the remaining types of weight matrices including WK. Wo. Wv. Wo. WUp and WGater we will determine whether each of them can cross-layer basis sharing by examining the Frobenius loss resulted from this sharing. To explain this concept, we use basis sharing across two layers for WK in LLaMA2-7B as an example. Assume that we remove small singular values by applying
SVD on S ( [L, 32) and is the corresponding S matrix for w layer under this compression ratio will be evaluated. To evaluate the Frobenius loss incurred by basis sharing, we horizontally concatenate w of the 7th layer and W of the j" as W (i-5) where j * i, 1.j E [1. SVD is applied on ((1-5) w-(t-)) to remove small singular values to achieve the same compression ratio, where S(X,3) is the corresponding S matrix for (( Afterwards, we evaluate the incurred Frobenius loss of basis sharing across two layers. Similarly, we repeat the process above for Wo. The results are illustrated in Figure 4, where the number/color in a block represents the resulting Frobenius loss if a basis matrix is shared between two layers and the numbers in the diagonal direction are obtained by applying SVD to the scaled weight matrix of a layer directly.
k to achieve a compression ratio of 20%, where W is Wx matrix in the zth layer (())
K
K The resulting Frobenius loss of each
Figure 4 compares the results of basis sharing for WK and Wo. Basis sharing across two layers for WK can reduce the Frobenius loss. For example, when SVD is applied on Sx Wx for the 9th and 10th layers separately, the resulting Frobenius loss is evaluated 33508.2 2 33174.7 66682.9. When the gth and 10th layers share a common basis matrix, the Frobenius loss resulting from compression becomes smaller, i.e, 61817.3 < 66682.9. This indicates that allowing parameter sharing across two layers for WK can enhance computation accuracy. This trend can be seen in WK. Wq. Wy, WUp and WGte (Appendix A.8 show the results). Accordingly, basis sharing across layers can be applied on such matrices.
On the contrary. basis sharing for Wo in 9th and 10th layers incurs the increase of the Frobenius loss, i.e., 10618.3 > 4355.1 + 4895.7. 7. Accordingly, this parameter sharing should not be applied on Wo to avoid significant computation loss. For such matrices, we will apply SVD to process the individual matrix in each layer separately.
3.3 SELECTION OF LAYERS FOR BASIS SHARING
Section 3.1 determines which types of weight matrices can shared across layers. This subsection then determines which layers can share basis vectors to represent such types of weight matrices. To select layers for basis sharing, the basis sharing of such layers should not incur Frobenius loss larger than without sharing. According to Figure 4, the group of two adjacent layers leads to smaller Frobenius loss than the sum of the Frobenius loss of two separate layers. Based on this analysis, we will group adjacent layers with the order from the first layer to the last layer. Take group of two layers as an example. The first layer and the second layer are grouped for basis sharing, followed by the group of the third layer and the fourth layer, etc.
5

---

Published as a conference paper at ICLR 2025
Frobenius
grouping 9th
grouping the 9th
and 10th layers
and 10th layers-
61837.3
323/4.77
12332.3
691574
10618.3
4395.7
11425,4
12352
32595.3
GS499
63322
360793
67143.1
73705
10600.7
11425
5050.5
12402
32095.3
71528
93307
67143.
20468
347478
L1887
12392.9
12492
6144.
1992322
423623
117525
32390.3
139232
layer
32
layer
32
20000
15000
1006
10000
5000
-2000
32
32
(a)
(b)
Figure 4: Frobenius loss incurred by basis sharing across any two layers. The number/color in a block represents the resulting Frobenius loss if basis matrix is shared by two layers and the numbers in the diagonal direction are obtained by applying SVD to the scaled weight matrix of a layer directly. (a) Frobenius loss incurred by basis sharing across two layers for Wx in LLaMA2-7B. (b) Frobenius loss incurred by basis sharing across two layers for Wo LLaMA2-7B.
4 EXPERIMENTS
4.1 SETTINGS
Baseline We compare with the work where SVD>-based weight approximation in each individual layer applied without eross-layer parameter sharing. Such work includes ASVD (Yuan et al., 2023), FWSVD (Hsu et al., 2022) and SVD-LLM (Wang et al., 2024b). We also compared our method with Dynamic Tying (Hay & Wolf, 2024), where weights in some layers are restricted to be the same by training from scratch. Since this method can only be applied small language models, only GPT2 (Radford et al., 2019) was used to compared our method and Dynamic Tying.
Models and Datasets. We evaluate our method using several models. For LLMs, many models are evaluated, namely LLaMA family (LLaMA-7B, LLaMA-13B. LLaMA-30B, LLaMA2-7B) (Touvron et al., 2023a;b), OPT-6.7B (Zhang et al.. 2022), Mistral-7B (Jiang et al., 2023a), GPT2. Three language modeling datasets used our experiment include WikiText-2 (Merity et al., 2016), PTB (Mareus et al., 1993) and C4 (Raffel et al., 2019). Seven reasoning datasets used in the experiments include OpenbookQA (Banerjee et al., 2020), WinoGrande (Sakaguchi et al., 2021) HellaSwag (Zellers al., 2019), PIQA (Bisk et al., 2020), MathQA (Amini et al., 2019), ARC-e, ARC-c (Clark et al., 2018). All the reasoning tasks are tested zero-shot setting with the implementation of LM-Evaluation--Hare framework (Gao al., 2024).
Implementation details All of our models are based on the model implemented by the Hugging Face. LLaMA-30B are implemented with FP16, the rest models are implemented with FP32. To evaluate S, FP64 is used to maintain the computation precision. All experiments are tested on two NVIDIA A100 80GB GPUs. S is derived through 256 samples from WikiText-2 with 2048 sequence length. When the compression ratio is 40% or larger than 40% the incurred compression errors increase, that the output of layer the input of the next layer deviates significantly from its

---

Published as a conference paper at ICLR 2025
Table 1: PPL()) and Zero-shot(t) performance of LLaMA-7B with Basis Sharing and baselines under 20% to 50% compression ratio on three language modeling datasets and seven common sense reasoning datasets. The S of all tasks is obtained with the dataset WikiText-2.
RATIO
МЕTHоD
WikiText-2.
PTB.
C44
Openb.
ARC_e
WinoG.
HellaS.
ARC_e
PIQA
MailQA
Average
0%%
Original
5.68
8.35
7.34
0.28
0.67
0.67
0.56
0.38
0.78
0.27
0.52
0.14
FWSVD
1727
2152
1511
0.15
0.31
0.50
0.26
0.23
0.56
0.21
0.32
20%
ASVD
11.14
16.55
15.93
0.25
0.53
0.64
0.41
0.27
0.68
0.24
044
SVD-LLM
7.94
18.05
15.95
0.22
0.58
0.63
0.43
0.29
0.69
0.24
0.44
Besis Shering
7.74
17.35
15,03
0.28
0.66
0.66
0.46
0.36
0.71
0.25
0.48
SVD
13103
17210
20671
0.13
0.26
0.51
0.26
0.21
0.54
0.22
0.30
FWSVD
20127
11058
7240
0.17
0.26
0.69
0.26
0.22
0.51
0.19
0.30
ASVD
0.37
SVD-LLM
9.56
29.44
25:11
0.20
0.48
0.59
0.40
0.26
0.65
0.22
0.40
Basla Sharing
9.25
29.12
22,46
0.27
0.63
0.63
0.40
0.30
0.68
0.24
0.45
SVD
52489
59977
47774
0.15
0.26
0.52
0.26
0.22
0.53
0.20
0.30
PWSVD
18156
20990)
12847
0.16
0:51
0.26
0.22
0.53
0.21
40%
ASVI
1407
3292
1109
0.13
0.28
0.48
0.26
0.22
0.55
0.19
0.30
SVD-LLLM
13.11
63.75
49.83
0.19
0.42
0.58
0.33
0.25
0.60
0.21
0.57
Besis Sharing
12.39
SVD
131715
87227
79815
0.16
0.26
0.50
0.26
0.23
0.52
0.19
0.30
FWSVD
24391
28321
23104
0.12
0.26
0.50
0.26
0.23
0.53
0.20
0.30
50%
ASVD
15358
47690
27925
0.12
0.26
0.51
0.26
0.22
0.52
0.19
SVD-LLM
0.16
0.29
0.56
Basta Sharing
19.99
126.35
88.44
0.18
0.42
0.57
0.31
0.23
0.58
0.22
0.36
Table 2: PPL(.) and Zero-shot(") performance of LLaMA2-7B with Basis Sharing under 20% to 50% compression ratios on three language modeling datasets and seven common sense reasoning datasets. The S of all language modeling tasks is evaluated with WikiText-2. For reasoning tasks, the S of the results outside the bracket is evaluated with 2ikiText-2, while inside is evaluated with Alpaca.
RATIO
2i
PTB]
C4.
Openh.
ARC_e
WinoG.
HellaS.
ARC_c
PIQA
MathQA
Averugel
5.47
7.29
0.31
0.69
0.55
20%
7.77
60.00
15.30
0.27 (0.28)
0.56 (0.20)
0.63 (0.63)
(.43(0046)
0.33 (0.35)
(0.74)
0.25 (0.25)
0.47 (0.49)
30%
9.69
97.40
23.86
0,26 (0.27)
0.58 (0.65)
0.62 (0.62)
0.38 (0.41)
0.27 (0.32)
0.56 (0.70)
0.23 (0.24)
0.43 (0.46)
40%
13.62
195.95
43.89
0.19 (0.21)
0.48 (0.57)
0.58 (0.57)
(0.36)
0.22 (0.27)
0.61 (0.66)
0.23 (.23)
0.38 (0:11)
21.3
(0.17)
(0.47
(0.5)
0.20 (0.25)
(0.60)
0.23
(0.36)
original values. This input deviation affects the evaluations of S with S(S)T = cholesky((X)) x). To incorporate this input deviation, we update the weights in the next layers for basis sharing with such deviated inputs, similar to that in SVD-LLM.
4.2 RESULTS
We evaluate the performance of the proposed cross-layer parameter sharing from four aspects: (a) Performance on generation and reasoning tasks and comparison with state of the art in zero-shot setting. (b) LLM Performance different LLMs in zero-shot setting. (c) Performance on LLMs with various scales in zero-shot setting. (d) LLM performance with LoRA (Hu et al., 2021) fine-tuninng. (c) Comparison with training from scratch for weight sharing across layers.
Performance on Generation & Reasoning Tasks We demonstrate the performance of LLaMA-7B and LLaMA2-7B on ten datasets under different compression ratios from 20% to 50%. In evaluating the LLM performance, we group two consecutive layers in the order from the first layer to the last layer to share basis matrix, while Basis Sharing with more than two layers will be discussed later. Table 10 shows the results of LLaMA-7B. The first three datasets are for text generation tasks and the rest seven datasets are for reasoning tasks. For text generation tasks evaluated by perplexity (PPL), Basis Sharing consistently achieves the lowest PPL among compared with the state-of-the-ar methods across all compression ratios and tasks. In reasoning tasks, Basis Sharing achieves an average accuracy at least 3% higher than the state-of-the-re art methods. As the compression ratio increases, model performance consistently declines across all the methods due to the incurred larger compression errors. In short, Basis Sharing outperforms SVD-LLM due to smaller compression emrrr as discussed in Section 3.
7

---

Published as a conference paper at ICLR 2025
Table PPL (L) of three different LLMs OPT- 6.7B, LLaMA 2-7B, and Mistral-7B under 20% compression ratio on WikiText-2.
Table PPL (4) of LLaMA-7B, 13B, 30B under
20% compression ratio on WikiText-2. OOM
means out of memory error during the model compression.
METHOD
OPT-6.7B
LLaMA 2-7B
Mistral-BB
METHOD
LLaMA-7B
LLaMA-138
LLaMA-30B
66275
18192
159627
SVD
20061
946.31
54.11
FWSVD
14559
2360
6357
FWSVD
ASVD
10.10
13.72
ASVD
11.14
6.74
22.71
SVD-LLM
16.04
8.5
10.21
SVD-LLM
7.94
Sogig Sharing ng
11.79
7.70
7.57
Basis Sharing
7.74
6.47
5.47
Table 2 presents the basis sharing results of LLaMA2-7B. For the common reasoning tasks, S are evaluated with both WikiText-2 and Alpaca (Taori et al., 2023) demonstrate the performance difference. The result outside the bracket is based on the evaluation of S with WikiTexx-2, while the result within the bracket based on the evaluation of S from Alpaca. can be seen that LLaMA2-7B is more sensitive to parameter compression, especially on the PTB task. When the compression ratio reaches 50%, the PPL of LLaMA2-7B four times of the PPL of LLaMA-7B, while the performance on the remaining tasks are still comparable.
According to Table 2, the input dataset from which derived plays crucial role in determining performance on common reasoning tasks in zero-shot settings. Generally, the model where S is evaluated with Alpaca achieves better aceuracy than the model where evaluated with WikiText- 2, especially on ARC_e under 50% compression ratio. The accuracy difference can reach 1 11%. However, on WinoG, the difference is not obvious, the model where evaluated with WikiText-2 achieves even higher accuracy under 40% and 50% compression ratios.
To evaluate the generalization of Basis Sharing across multiple LLMs, evaluate its PPL on three distinct models from three LLM families: OPT-6.7B (from the OPT family), LLaMA 2-7B (from the I.LMMA family), and Mistral-7B (from the Mistral family). This comparison is conducted under 20% compression ratio using the WikiText-2 dataset without any fine-tuning. It can be seen from Table 3, Basis Sharing consistently achieves the lowest PPL. Especially for OPT-6.7B and Mistral-7B, Basis Sharing achieves PPL reduction up 25% compared with SVD-LLM.
Performance on Different LLMs
Performance on LLMs with Various Scales Basis Sharing can be applied to LLMS with large scales. To demonstrate this, we apply Basis Sharing on LLaMA with three different scales under 20% compression ratio, namely LLaMA-7B, LLaMA-13B and LLaMA-30B against the state-of-the-art methods. The result is shown in Table 4. According to this table, Basis Sharing achieves the best performance across all the scales. Since gradient needs to be computed with FWSVD, out of memory еrror occurs A100 GPU. In contrast, Basis Sharing can still be realized with A100 GPU.
Performance with LoRA Fine-Tuning LoRA
22
LLM
(Hu al., 2021) is one of the most promis- ing fine-tuning techniques to recover perfor- mance/accuracy. LoRA can also be applied Basis Sharing to recover performance/accuracy We used lora_r= 8, lora_alphh = 32, and leamn- ing rate lc-4, and used defaults for all other hyperparameters the Hugging Face PEFT. Each model is fine tuned with WikiText-2 training dataset for two epochs.
20
Basic-Share Sharing
18
16
10
Figure 5 shows the result after applying LoRA LLaMA-7B with WikiText-2. It can be seen from the figure that all compression methods achieve similar PPI. under 20% compression ratio, and PPL difference increases as the compression ratio goes Basis Sharing achieves the lowest PPL when the compression ratio reaches 50%.
20%
30%
40%
50%
Compression Ratio
Figure LoRA fine-tuning results of LLaMA- 7B under 20% compression ratio with different compression methods.
8

---

Published as a conference paper at ICLR 2025
Table 6: Impact of grouping different numbers of layers on LLaMA-7B under compression ratios from 20% 50%.
Table 7: Impact of grouping different numbers of layers on LLaMA-7B under compression ratios from 20% to 50% after LoRA Fine-Tuning
# LAYERS
20%
30%
40%
50%
LAYERS
20%
30%
40%
50%
7.94
9.56
13.11
23.97
1
7.78
9.56
10.65
13.26
7.74
9.25
12.39
19.99
7.14
7.84
8.91
10.56
7.72
9.27
12.60
20.06
3
7.00
7.81
9.04
10.35
7.65
9.18
12.58
20.86
4
7.07
7.86
9.02
10.36
7.62
9.19
12.81
24.45
5
6.98
8.05
9.23
10.14
6
7.64
9.20
14.13
25.40
6
9.06
10.32
7
7.67
9.24
14.64
27.30
7
6.75
7.57
9.08
10.76
7.75
9.49
14.60
27.92
8
6.89
7.68
9.14
10.32
7.95
10.58
19.72
49.11
16
7.02
7.82
9.27
11.20
7.94
9.56
30.82
85.24
32
8.25
9.37
11.64
Table 5: GPT2 20% compression ratio compared with Dynamic Tying.
Comparison with Training from Scratch Ta- ble 5 compares Basis Sharing with Dynamic Ty-
ing(Hay & Wolf, 2024), where parameter sharing is realized by training from scratch. Instead of training from scratch, Basis Sharing leverages pre- trained models that have been trained on large
METHOD
Parm.
Time
PPI.
Dynamic Tying
264M (GPT2-XL)
13.75h
49.37
Basis Sharing
94M (GPT2)
26.47s
43.15
datasets and trained with more computational resources. As a result, Basis Sharing achieves fewer parameters, faster compression, and better PPL WikiTeext22 compared to Dynamic Tying.
4.3 IMPACT OF LAYER SELECTION OF BASIS SHARING ON LLM PERFORMANCE
In section 3, we analyzed the change of Frobenius loss when two layers are grouped to share a set of basis vectors. In this section, we will demonstrate how grouping more than two consecutive layers affects the LLM performance.
Impact on LLM Performance in Zero-Shot Setting We grouped different numbers of consecutive layers to examine the impact of the number grouped layers the LLM performance without any fine-tuning. Table 6 shows the result. The number in the first column indicates the number of consecutive layers sharing common basis matrix. For example, 4 means that every four consccutive layers share a basis matrix in the order from the first layer to the last layer. Compared with no basis sharing in SVD-LLM (# LAYERS 1) under 20% compression ratio, Basis Sharing achieves similar performance. Grouping four or five layers to share a basis matrix is more reasonable when compression ratio is lower than 30%, since they have the lowest PPL. Two layers sharing basis matrix is good choice when the compression ratio is larger than 30%.
Impact LLM Performance with LoRA Fine-Tuning We also examined the impact of grouping different number of layers on LLM performance after LoRA Fine-Tuning. Table 7 shows the result. According to this table, the performance of LLM can be enhanced compared with that without fine- tuning. In addition, this table also shows that after LoRA fine-tuning, grouping layers in LLaMA-7B for Basis Sharing can achieve better performance than that without basis sharing in SVD-LLM (# LAYERS I). Even when the number of grouped layer is 32, the performance of Basis Sharing is still better than that without basis sharing SVD-LLM (# LAYERS 1).
To examine the full potential of the Basis Sharing, we also conducted the full parameter fine-tuning to examine the impact of grouping different numbers of layers on LLM performance. Due to the high computational cost, only fine tuned the LLaMA-7B on grouping 2, 4, 8, 16, 32 layers, respectively. The differences from LoRA fine-tuning are that we use here learning rate 2 and two A100 GPUs. The results of full parameter fine-tuning can be found Table can be seen that the performance with full parameter fine-tuning is only a little bit better than the performance with LoRA fine-tuning. The reason could be
Impact on LLM Peformance with Full Parameter

---

Published as a conference paper at ICLR 2025
that WikiText-2 is relatively small dataset to fine-tune the large model. Directly using this dataset to fine-tune could easily lead to overfitting. Therefore, we reduce the learning rate from le-4 to 2e-6.
4.4 PERFORMANCE ON REAL HARDWARE
Table 8: Result of full parameter fine- tuning by grouping different numbers of layers.
Basis Sharing not only reduces the memory required for storing parameters, but also enhances inference efficiency on real hardware. To demonstrate this advantage, we com- pared the performance LLaMA-7B with and without Basis Sharing single A100 GPU, using batch size of 512 and a sequence length of 32 to generate one token for each batch. With this setting, throughput was evaluated as the total number of tokens that can processed by the model per second.
LAYERS
20%
30%
40%
50%
2
6.57
7.41
8.29
9.71
4
6,64
7.39
8.41
9,91
8
6.63
7,46
8.54
10.23
16
6.66
7.66
9.04
10.48
32
6.67
7.90
9.24
10.94
CONCLUSION
Figure 6 shows the throughput result. It be seen that as the compression ratio increases, the throughput of model with Basis Sharing also increases. Under 50% compression ratio, the throughput of Basis Sharing 1.57 times of the dense model. In this paper, we explore parame- ter sharing across different layers with SVD to achieve effective compression for LLMs. Specif- ically, weight matrices in different layers are de- composed and represented as linear combination of a set shared basis vectors and unique coef- ficients. The types of weight matrices and the layer selection for Basis Sharing are examined when compressing LLMs to maintain the perfor- mance. Comprehensive experiments demonstrate that Basis Sharing outperforms state-of-the -hettrrt SVD-based compression approaches, especially under large compression ratios.
Models
dense
2200
Basis Sharing
2000
1800
Toooms
1200
1000
800
20%
30%
40% Compression Raiio
50%
Figure 6: Throughput of dense LLaMA-7B model and the compressed model with Basis Sharing under compression ratios from 20% to 50%.
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel- Kedziokkiiii Yejin Choi, and Hannanch Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367. Association for Computational Linguistics, 2019.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian I. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404. 00456, 2024.
Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. Careful selection of knowl- edge to solve open book question answering. In 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp- 6120-6129, 2020.
Srinadh Bhojanapalli. Ayan Chakrabarti. Andreas Veit, Michal Lukasik, Himanshu Jain, Frederick Liu, Wen Chang, and Sanjiv Kumar. Leveraging redundancy in attention with reuse transformers. arXiv preprint arXiv:21 10.06821, 2021.
Yonatan Bisk, Rowan Zellers, et al. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference Artificial Intelligence, 34:7432-7439 2020.
10

---

Published as a conference paper at ICLR 2025
Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism, 2024. URL https:
//t////aaaaa //arxiiv. www.ivi..og/aa.oooo abaaabaalll// 231.. 04916.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. 2018.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference Learning Representations, 2019.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488,
2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. framework for few-shot language model evaluation, 2024. URL https: zenodo org weccordd/22600000
Gene H Golub, Alan Hoffman, and Gilbert Stewart. generalization of the eckart-young-mirkky matrix approximation theorem. Linear Algebra and its applications, 88:317-327, 1987.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MinilLM: Knowledge distillation large
language models. In The Twelfth International Conference on Learning Representations, 2024.
Tamir David Hay and Lior Wolf. Dynamic layer tying for parameter-efficient transformers. In The Twelfth International Conference on Learning Representations, 2024.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 2021.
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distillation: Trans- ferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.1067.. 2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint 2r.iv:2310.06630 2023a.
Mengnan Jiang, Jingcun Wang, Amro Eldebiky, Xunzhao Yin, Cheng Zhuo Ing-Chao Lin, and Grace Li Zhang. Class-aware pruning for efficient neural networks. In Design, Automation and Test in Europe Conference and Exhibition (DATE), 2024.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. The 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023b.
Lin, Jiaming Tang, Haotian Tang. Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 87-100, 2024.
11

---

Published as a conference paper at ICLR 2025
Xiuqing Lv, Peng Zhang, Sunzhu Li, Guobing Gan, and Yueheng Sun. Lightformer: weiht- transformer using svd-based weight transfer and parameter sharing. In Findings of the Association for Computational Linguistics: ACL 2023, pp.
2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702- 2023.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Dominik Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models reason. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)313-330, 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Richard Petri, Grace Li Zhang, Yiran Chen, Ulf Schlichtmann, and Bing Li. Powerpruning: Selecting weights and activations for power-efficient neural network acceleration. In Design Automation Conference (DAC), 2023.
Ruidi Qiu, Amro Eldebiky. Grace Li Zhang, Xunzhao Yin, Cheng Zhuo, Ulf Schlichtmann, and Bing Li. Oplixnet: Towards area-efficient optical split-complex networks with real-to-complex data assignment and knowledge distillation. In Design, Automation and Test in Europe Conference and Exhibition (DATE), 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
Machel Reid, Edison Taylor, and Yutaka Matsuo. Subformer: Exploring weight sharing for parameter efficiency in generative transformers. arXiv preprint arXiv:2101.00234, 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversar- ial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, 2021.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruming approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Wenhao Sun, Grace Li Zhang, Huaxi Gu, Bing Lil, and Ulf Schlichtmann. Class-based quantization for neural networks. In Design, Automation and Test in Europe Conference and Exhibition (DATE), 2023.
Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. arXiv preprint arXiv:2104.06000 2021.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An Entruuctionfoolowingg llama model, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation 2023b.
and fine-tuned chat models. arXiv preprint arXiv:2307.092266
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Jingcun Wang, Bing Li, and Grace Li Zhang. Early-exit with class exclusion for efficient inference of neural networks. In International Conference Al Circuits and Systems (AICAS), 2024a.
12

---

Published as a conference paper at ICLR 2025
Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation- aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07376, 2024b.
Yifan Wu, Shichao Kan, Min Zeng, and Min Li. Singularformer: Learning to decompose self- attention linearize the complexity of transformer. In International Joint Conference on Artificial Intelligence, pp. 4433-4441, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. International Conference Machine Learning, pp. 38087-38099, 2023.
Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. International Joint Conference on Artificial Intelligence, 2019.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6: 1969290 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 FINAL STRUCTURE OF TWO LAYERS IN LLAMA-7B WITH BASIS SHARING
Pun
pm
RMSNeem
the (i 17th Zayer
RMSNNrm
Activation
Rane
RMSNNorm
(1)th layer
MTTA
Ingedii
Figure 7: The final structure of two layers in LLaMA-7B with Basis Sharing. MHA represents multi-head attention. RMSNorm represents root mean square layer normalization.
A.2 RELATION BETWEEN COMPRESSION RATIO AND NUMBER BASIS VECTORS
For a given compression ratio, the derivation of the number of basis vectors k is explained as follows. Consider compressing WK weight matrices in consecutive n layers x% of their original sizes. Assume each Wx matrix have dj rows and d, columns. The number of basis vectors & can be calculated as follows:
dik k + kd2n did x x% + A d1d2n x x%
(d, d2n)
where didgn is the number of parameters of Wx weight matrices in layers before compression and d, + kdgn is the number of parameters after sharing basis vectors for weight matrices consecutive layers.
To compare with traditional SVD methods, the same compression ratios were used to evaluate the rank of the weight matrix in each layer individually. Consider compressing Wx weight matrix x% of its original size. Assume this matrix have dj rows and d2 columns. The rank of this matrix k can be calculated follows:
d,d2 x x%
h + kdg = d1 d2 x 5% +
d1 + d2
Under the same compression ratio (1-x%), basis sharing can lead to larger compared with that with traditional SVD-LLM, that the performance of L.LMs be enhanced.
14

---

Published as a conference paper at ICLR 2025
A.3 ANALYSIS OF MATHEMATICAL PROPERTIES OF MATRICES SHARED ACROSS LAYERS
Suppose A = SW is a matrix of the jth layer, which has di rows and d2 columns. S is the scaling matrix imposed on original weight matrix to incorporate the impact of imput data. Assume that we want to apply Basis Sharing on n such matrices in layers, where n >= 2. B is the horizontal concatenation of such n matrices, which has dj rows and nd2 columns. We analyzed the Frobenius loss F F_loss incurred by compression without and with basis sharing as follows. In the following equations, x% represents to compress the matrix to x% of its original size. The maximum value of x is 100. kand and kishre represent the number of top singular values after SVD is applied in each layer and the number of basis vectors after SVD applied in the concatenated matrices of n layers, respectively. F "_lossaed and F __oosshaare represent the Frobenius loss without and with basis sharing, respectively. σ; is the ith removed singular value after SVD decomposition. avv is the average singular value after applying SVD decomposition A. share is the average singular value after applying SVD decomposition on B.
Case 1: d1 d2, (ank(A)=rank(BB)])
d1d2
2%
kevd=
2%
d12%
di + d2
+ d2
d1
d1
1
F. Fodosssed <
Oi Rd (d)
-
d
x
% Jo avd
2
i=koud
ndd d2
5%
2 n
kshare
x%
d12%
d1 nd2
d1
ndg
d,
F F_lo88share
oi Fc (di
d10%) sshare
n
i= hare
In case that sod Jahare d, derive the following relationship:
max(nF maz(nn_losssuu) max F loss share) (n 1)(d; 2(n + 1)
d 216%
In this case, we have max(nF losssvd) > mar(F_loss_pae lossghare). which indicates basis sharing across layers can reduce the upper bound of the Frobenius loss and potentially reduce the the Frobenius loss. our work W K. Wo, Wv. Wup and W gatte in LLaMA-7B have such mathematical properties and thus can benefit from this basis sharing. However, for Wo, the assumption of awndd Tshore does not hold and share is much larger than snds so that the Frobenius loss with sharing is larger than that without sharing. Accordingly, such a matrix can not take advantage of basis sharing across layers.
Case 2: di 2 nd2, rank(A)=d2, rank
ksdd
x%
2
n
d21%
dn
dg
d2
F. P_losssed sud S
Σ
T R (dg
n
d2%% D
d2
n 1
i Ameese
x%
n
kshare
>
d22%
1
2
di
ndz
ndg
n d2a% share
F. F_lossshr
a, R (nd2
ahhare
2
AaRabare
In case that axdd T'saare T, derive the following relationship:
n²
max(nF (nF__oossssdd max(F (F lossshare) = -ndd20%
d2 1/0 <0
n +
15

---

Published as a conference paper at ICLR 2025
In this case, we have max(nF Jos) < max(F llosshhre),, which indicates basis sharing can increase the upper bound of the Frobenius loss and potentially increase the Frobenius loss.
In our work, Wdown in LLaMA-7B has such mathematical properties when and thus can not benefit from this basis sharing.
Case 3: d2 d1 nd2, rank(A)=dg, rank(B)=d)
2%
kavd
d1v%
d2
d2
1
F P_loss sud
Σ
oi (d2
d2 d2.%) 112%%)oool a
n n 1
inksed
%%
kahare
d12%
2
di
nda
F Fooossharee
0, (d) d1v%
i=kehare
In case that aodd = O Jshare = O, we can derive the following relationship:
1 1
mox(nF llssseve max(F (osssharee (nd2 d1 + 2(n
d1 21)%
11 1
1 1 n di d1æ%)o < (nd2 )
2(n d1 < (nd2 d1 +
2(n + 1)
In this case, whether basis sharing across layers has potential to reduce the Frobenius loss cannot be determined. In our work, Wowwn in LLaMA-7B has such mathematical properties when >= 3 and we decide not to share basis for Wdown across layers in LLaMA-7B.
Future work To reduce the Frobenius loss after basis sharing, we will explore the potential of vertically concatenating n matrices across layers. The vertically concatenated B has nd1 rows and d2 columns. this case, there still potential reduce the Frobenius loss follows.
For such a matrix d2 < d1 and (Ba((A))aanB))dd
x%
kavd
1
>
2
d22%
d2
do
1
F P_losssed
<
R (d2
2
i=koud
x%
kehare
d2ø%
di
ndz
n
de
F sllossshare
0,
(d2
n
d2 % shore
In case that avdd Tahare = 0, can derive the following relationship:
mar(nF F_o88sud) max F_los8share) = (n - () 2(n d20%)
T
>
In this case, the upper bound of Frobenius loss with basis sharing can be reduced. For weight matrix such as W Wdowm. will concatenate such matrices across layers vertically and decompose the concatenated matrix to obtain their basis vectors.
However, the computation of scaling matrix S to consider the impact of activations becomes more time-consuming due to the increasing number of rows, We will address this challenge in our follow-up work.
16

---

Published as a conference paper at ICLR 2025
A.4 EVALUATING ZERO-SHOP COMMON-SENT REASONING TASKS AFTER LORA FINE-TUNING
In this section, will show that LoRA fine-tuning can also enhance the accuracy of zero-shot common-sense reasoning tasks.
Ratio
Openb.
ARC_e
WinoG.
HellaS,
ARC_c
PIQA
MathQA
Avg
%
0.28(
0.67(0.67)
0.66(0)66)
(.49(0.46)
0.35().36)
0.72(0.71)
0.25(0.25)
0.49(0.48)
30 %
0.28(0.27)
0.63(0.63)
0.64(0.63)
0.45(0.40)
0.32(0.30)
0.7(0.68)
0.25(0.24)
0.47(0.45)
40 %
0.24(0.22)
0.54(0.52)
0.60(0.61)
0.40(0.35)
0.29(0.27)
0.66(0.62)
0.24(0.23)
0.42(0.40)
50%
0.22(0.18)
0.49(0.42)
0.59(0.57)
0.36(0.31)
0.24(0.23)
0.62(0.58)
0.22(0.22)
0.39(0.36)
Table 9: The performance on zero-shot common-senss reasoning tasks using LLaMA-7B compressed with Basis Sharing, with and without LoRA fine-tuning. The number in the bracket is without LoRA fine-tuning.
A.5 PERFORMANCE OF LLAMA3.2-3B WITH BASIS SHARING
Table 10: Zero-shot performance of LLaMA-3322 compressed using Basis Sharing and baselines under 20% to 50% compression ratios on WikiText-2 (measured by perplexity (J)) and seven common- sense reasoning datasets (measured by both individual and average accuracy (1)).
RATIO
MCTIOD
WikiText-2
Openb.
ARC_e
WinoG.
HellaS.
ARC_c
PIQA
MathQA
Average
0%
Original
7.84
0.31
0.75
0.70
0.55
0.42
0.77
0.35
0.55
SVD-LLM
38.39
0.19
0.53
0.57
0.33
0.24
0.63
0.24
0.39
20%
Basis Sharing
22.48
0.20
0.54
0.58
0.35
0.25
0.65
0.25
0.40
30%
SVD-LLM
44.22
0.14
0.41
0.54
0.30
0.19
0.59
0.23
0.34
Sharing
27.41
0.15
0.44
0.56
0.30
0.20
0.59
0.23
0.35
SVD-LLM
65.09
0.12
0.34
0.54
0.28
0.18
0.55
0,23
0.32
40%
Rasis Sharing
59.95
0.14
0.34
0.54
0.28
0.19
0.56
0.23
0.33
SVD-LLM
106.42
0.31
0.51
0.27
0.54
0.22
0.30
50%
Basis Sharing
104.69
0.12
0.31
0.49
0.27
0.19
0.54
0.23
0,30
A.6 COMPRESSION GAINS
To demonstrate the compression gains through layer sharing, we did two further experiments. In the first experiment, we used SVD to decompose weight matrices each layer of LLaMA-7B and compressed matrices with 20% compression ratio. Under this compression ratio, evaluated how many top & singular values were kept in the E after SVD decomposition. When basis sharing is applied group every 2, 8, 16 and 32 consecutive layers, the same value of k used as the number of basis vectors to evaluate the model performance after basis sharing. The results are shown in the following left table. According this table, with more layers shared, the compression ratio increases while the performance degrades without LoRA fine- the-tuning. However, the performance can be enhanced significantly after LoRA fine-tuning-
In the second experiment, 30% compression ratio was used to compress weight matrices in each layer evaluate the number of top singular values i kept in the Σ after SVD decomposition. Afterwards, this number was used to evaluate the performance of basis sharing, the result of which is shown in the following right table. Similarly, compression ratios increase when basis sharing enabled. The performance of basis sharing still be enhanced by LoRA fine- fine-tuning.
17

---

Published as a conference paper at ICLR 2025
Table 11:
Compression gain with basis sharing, start from 20% compression ratio. # Layers is the number of shared layers. PPL' is the PPL after LoRA fine-tuning.
Table 12:
Compression gain with basis sharing, start from 30% compression ratio. # #Layers is the number of shared layers. PPL is the PPL after LoRA fine-tuning.
#Layers
Comp. Ratio
PPL
PPL
#Layers
Comp. Ratio
PPL
PPL
20%
7.94
7.78
30%
9.56
9.14
2
29%
8,94
7.52
2
37%
11.32
8.74
4
34%
10.1
8.15
4
42%
13.56
9.12
8
36%
11.99
8.27
8
43%
19.72
9.48
16
37%
20.99
9.16
16
44%
35
10.57
32
38%
35.48
9.45
32
45%
93.85
11.00
A.7
GENERATED TEXT WITH COMPRESSED LLM
RATIO
BASIS SHARING
Original
What the universe? The universe is vast collection galaxies and stars. The Sun, Earth, Moon are all of this Universe which includes everything that can be seen with our naked eyes or telescopes such as...
20%
What the universe? The universe huge collection of interstellar objects. The Sun such object and, fact are located within this vast system known as our home star system (the solar system)...
universe? The The Sun has been at its present distance from Earth since it formed over billion years ago...
gigantic system
held logether by gravity. which binds
giant star system that contains many stars planet systems. Milky Way, the galaxy containing solar system, has two main components: the inner of the system compused of small gus...
universe? The
50%
What universe? The universe is a large collection of objects, stars. These stars are arranged in layers and form different stellar classes. The solar regions have many denser called main sequences with massive hydrogen masses, which...
Table 13: An example of contents generated by the compressed LLaMA-7B with Basis Sharing under different compression ratios. The input marked bold and the normal texts are the generated sentences.
A.8 SHARE ERROR HEAT MAP
The Frobenius loss inccured by basis sharing for Wo. Wv, WUp and WGate-
COOBE
A4TELE
10110
TAGLLE
060
Figure Frobenius loss incurred by basis sharing across any two layers. The number/color in a block represents the resulting Frobenius loss if a basis matrix is shared by two layers and the numbers in the diagonal direction are obtained by applying SVD to the scaled weight matrix of layer directly. (a) Frobenius loss incurred by basis sharing across two layers for We in LLaMA2-7B. (b) Frobenius loss incurred by basis sharing across two layers for Wy LLaMA2-7B.
18

---

Published as a conference paper at ICLR 2025
1011
10114
eecha 10011
10114
bayer
byyzr
10044
2006
2000
Figure 9: Frobenius loss incurred by basis sharing across any two layers. The number/color in a block represents the resulting Frobenius loss basis matrix is shared by layers and the numbers the diagonal direction are obtained by applying SVD to the scaled weight matrix of a layer directly, (a) Frobenius loss incurred by basis sharing across two layers for WCp in LLaMA2-7B, (b) Frobenius loss incurred by basis sharing across two layers for Wate in LLaMA2-7B.
19


Reviews:
[{"review_id": "B0N9otCxMq", "reviewer": "Reviewer_8FuT", "summary": "The paper introduces \"Basis Sharing,\" a method for compressing Large Language Models (LLMs) by sharing basis vectors across different layers. This approach decomposes weight matrices into shared basis vectors and unique coefficients, aiming to reduce memory requirements.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The concept of concatenating weight matrices from multiple layers and sharing basis is interesting. It potentially increases the number of ranks to achieve a given compression rate.\n2. The performance of the proposed method is better than previous methods on SVD like ASVD or SVD-LLM.", "weaknesses": "1. Seems the paper only considers the two-layer sharing case, it will be stronger if it can be expanded to multi-layers. The key drawback of the previous SVD method is that to achieve a 10% compression rate, for example, it requires pruning 55% of ranks. If more layers can be shared, this challenge can be mitigated to some extent.\n2. The paper does not discuss how to allocate non-uniform ranks for SVD decomposition. Non-uniform allocation of ranks has been shown promising in recent works [1]. \n3. The final structure of the model is not presented in the paper. For example, what layers and what operations have been shared in the final compressed model?\n4. Compared to pure SVD methods like SVD-LLM, a sharing basis can result in more memory costs or lower throughput since the proposed method can keep a higher rank and thus more active parameters at the inference time. \n\n[1] Gao, Shangqian, et al. \"Adaptive Rank Selections for Low-Rank Approximation of Language Models.\" Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9lSNw3z5Uy", "reviewer": "Reviewer_zP6R", "summary": "This work investigates the layer sharing of SVD bases across different layers and, based on these insights, proposes a new low-rank compression method. This method demonstrates improved performance compared to previous approaches under the same compression ratios. Plenty of ablation studies and end-to-end throughput are conducted to demonstrate the effectiveness of the proposed methods.", "questions": "Is there any insights on why different layers would share the same SVD basis?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The studies of basis sharing across different layers is interesting.\n\n- The improvements shown in Table 1 is significant compared with previous baselines. \n\n- End-to-end throughput are reported to demonstrates the effectivenss of the proposed methods.", "weaknesses": "- As shown in Table 1, while basis sharing can effectively enhance the performance of compressed models, some performance drop remains—such as a 0.04 decrease at 20% compression and a 0.07 decrease at 30%. Additionally, Figure 6 illustrates that only marginal throughput improvement is observed at a 20% compression ratio, which makes the proposed method less practical, as meaningful efficiency gains are accompanied by performance degradation.\n\n- It would be good to show whether LoRA fine-tuning can recover the performance on zero-shot common-sense reasoning tasks, rather than perplexity.\n\n- The evaluted models should contains more recent ones, like llama-3.1/3.2.\n\n- The proposed method seems like a layer-sharing and SVD co-design methods for LLM compression. How does such method compared with direct apply layer-sharing and SVD compression, e.g., applying layer-sharing, then SVD or reversely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBpLg4GMWq", "reviewer": "Reviewer_NiaD", "summary": "This paper introduces “Basis Sharing,” a method for compressing large language models by sharing basis vectors of SVD across layers, each with unique coefficients.  Basis Sharing significantly reduces model size while maintaining performance. Experiments demonstrate its superiority over traditional SVD-based techniques across various tasks and settings, achieving enhanced efficiency and accuracy at high compression ratios.", "questions": "* Can this method be used in conjunction with other compression methods (e.g. pruning, quantization, etc.) to go one step further?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* This paper analyzes the feasibility of SVD for different modules and outliers in depth, and designs a simpler and more general SVD compression method for LLM: BASIS SHARING.\n\n* The advantages and robustness of the proposed method are fully verified on various tasks, different scales, software and hardware Settings.\n\n* The presentation of the technical discussion is accurate and well-organized.", "weaknesses": "* It looks like BASIS SHARING will also have better acceleration in hardware, which I think would be better if compared to other SVD-based methods.\n\n* In this paper, continuous groups are designed based on the interesting insights in 3.2. However, ablation experiments on discontinuous groups are still missing. Is it possible to realize CROSS-LAYER PARAMETER SHARING of functions (such as W_O) that are not suitable for continuous groups through discontinuous groups?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5KvMTZnUc", "reviewer": "Reviewer_jR33", "summary": "The paper proposes a novel layer-sharing SVD compression strategy for LLMs, building upon previous SVD compression methods. The authors suggest that by sharing the left singular matrix across different layers, the model can achieve better performance. Experiments demonstrate that this method outperforms existing activation-aware SVD compression techniques.", "questions": "1. In the experiments section, the paper provides results for different compression ratios. However, I did not see an explanation in the paper on how the specific compression ratios were determined. Could you provide further clarification? And how did the authors determine the cutoff values ​​for different layers?\n2. Could the authors analyze the properties of matrices that can be shared across layers, particularly their mathematical properties, and explain why performance is improved? This result seems should be sensitive to the training data and is counterintuitive, as, in theory, not sharing should provide a higher performance ceiling compared to sharing.\n3. Could the authors provide the actual number of layers that can be shared in the model and the compression gains achieved through layer sharing?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea of sharing the left singular matrix between layers is insightful. This approach significantly improves the compression ratio of the model.\n2. The finding that model performance can be improved through layer sharing is intriguing.\n3. The experiments are comprehensive.", "weaknesses": "1. While the paper claims that layer sharing can improve performance, it does not explain why. The paper lacks an explanation of the properties of the weight matrices that enable this improvement, making the conclusion somewhat abrupt and counterintuitive.\n2. Sharing left singular vectors may not be applicable to all matrices in the model, which suggests that this is not a general approach. The paper does not provide in-depth analysis of this phenomenon.\n3. Using layer sharing implies that the truncated ranks of both layers are identical, which could limit the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)