MoDeGPT: Modular Decomposition for Large Language Model Compression
Authors: Chi-Heng Lin
Shangqian Gao
James Seale Smith
Abhishek Patel
Shikhar Tuli
Yilin Shen
Hongxia Jin
Yen-Chang Hsu
Venue: ICLR 2025 Oral (2025)
Keywords: LLM
model compression
matrix decomposition
Primary area: foundation or frontier models, including LLMs
TL;DR: A framework that expands matrix decomposition for LLM compression beyond SVD

Abstract: Large Language Models (LLMs) have significantly advanced AI with their exceptional performance across a wide range of tasks. However, their extensive computational requirements restrict their use on devices with limited resources.
While recent compression methods based on low-rank matrices show potential
solutions, they often suffer from significant loss of accuracy or introduce substantial
overhead in parameters and inference time. In this paper, we introduce Modular De-
composition (MoDeGPT), a new, efficient, and structured compression framework
that overcomes these limitations. MoDeGPT jointly decomposes pairs of consecu-
tive subcomponents within Transformer blocks, reduces hidden dimensions through
output reconstruction on a larger structural scale than conventional low-rank meth-
ods, and repurposes three classical matrix decomposition algorithms—Nyström
approximation, CR decomposition, and SVD—to ensure bounded errors in our
novel decomposition approach. Our experiments show that MoDeGPT, without
relying on backward propagation, consistently matches or surpasses the performance of prior techniques that depend on gradient information, while achieving a
98% reduction in compute costs when compressing a 13B-parameter model. On
LLaMA-2/3 and OPT models, MoDeGPT retains 90-95% of zero-shot performance
with compression rates of 25-30%. The compression process can be completed on
a single GPU in a few hours, boosting inference throughput by up to 46%.

Published as a conference paper at ICLR 2025
MODEGPT: MODULAR DECOMPOSITION FOR LARGE LANGUAGE MODEL COMPRESSION
Chi-Heng Lin" Samsung Research America
Shangqian Gao
James Seale Smith Samsung Research America
Florida State University
Abhishek Patel Samsung Research America
Shikhar Tuli
Yilin Shen
Samsung Research America
Samsung Research America
Hongxia Jin Samsung Research America
Yen-Chang Hsu Samsung Research America
ABSTRACT
Large Language Models (LLMs) have significantly advanced Al with their ex- ceptional performance across a wide range of tasks. However, their extensive computational requirements restrict their While recent compression methods based on low-rank matrices show potential solutions, they often suffer from significant loss of accuracy or introduce sub- stantial overhead in parameters and inference time. In this paper, we introduce Modular Decomposition (MoDeGPT), a new, efficient, and structured compres- sion framework that overcomes these limitations. MoDeGPT jointly decomposes pairs of consecutive subcomponents within Transformer blocks, reduces hidden dimensions through output reconstruction a larger structural scale than con- ventional low-rank methods, and repurposes three classical matrix decomposition Migorithms-Nysttiee approximation, CR decomposition, and SVD- to ensure bounded errors our novel decomposition approach. Our experiments show that MoDeGPT, without relying backward propagation, consistently matches surpasses the performance of prior techniques that depend on gradient informa- tion, while achieving 98% reduction in compute costs when compressing 13B- parameter model. On LLaMA-2/3 and OPT models, MoDeGPT retains 90-95% of zero-shot performance with compression rates of 25-30%. The compression process can be completed on single GPU in few hours, boosting inference throughput by up to 46%.
devices with limited resources.
INTRODUCTION
Recent advancements in Large Language Models (LLMs) (Thoppilan al., 2022: OpenAI, 2023; Touvron et al., 2023; Zhang et al., 2022; Al@Meta, 2024) have led to remarkable breakthroughs in the understanding and generation of natural language. Despite their significant capabilities, these models are computationally and inttory-nttensive, posing deployment challenges resource- limited devices. To mitigate these challenges, model compression (Gupta Agrawal, 2022; Zhu et al., 2023) has emerged as popular post-training solution, reducing model size and complexity.
Predominant compression techniques encompass model distillation (Sun et al., 2019; 2020; Pan et al., 2020), pruning (LeCun et al., 1989; Hassibi et al., 1993; Suzuki et al., 2018; Wang et al., 2019b; Zafrir et al., 2021; Xia et al., 2022: Kurtic et al., 2022: Ma et al., 2023; van der Ouderaa et al., 2023), matrix decomposition (Hsu et al., 2022; Noach & Goldberg, 2020; Golub & Reinsch, 1971), and quantization (Gholami et al., 2022: Bai et al., 2020; Frantar et al., 2022; Wang et al., 2023). This study focuses matrix decomposition techniques that require minimal computing sources and do not involve backward propagation as seen in recovery fine-tuning (RFT) or Fisher matrix calculations from Taylor expansion (Ma al., 2023; der Ouderaa al., 2023). Conven-
"chi heng. 1 in@samsung.com

---

Published as a conference paper at ICLR 2025
tional matrix decomposition such SVD typically splits each matrix W Rd into two low-rank matrices W = AB, requiring the rank less than d/2 to achieve true compression, as shown in Figure 1(b). This stringent requirement often results in a significant drop in accuracy, necessitating the use of RFT (Hsu et al., 2022). A novel decomposition approach, SliceGPT (Ashkboos et al., 2024), multiplies the original matrix by orthogonal matrix, effectively projecting inputs into lower-dimensional subspace and reducing the matrix's 's overall dimensionality. However, this ap- proach requires additional adapters manage the reduced dimensions; as illustrated in Figure I(c), adapters Q, 2 are added to the residual paths to facilitate this reduction. For a target sparsity s, this introduces additional 2(1 -)) parameters per layer, which can add up to 10% of additional param- eters, significantly offsetting the parameter savings. In summary, matrix decomposition approaches either (i) discard a large portion of ranks, or (ii) introduce substantial parameter overheads. These challenges significantly hinder the effective reduction parameters without compromising accuracy.
In response to these challenges, we introduce MoDeGPT, which applies matrix decomposition to multiple matrices jointly, avoiding the dual-matrix structure and extra adapters used in prior meth- ods. As depicted in Figure I(d), MoDeGPT elevates the matrix decomposition approach to modu- lar level by grouping weight matrices into modules and then applying matrix decomposition jointly within each module. Unlike SliceGPT, MoDeGPT reduces the intermediate dimensions within each module rather than between blocks, as illustrated by the matrix shapes in Figure 1(c) and (d). This crucial difference eliminates the need for adapters while still enabling dimension reduction in the compressed matrix. Importantly, MoDeGPT establishes comprehensive mathematical framework that maps each module's compression task to one of the three matrix approximation techniques: CR decomposition (Drineas et al., 2006), singular value decomposition (SVD) (Golub & Reinsch, 1971), and Nyström approximation (Gittens & Mahoney, 2013; Musco & Musco, 2017). These methods enable MoDeGPT to efficiently compress matrices. In summary, we make the following contributions:
We introduce MoDeGPT, a training-free -frei compression method that jointly decomposes mul- tiple matrices within module using fosed-formm expressions. To our knowledge, this is the first method to apply matrix decomposition at the module level for model compression.
We extend the theoretical foundations of language model weight decomposition beyond SVD, introducing systematic framework for categorizing approximation challenges in Transformer compression, complete with error guarantees.
To our knowledge. this is the first demonstration in large language models where purely matrix decomposition- based approach achieves oftte-of-hh-ar structured compression ef- ficiency, rivaling the compression rates of semi-structured pruning methods-al all without the need for recovery fine-tuning
We present a thorough evaluation of MoDeGPT, comparing it against existing methods across key metrics, including perplexity, downstream accuracy, and real-world speed im- provements. MoDeGPT preserves up to 90% of zero-shot performance with compression rates of up 30% on LLaMA 2 and 3, significantly outperforming prior approache. More- over, MoDeGPT delivers a 46% increase in inference throughput, further enhancing its practical value.
BACKGROUND AND RELATED WORK
In this section, begin by reviewing the existing body of literature related to LLM compres- sion, highlighting key contributions and methodologies in the field. Subsequently, we examine the standard components of a transformer decoder layer. Lastly, we delve into the three matrix ap- proximations employed for our proposed compression across various components in transformer layer.
2.1 RELATED WORKS
Pruning In early pruning methods like magnitude-based tuning, scalability is achieved but of- ten at the cost of reduced effectiveness in large language models (LLMs) (Hagiwara, 1994; Han al., 2015; Li al., 2017; Frantar & Alistarh, 2023; van der Ouderaa et al., 2023). To improve
2

---

Published as a conference paper at ICLR 2025
WejalaMains
Resiital Adagter
Decorpaioon Soape
MLPP
0,9
MBLA
0
Mtdis-oaa Alectiin
Hals-ical Alcesce
W
(a) Dense Transformer Layer
(b) SVD
(c) SlieeGPT
(d) MoDeGPT(ouns)
Figure Comparison of Matrix Decomposition-Bsse Based Methods for Transformer Compression. (a) Orig- inal transformer layer. (b) SVD applied each weight matrix separately, resulting in dual matrices. (c) SliceGPT multiplies each weight matrix by an orthogonal matrix Q, reducing dimensions and introducing ad- ditional adapters. (d) MoDeGPT organizes matrices into modules (highlighted by green boxes) and jointly decomposes them, producing reduced-size matrices without extra adapters.
performance while managing computational demands, frameworks such as Optimal Brain Damage (LeCun et al., 1989) and Surgeon (Hassibi et al., 1993; Yu et al., 2022; van der Ouderaa et al., 2023) incorporate second-order loss information, necessitating substantial resources for Hessian calcula- tions. Recent adaptations like WoodFisher (Singh & Alistarh, 2020), Kronecker factorization (Wang et al., 2019a; van der Ouderaa et al., 2023), and layer-wise compression (Dong et al., 2017; Frantar & Alistarh, 2022) aim to streamline these intensive methods. Concurrently, learnable parameters for pruning in vision and language models have been investigated (Liu et al., 2017: Huang & Wang, 2018: Xia et al., 2022), although these techniques generally demand significant computational meth-
sources for intensive backward propagation. Other approaches, such as ods (An et al., 2024; Ji al., 2024), have not matched the performance of gradient-basee methods like LLM Surgeon (van der Ouderaa et al., 2023). Alternatives like SparseGPT (Frantar & Alistarh, 2023), Wanda (Sun et al., 2024), and ZeroPruner (Dong al., 2024), exploring unstructured and semi-structured pruning, offer scalability but often compromise runtime speed. Additional research has utilized layer importance scores for layer pruning and sparsity distribution, as demonstrated by ShortGPT (Men et al.. 2024), OWL (Yin et al., 2023), LaCo (Yang et al., 2024), and others (Chen et al., 2024). Recent advances in LLM compression have introduced innovative methods such as LLMM-runer (Ma et al., 2023), LLM Surgeon (van der Ouderaa et al., 2023), and SliceGPT (Ashkboos et al., 2024), marking significant progress in the field by providing effective compression techniques for LLMs.
Low-Rank Matrix Approximation In related low- rank matrix techniques for compression, the traditional decomposition approach substitutes matrices with low-rank matrices but retains the original dimensions, which can limit effectiveness (Noach & Goldberg, 2020;
Table 1: LLM Compression Comparisons.
Method
No Backward
No Additional
Fully-
Propagation
Parameters
Structured
Hsu et al., 2022; Golub & Reinsch, 1971; Povey et al., 2018; Xu et al., 2023: Yuan et al., 2023; Wang al., 2024;
LLM Surgeon
Slicetar
Yu & Wu, 2023; Chen et al., 2021). MoDeGPT improves upon this by applying low-rank approximation matrix pairs, reducing the size of individual matrices and merging the additional matrices from the decom- positions. SliceGPT introduces a technique involving matrix multiplication with orthogonal matrices derived from PCA to compress weights, which reduces matrix sizes but adds additional parameters (Ashkboos et al., 2024). In contrast, MoDeGPT compresses without adding parameters by folding the decomposed matrices back the original weights. A summary of MoDeGPT's comparison other leading L.M compression methods is provided in Table 1.
SparseGPPT
MoDeGiT (ours)
2.2 TRANSFORMER ARCHITECTURE
The transformer architecture (Vaswani et al., 2017) consists of multiple decoder layers. A typical layer such in LLAMA (Touvron et al., 2023; AI@ Meta, 2024) includes two blocks: the Multi Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP). Let T, das dmm and H denote the sequence length, hidden dimension, intermediate dimension, and the number of attention heads, respectively,
3

---

Published as a conference paper at ICLR 2025
the formulation of these blocks is follows:
Type
(MLP block) [MLP(X) (xWo
(1)
Type
Type-
(MHA block) [MHA(X)= = Σ Softmax
0r (X xW
WWv, Wo,
(2)
where X E RT'xd is the input matrix, We.h WK,6, WV, E Rd * Wos R xdn the head-specific query, key, value, and output matrices. The matrices Wy R.dx and WD Rdm dd denote up and down matrices, respectively, with and 0, denoting positional embedding and nonlinear activation functions. Note that our MLP formulation encompasses the gated MLP: the up matrix is defined by the concatenations of the gated and up matrix Wy [W W nonlinear function is defined by o, (xWW)) = XW, O (XWW),), where dg is the gate function.
IT and the
In the expressions of equation 1 and equation 2, the blocks can be divided into three types of func- tional modules, each associated with a pair of matrices:
frype- X; We. WD) 0 W WD. flype- (X; W W or o, (XWo.oo (XWx.i),
mmype-m(X W W X WW,, Wo.i.
where X denotes the input and the variables after "," denote the associated matrices. These three types are distinguished by varying levels of nonlinearity. We will employ different matrix decom- position methods for compression based on the optimization tractability of each type.
2.3 LOW-RANK MATRIX APPROXIMATION
The goal of low-rank approximation method is approximate matrix W E Ed Xd2 with two low-rank matrices A E &d x and B E Igxx2, For formalism, we make the following definition: Definition 1. For low-rank approximation method M that decomposes matrix W into A and B, the approximation matrix is WM AB and the error relative to W is EM(W) = W - WMllE.
We review three approximation methods that facilitate our algorithms in the next section.
Nyström approximation (Gittens & Mahoney, 2013) W is a positive semidefinite matrix, let Sş be k-column selection matrix where each column has single non-zero element indicating the selected index, then the corresponding Nyström approximation W is,
WNys AB, where A WS, and B B W " S W. IL CR decomposition (Drineas al., 2006) Assuming W can be factored as W, W2, let S be k-column selection matrix, the corresponding CR approximation of W is
(3)
WCR = AB, where A = W, and B = S W2.
(4)
III. Singular value decomposition (Golub & Reinsch, 1971) SVD is renowned for yielding the minimum approximation error when measured the Frobenius norm. decomposes W into:
WSV AB, where A UR and B V.V.
(5)
Here, Us and V₁ are matrices containing the top-k left and right singular vectors, respectively, and Σg is the diagonal matrix consisting of the top-k singular values of W.
3 MODEGPT
MoDeGPT introduces a module-level optimization that jointly compresses two matrices within each of our three defined functional modules, rather than compressing each matrix independently in traditional low-rank upproximation methods.
An illustration of MoDeGPT is presented in Figure 2, where different colors distinguish the various modules. For each module, we apply a tailored low-rank approximation to compress the matrix pair within it. The twill hatch pattern represents dimension reductions.
In the rest of this section, we first present the mathematical objective for our approach. Then, we detail our application of low-rank approximations for effective compression within each module. Finally, we introduce a method for assigning sparsity levels across different layers that requires only one forward pass of the model on the calibration data.
4

---

Published as a conference paper at ICLR 2025
3.1 MODULAR RECONSTRUCTION OBJECTIVE
The objective of MoDeGPT is to jointly optimize two matrices within the module types described in Sec. 2.2, process we term modular decomposition, to minimize the modular reconstruction error:
V - min W. W.
E1s((, W2) f(x,; W1, W2)ll such that (W1, W2) C, (6)
where X; E Txx are samples in the calibra- tion set, and C represents the constrained search space for compressed matrices that mandates specific structures dimensions.
Up. Gate BEEE
Deenn mustis
MLP
A key motivation of our objective that it ex- pands the search space include dimension- reduced matrices, thereby increasing opti- mization flexibility and enhancing inference speedup in the compressed model. This con- trasts with independent optimization, where each matrix must adhere to the original dimen- sions.
Eagornom
HIE
Malti Head Altntic
MHA
3.2 ALGORITHMS
Layemor
From LLM compression to matrix decompo- sition The core technical contribution of this work is the establishment of one-to-one map- ping between a specific type of modular com- pression problem and a corresponding matrix decomposition problem. As outlined in Section 2.2, the modules in the transformer architecture can be categorized based on the number of nonlinear functions they contain: Type I, II, and III modules contain 1, 2, and 0 nonlinearties, respectively. For a weight matrix W within a nonlinear function, we compress it into structured form W = W Ss. where Sy is a k-column selection matrix to be optimized. This restrictive structural form is a cor- nerstone of our framework, as it ensures the tractable optimization of equation 6.
The MoDeGPT Framework. MoDeGPT divides transformer layer into three distinet colored modules, cach optimizing two matrices using specific low-rank approximation method. A twill hatch pattemn represents the dimension reduction.
Figure 2:
After characterizing the modules and the struc- ture of the compressed matrices, framework solves the modular decomposition problem in equation 6 for each module. Since each module contains a different number of nonlinear func- tions, the corresponding solutions vary. As we demonstrate in the subsequent sections, the so- lutions correspond to Nyström, CR, and SVD for Type I. II, and III modules, respectively. A summary of this roadmap is provided in Table
Module Type
Weight Matrices
up.dkwn.gatic
keyaqwrry
valuc,output
Associated Decomp.
Nystrim
CR
SVD
Nonlimearities
2
0
Compression Alg.
Alg.
Alg. 2
Alg.
Table 2: Module characteristics and their associated
matrix decompositions.
2. The detailed connections are formalized in the following subsections, with detailed proofs in- cluded in Appendix A.
TYPE-I COMPRESSION First, focus the MLP module. As detailed in Section 2.2, the ma- trices W1 and W2 that require compression are Wy and WD. Since Wy resides within a nonlinear function a. we constrain its approximation the form Wo for tractable optimization of equa- tion 6, where St is the k-column selection matrix. For Wp, we simply ensure that its dimensions are compatible with Sy. Our first theorem suggests that when single column selection ma- trix is used, the optimization in equation is closely related to the Nyström approximation of the activation correlation matrix.
Theorem 1 (MLP compression by Nyström approximation). Let Wy be searched over the matrix multiplication form WuSa, where S is a k-columm selection matrix, and let Wp be searched over Rkxda The optimal W D is then given by: (S SzS) ST. C. CD Using Wy St. and Wi as the
5

---

Published as a conference paper at ICLR 2025
Algorithm Type-I compression for MLP by Nyström approximation.
1: Input: concatcnated up and gated matrices We R4 xdm, down matrix Wo E Rduxdi, activation
correlation C, = Ei (X, Wo) (.) rank = [(1 - -parsitty and ridge intensity & Calculate the ridge leveruge score
2: 8; f (C, + + for L...., dim
3: Let Sp. E Rdux x be the matrix that selects the top h columns based on 8; scores
4: return (Wu, WD) (WuSk, (Sk C. S C. W))
compressed matrices, the Type-I reconstruction error in equation 6 satisfies:
Vi S || 1
where (C,) denotes the Nyström approximation error, defined in Def. 1, relative to the acti- vation correlation C a E (X,Ww) (X,We,) using the same Sy the compression of Wu.
Theorem shows that effective Type-I compression can be achieved through a well-designed Nys- tröm approximation of C.. Thus, propose Algorithm 1 to control the error shown below. Proposition 1 (MLP compression error). Suppose that the rank k and the scores st in Algorithm chosen such that there exists error s > 0 satisfying
Edmi Si, then the Type-I modular
2 Edant (Co), reconstruction error in equation 6 bounded by V1 S || (12 (1]-e) where dim and o; denote the intermediate dimension (i.e., the imput dimension of WD) and singular values, respectively.
TYPE-II COMPRESSION Next, we turn our attention to the Type-II module, which includes the key-query interactions within the multi-hear attention mechanisms. We will apply compression to each head independently Given that both Wq and W are embedded with nonlinear functions, for tractability in the optimization of equation 6, the matrices are compressed using a column selec- tion matrix: We = Wq and WK = WxS,, where S₃ is a shared k-column selection matrix. When both two compressed matrices are multiplied by the column selection matrix, the modular reconstruction problem naturally connects to the CR decomposition of the product of key-query correlations, elaborated in the following theorem.
Theorem (Key-Query compression by CR approximation). Let the compressed Wq. WK to be the form of WQSs, S then Type-ll reconstruction error in equation 6 has
Vn S EOR (C
(8)
where Ecr denotes the CR approximation error, defined in Def. 1, relative to C G/2 utilizing the same Sy in the compression. Here, the matrices Cq o(XWW) (X;WQ) and
Cg A EN o Toooo(Xo (X;WK)) denote the correlations of query and key states, respectively.
The preceding theorem indicates that effective compression for the Type-ll module can be achieved using a thoughtfully constructed CR approximation. In response, we present Algorithm 2, which offers the following guarantees for reconstruction:
Proposition 2 (Key-Query compression error). If we adopt Algorithm 2 then Type-ll modular reconstruction error is bounded by Vn (di=k) (Et ( (.(a)) where a; denotes the singular values.
TYPE-III COMPRESSION Finally, we focus on the Type-Ill module, which involves the value- output matrices. For clarity and simplicity, we omit the head dependency. The module has no nonlinar function involved f(X) XWy.Wo
we seek general low-rank matrices for com- pressions: Wy x Wo E R such that Wy Wo z W. Wo. The subsequent theorem
Dependency the head is omitted in the equations for of notation.

---

Published as a conference paper at ICLR 2025
Algorithm Type-II compression for key-query matrices by CR decomposition.
Input: head-specific query matrices Was E Xdda key matrices WKy E Rdaxd//M, query state correlations Co.s = E O,(X, (osssaooo (oos)"",((, (X, key state correlations Ck.j
Σ ((X.Kx.) oX,,ooX for head j = 1,..., H, and rank & = [(1 sparsity dan H Apply compression to each head independently Calculate the
2: for I..... H do
3:
4: Let S. R be the matrix that selects the top & columns based on 8, scores
5: (Was, WK.)) ) + Sw, SR)
6: return (WQ. Wx) +- (We...... Wo.....,,,...i [........ WK,H)
Concatenate the heads
Algorithm 3 Type-Ill compression for value-output matrices by SVD.
1: Input: head head-specific value matrices Wv.j E R xda/H output matrices Wo., E BBd//Hxda for head
1...., , H, input correlation C = >N x X,, and rank k [(1 sdarsittyda./aa
2: for -....,....4 H. do
Apply compression each head independently > Efficient SVD of C Wv. Wo.s (1/2) b Efficient SVD of C Wv. Wo.s (2/2)
3:
(U,E.V)) < SVD(C
4:
+ SVVV Wo,))
5: (Wv,, Wo,) - UU-" k). El: k)
6:
return (Wv, Wo) < ((Wr...... Wv,u), [Wo, Wo.s))
D. Concatenate the heads
reveals that the reconstruction can be solved optimally by applying the well-known Singular Value Decomposition.
Theorem 3 (Valuc-Output compression by SVD). we search Wv and Wo over Rd and Rkxds, respectively, the optimum in equation 6 is Wy C-1/2U, and Wo EVT. Here, UEVT and C & EN= X.T X; are the SVD of CV Wo and input correlation, respectively. The corresponding Type-Ill reconstruction error in equation 6 is exactly the SVD approximation error, defined in Def. I, relative to C' Wv Wo:
VII EvD (r
Building on the established equivalence to SVD via Theorem 3, we introduce Algorithm 3. This algorithm guarantees the following:
Proposition 3 (Value-Output compression error). Denote as the singular values, Algorithm 3 yields the optimal Type-Ill modular reconstruction error Vm CC
3.3 GLOBAL SPARSITY ALLOCATION
While MoDeGPT modules optimized locally, propose global optimization strategy that translates layer importance scores into sparsity allocations across layers. This strategy seeks to maximize the sum of importance scores, weighted by the parameters retained in each layer. To avoid the negative effects of excessive sparsity (Yin et al., 2023), we incorporate entropic regularization for smoothing. The formulation of this constrained optimization problem follows:
max PILL
s,(1 0i) + (o.) such that Σ
Φ. Lag. 0 0 di 1
(10)
where di and represent the sparsity and importance score of layer i, respectively, and davg denotes the overall target sparsity. For sufficiently large E, the following theorem demonstrates that the optimal layer sparsity distribution can be easily computed as:
Φ = Lowv x Softmax(-s/e).
(11)
Theorem 4. For sufficient large e, (11) is the optimal sparsity allocation the equation 10.
In our implementations, we adopt the Block Influence (BI) score in Men et al. (2024), which the negative correlation between layer's input and output defined by: Ex, Xout/ Xin Xomt

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 SETUPS
Models We evaluated MoDeGPT on several models that employ a sequential transformer block structure: OPT (Zhang et al., 2022) across multiple scales (125M, 1.3B, 2.7B, 6.7B), LLAMA-I at 7B, LLAMA-2 (Touvron et al., 2023) at 7B, 13B, 70B, and LLAMA-3 (AI @ Meta, 2024) at 8B.
Implementations and environments We implemented our models using Hugging Face Trans- formers (Wolf et al., 2019), with correlation computations in FP64. Model compression and perfor- mance testing were conducted on a single NVIDIA A100 80GB GPU, except for the 70B model, which we used 8 A100 GPUs. Additional details are in Appendix B.2.
Datasets Following calibration setups similar to prior studies (Frantar et al., 2022; Ashkboos et al., 2024: Dettmers al., 2023), we employed the WikiText-2 (Merity et al., 2016) and Alpaca datasets (Taori et al., 2023), each comprising 128 samples of 2048 characters. Zero-shot performance was evaluated using the LM Evaluation Harness (Gao et al., 2021), with task details provided in Ap- pendix B.2.
Baseline comparisons We benchmarked our approach against several baselines. For non- gradient-based large language model pruning, we compared it with Uniform Pruning, Magnitude Pruning, SVD, SliceGPT (Ashkboos et al., 2024), ShortGPT (Men et al., 2024), SLEB (Song et al., 2024) and Optimal Brain Damage (LeCun et al., 1989). For methods involving backward propaga- tion, our comparisons included LLM-Prumer (Ma et al., 2023) and LLM Surgeon (van der Ouderaa al., 2023). Additionally, Appendices B.4 and B.5, we evaluated our methods against feature- mimic compression techniques and SVD-based methods, respectively.
4.2 GENERATION PERFORMANCE
Table Perplexity comparisons of structured pruning methods for LLAMA-2 7B and 13B on WikiText-2, calibrated with 128 sequences of 2048 tokens.
Method
7B (ppl: 5.12
LLAMA-2
(ppl)
()
Gradlent
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
K-DRD (LaCas at., 1989)
X
5.48
9.14
15.43
28.03
4.91
6.29
10.08
13.06
16.06
L.uM-rrueer (Ma al., 2023)
X
7,11
9.29
13.56
17.90
31,05
5.57
6.67
12.19
19.56
32.20
1.M surgoon (van der Oadema at., 2023)
X
5.25
6.18
7.83
10.39
15.38
4.69
5,29
7.25
9.43
Unifom
19.09
27.13
46.25
176.24
327.99
13:78
18.18
29.05
45.44
82.60
Magnitude
861.76
821.34
9623
Overflow
Overllow
320.30
723.60
2105
3004
SVD
Overtlow
Overflow
52719
51229
Overfloo
7665
9919
21248
53671
39521
ShonGPT (Men al., 2024)
6,98
14.31
33.21
71.04
268.11
5.40
7.69
30.48
48.83
187.23
SLEB (Song et al., 2024)
6.05
7.64
11.25
29.10
103..00
5.23
6.31
8.24
11.76
27.67
SliceGP (Astiibous al., 2024)
6,46
7.68
10.47
15.19
24.82
5.67
6.68
3.68
12.56
20.57
MoDDeGPT (ours)
5.48
6.16
7.51
8.41
11.88
4.33
5.29
6.10
6.95
8.95
We evaluated the generation performance of compressed LLAMA-2 models (7B and 13B) using the WikiText-2 test split in Table 3, 19 and B.3. Results for OPT and LLAMA-3 8B are included in Ap- pendices B. and B.3. The table distinguishes between compression methods using gradients (top rows) and those without (bottom rows). Among non-gradient methods, the traditional matrix de- composition approach using SVD performed the worst. In sharp contrast, MoDeGPT outperformed all other baselines at various compression rates by jointly applying decomposition to multiple matri- ces within a module; it only increased the perplexity by 20% for 20% compression of the 7B model, which is substantially better than the next best alternative that saw 50% increase. In comparison to gradient-basee methods, MoDeGPT surpassed other structured compression techniques except for a low compression rate (20%). This demonstrates that using local reconstruction as a proxy for true loss can achieve state-of-the-art compression.
4.3 ZERO-SHOT PERFORMANCE
We evaluated our method on zero-shot tasks, comparing to leading baselines in Table 4. Our method showed superior performance at higher compression rates. The bottom rows indicate that calibrating with the Alpaca dataset (instead of WikiText-2) significantly improved performance, with 30% compression resulting in only 10% accuracy drop. This effect more pronounced for LLAMA- 13B. as shown in Table 13 in Appendix B.3. We also tested the newer LLAMA-3 8B model, adapting our algorithm for grouped query attention head dependency detailed in Appendix
8

---

Published as a conference paper at ICLR 2025
Table 4: Zero-shot task performance of compressed LLAMA-2 7B and LLAMA-3 8B.
Model
Compress.
Method
ARC-e
ARC-c
PIQA
WinoG.
HellaS.
Average
0%
Dense
74.58
46.25
79.11
69.06
75.99
69.00
ShonGGPT (Men et 2024)
48.65
32.85
64.31
64.33
56.13
53.25
SliceGPT (Ashikbues et al., 2024)
58.88
33.36
68.55
58.01
49.86
53.73
30%
LLM surgoon (van der Ouderaa et al., 2023)
63.09
36.69
73.56
61.09
60.72
59.03
MoDeGPT (ours)
63.26
38.73
70.40
67.32
63.26
60.78
LLAMA-2
MoDeGPT-AIpan (ours)
65.49
39,16
73.34
66.22
65.90
62.02
7B
ShonGPP (Men al., 2024)
41.16
29.94
60.12
60.46
43.67
47.07
SliceGPT (Ashkboos et al., 2024)
36.49
24.57
54.90
53.43
34.80
40.84
40%
LLM surgeon (van der Ouderua et al., 2023)
52.31
30.29
69.26
54.38
48.04
50.86
MoDeGPT (ours)
49.45
30.03
64.96
61.96
53.01
51.88
MoDeGPT-Alpacu (ours)
$9.76
34.73
70.35
64.40
58.63
57.58
0%
Dense
77.69
53.58
80.63
72.69
79.16
72.75
LLAMA-3
ShortGiPT-- PBoriciP-AAlpaca (Men et al., 2024)
38.13
31.40
60.94
54,22
31.52
43.24
8B
25%
BiceeiPT-Alpuc (Ashikbous et al., 2024)
44.44
29.27
57.56
58.48
41.08
46.17
MoDeGPT Alpaca (ours)
67.05
41.13
75.52
69.61
66.49
63.96
Table 5: Comparisons of 30% compression on LLAMA-2 70B using 128 wikitext-2 samples for calibration.
Method
WikieTest22
ARCC
ARC-
P1Q4
WinGG.
HellaS.
BodO
08QA
MatMQA MMI.L-ml
COPA
Avurage.
57.25
82.75
77.90
83.83
48.80
42,86
94.00
SlxeGPT (Achkhooss et al. 2024)
5.76
67.05
42.06
67.52
71.11
55.57
41.56
40.20
27.87
52.14
$2.00
52.03
52,65
ShuiGPT (Maretal,, 2024)
66.3
60.65
34.47
72.74
64.01
63.80
66.83
34.40
23.05
31.25
75.00
27.01
43.06
SI.BR (Song ot 2024)
5.54
71.97
44.20
27.74
69.38
73.54
67.25
41,80
27.47
32.15
$8.00
64.22
90.79
McDeGPT LWL. spunity
4.67
76.01
50.34
74.70
72.85
72:45
69.85
44.20
32.26
44.64
$7.00
69.11
63.08
McDaGPPT+ our spesily
4.89
77.49
$0.94
77.53
76.87
78.16
74.71
45.60
35.04
43.86
90.00
72.17
63.51
McDeGPT spuriiy 4 Alpaca
78.57
51.54
$0.85
77.19
79.60
33.83
40.18
94,00
70.73
B.I. The performance gap between our method and baselines was notably larger with this model, aligning with quantization challenges observed in (Huang et al., 2024).
Finally, we compared our method against decomposition and layer-pruning baselines in a large-scale experiment on the LLAMA-2 70B, as shown in Table 5, Our method demon- strates improved performance in larger models, with mini- mal drops of 4.51% 51% and 3. 23% in zero-shot task performance 30% compression. This is achieved using only 128 sam- ples from WikiText-2 and Alpaca, respectively, for calibration, without requiring recovery fine-tuning. This result highlights the scalability and effectiveness of our approach in large mod- els. On the middle two rows, we compared our sparsity al- location strategy with the recent state-of-the-art OWL method (Yin et al., 2023). While our method shows a slightly higher perplexity, it consistently achieves superior zero-shot perfor- mances. Appendix 1. provides additional analysis on the comparisons with OWL.
Table 6: Compute time.
MotheGaT
L.M
Time
GPUs
Time
GPUS
LLAMA 2 7B
4149m0 IxA100
17k08m
AxH100
ELAMS-2 138
4k7hhm 1sA100
100
McDxGPT
PPL (WNTTMM)
-00
ShwnGPP
SLEB
1400 1600 1800
2000 2200 2400 2600 2800 Thooulapaa (skensiee))
Figure 3: PPL vs. throughput.
4.4 COMPUTATION AND THROUGHPUT
Table 6 compares the compression costs of MoDeGPT with LLM surgeon, the best-performing prior work. Given that the hourly rate for H100 about twice that of A100 (Lambda, 2024), MoDeGGT offers significant cost savings-97% for the 7B model and 98.5% for the 13B model. Next, we analyze the trade-off between model performance, measured in perplexity (on WikiText-2), and throughput (tokens/sec), as illustrated in Figure 3. For this experiment, we set the sequence length to 256 and measured the average generation time of LLAMA-2 7B on a single A100 GPU with a batch size of 256. As illustrated, MoDeGPT consistently outperforms other models in both speedup and accuracy across various compression sizes, with size ratios annotated alongside each point. Remarkably, at 50% compression, MoDeGGT increases throughput by 58% while maintaining per-

---

Published as a conference paper at ICLR 2025
Type (Nalue Oulput))
Type
II
II1
LLAMA-78
Type (Key Query)
Purumeters
MLP
K-Q
V-O
(model size: 13.81 GIB)
Perplesiy
7.5
Type (Galed MLP)
Method
Nystrom
CR
SVD
Peak Memory
GPU
(GiB)
bours
Size Ratão
66.84%
16.58%
16.58%
6.5
15.54
Complexity
(14)/111)
((Hdd)
MHAA
2h52m
(+11.556)
5.5
Effective
0.094
0.121
0.095
23.33
Compression
Time
1h13m
0h36m
2h26mm
0%
30%
50%
Figure 4: Module-wisc compression.
Table 7: Module breakdown statistics. Table 8: Memory utilizations.
plexity on par with the top competitor at 30% compression. Further details on speedup, including different batch sizes and hardware setups, are in Appendix B.16."
4.5 ABLATION STUDY
WinTeat-2 pp
We first analyzed the impact of compression on each mod- ule type within the LLAMA-2 7B model using single A100 GPU. As shown in Figure 4 , the majority of perplexity degra- dation occurs when the MLP module compressed. How- ever, after normalizing by parameter size, i.e., the effective ra- tio in Table it becomes evident that the Type-II module is the most sensitive to compression. This observation aligns with our theoretical analysis, which demonstrates that Type- II has the weakest error bounds, as it is constrained by the complete spectrum rather than just the residuals (see Proposi- tions 1, 2, 2) 3). In the middle, Table 7 provides detailed break- down of the module-wise compression statistics. Notably, the SVD method dominates the compression time for the value- output components, suggesting that techniques such as SVD approximation (Yuan et al., 2023) have the potential to reduce overall compression time. Meanwhile, Table 8 reports mem- ory usage, showing that MLP compression requires the most memory, as it has the largest correlation dimension among the modules. Despite this, all compression tasks only consumed up to 23 GiB of memory, which is approximately double the model size. A similar memory consumption pattern for the 13B model is discussed in Appendix B.9.
7.8
60
50
7.4
32
04
128
256
612
1024
Figure 5: Impact of calibration size.
105.
80%
40%
30%
20%
Sparsity
10%
Figure Score-to- -oarsstt conversion.
Table Sparsity allocation.
Perplexity
Zero-Shot
Thens
3.12
60 69.00%
Unitonn
53.47%
Spursity Alloc
60.78%
Second, we explored effects of calibration size on a 30% compressed LLAMA-2 7B model. As shown in Figure 5, increasing the calibration size initially boosts performance; however, the gains zero-shot performance diminish for sizes larger than 128.
Lastly, we evaluated sparsity allocation effects the same model. Figure 6 shows the score-to- sparsity mapping from Section 3.3 with s = 0.1, highlighting areas higher sparsity in darker shades. Our findings indicate that certain layers, such layer 26, can forgo up 82% of parameters with minimal accuracy loss. Table 9 demonstrates that our global sparsity allocation significantly surpasses uniform approach, affirming the efficacy of decomposition method with simple scoring function for sparsity distribution.
5 CONCLUSION
In this work, we introduced MoDeGPT, a novel structured compression method that generalizes matrix decomposition to the modular level, achieving state-of-the-art results for structured model compression via low-rank decomposition. Our approach has a strong theoretical grounding, of- fering bounds on the reconstruction errors for the components in the Transformers. Furthermore, MoDeGPT stands out by relying solely on forward propagation to achieve comparable or better compression performance methods that use the gradients from backward propagation. We be- lieve our novel methods and findings will inspire more theoretical and algorithmic innovations for training-free model compression.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Al@Meta. Llama 3 model http:///gihub..co/weiiiee
2024.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv: 13245, 2023.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Joncel-Kedzionse Yejin Choi, and Hannanch Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. arXiv preprint arXiv: /905. 2019.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive struc- tured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.1502 2024.
Haoli Bai, Wei Zhang. Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit bert quantization. arXiv preprint arXiv:2012.15701, 2020.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Patrick Chen, Hsiang-Fu Fu Yu, Inderjit Dhillon, and Cho-Jui Hsich. Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems, 34:29321- 29334, 2021.
Xiaodong Chen, Yuxuan Hu, and Jing Zhang- Compressing large language models by streamlining the unimportant layer. arXiv preprint 1rXIv:2403.19135, 2024.
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: /905. 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv:1803.05453 2018.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306. 200.:233000000788. 2023.
Peijic Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2446.0292. 2024.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pp. 4857-4867, 2017.
Petros Drineas, Ravi Kannan, and Michael W Mahoney. Fast monte carlo algorithms for matrices Approximating matrix multiplication. SIAM Journal on Computing, 3621)1332-157, 2006.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.

---

Published as a conference paper at ICLR 2025
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322. 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et framework for few-shot language model evaluation. Version vO. 0. Sept, pp. 8, 2021.
Amir Gholami, Sehoon Kim, Zhen Dong. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Alex Gittens and Michael Mahoney. Revisiting the nystrom method for improved large-scale ma- chine learning. In International Conference on Machine Learning, pp. 567-575. PMLR, 2013.
Gene H Golub and Christian Reinsch. Singular value decomposition and least squares solutions. In Handdooo for Aulomatic Computation: Volume II: Linear Algebra, pp. 134-151. Springer, 1971.
Manish Gupta and Puneet Agrawal. Compression of deep learning models for text: A survey, ACM Transactions Knowledge Discovery from Data (TKDD), 16(4):1-55, 2022.
Masafumi Hagiwara. A simple and effective method for removal of hidden units and weights. Neurocomputing, 6(2):207-218, 1994.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299, IEEE, 1993.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Emma Tang, Dawn Song, Jacob Steinhardt, and Andy Holzinger. Measuring massive multitask language understand- ing. arXiv preprint 2009 200900000000000 2020.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint DrXiv:2/06.09685, 2021.
Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of Ilama3 quantization: From Ilms to mllms. arXiv preprint arXiv:2404. 14047, 2024.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-320, 2018
Yixin Ji, Yang Xiang. Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, and Min Zhang. Feature- based low-rank compression of large language models via bayesian optimization. arXiv preprint arXiv:2405. 10616, 2024.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-orde pruning for large language models. arXiv preprint urXiv:2203.07259, 2022.
Lambda. Lambda cloud. https:/tambdalaas.co com// 2024.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017.
12

---

Published as a conference paper at ICLR 2025
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-217200 2023.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):13-330, 1993.
Shannon McCurdy. Ridge regression and provable deterministic ridge leverage score sampling. Advances in Neural Information Processing Systems, 31, 2018.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXhv:2403.03853, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 179.0436,9.78843
07843, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint (0XXv:/809.027000 2018.
Cameron Museo and Christopher Musco. Recursive sampling for the nystrom method. Advances in neural information processing systems, 30, 2017.
Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix de- composition. In Proceedings of the 1st Conference of the Asia- Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 584-889. 2020.
OpenAl. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. arXiv preprint 2xXi:20022.00266 2020.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1525-1534, 2016.
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and San- jeev Khudanpur. Sethoorth low-rank matrix factorization for deep neural networks. In Interspeech, pp. 3743-3747, 2018.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series, 2011 1.
Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neurul Information Processing Systems, 33:18098-18101 2020.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining Ilms through redundancy verification and elimination transformer blocks. arXiv preprint arXiv:2402.0002 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
13

---

Published as a conference paper at ICLR 2025
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Tweifth International Conference on Learning Representations, 2024. URL https: : wwwp///wwwweeeeeeeeeeeeeeeeeeeep
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprini arXiv: /90:: 093350 2019.
Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang. and Jingjing Liu. Contrastive distillation intermediate representations for language model compression. arXiv preprint arXiv:2009.14167, 2020.
Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural net- works via spectral analysis and its generalization error. arXiv preprint 1808.0880 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An mostruction-followingg Ilama model, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv24403.08295 2024.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201. 082399 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 13X11 2023.
Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, Yuki M Asano, and Tijmen Blankevoort. The Ilm surgeon. arXiv preprint arXiv: 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the tronecker-factored eigenbasis. ICML, 2019a.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang. Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXhv:2310.11433 2023.
Xin Wang, Yu Zheng. Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-awan singular value decomposition for large language model compression. arXiv preprint urXiv:2403.073777 2024.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint 190000433 2019b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-ther the-art natural language processing. arXiv preprint arX1v:1910.03771. 2019.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00442 2022.
Mingxue Xu, Yao Lei Xu, and Danilo P Mandic. Tensorgpt: Efficient compression of the embedding layer in llms based the tensor-train decomposition. arXiv preprini urXiv:2307.005220 2023.
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.111882 2024.
14

---

Published as a conference paper at ICLR 2025
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang. and Shiwei Liu. Outlier weighed layerwise sparsity (owl): missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05175. 2023.
Hao Yu and Jianxin Wu, Compressing transformers: features low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11007-11015,
2023.
Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: pruning weights that cancel one another in neural networks. In International Conference Machine Learning, pp. 15668-25688 PMLR, 2022.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0582. 2023.
Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:21 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1905007830, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. Finercut: Finer- meainee interpretable layer pruning for large language models. arXiv preprint arXiv:2405. 162055888188 18218, 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey on model compression for large language models. arXiv preprint arXiv:2308.076333 2023.
15

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL
CONTENTS
A PROOFS
17
A.I Proof of Theorem and Proposition 1: ML.P Compression with Nyström Approxi- mation
17
A.2 Proof of Theorem and Proposition 2: Key- eeyyrrrry Compression with CR Approxi-
mation
19
A.3 Proof of Theorem and Proposition Output Compression with SVD
20
A.4 Proof of Theorem 4: Global Sparsity Allocation
20
ADDITIONAL EXPERIMENTS
22
B.1 Modified Algorithms for Grouped Query Attention
22
B.2 Implementation Details
22
B.3 Additional Generation and Zero-Sho Shoo-Sht Experiments
23
B.4
Additional Baseline Comparisons: Feature-Mimic and SVD Approaches
26
B.5
Additional Baseline Comparisons: Unstrnctured/Sem-SSmruuudd Compression....
26
B.6
Recovery Fine-tuning
27
B.7
Experiments with Equal Computational Budgets
28
B.8
Combination of MoDeGPT and SliceGPT
28
B.9
Compression Time and Memory Consumption
29
B.10 Global Sparsity Allocation
30
B.11 Refined Sparsity Allocation: Differentiating MLP and MHA Blocks with Finer- Grained Scores
B.12 Ablation Study on Compression in Each Module
32
B.13 Scalability Larger Models
33
B. 14 High Compression Rate Experiments.
34
B.15 Sensitivity Analysis of Dfferent Calibration Sets
34
B.16 Additional Speedup Experiments
34
Limitations and Broader Impacts
35
16

---

Published as a conference paper at ICLR 2025
A PROOFS
This section provides proofs for the theorems and propositions in the main text, along with defini- tions and assumptions for formalism.
First, we define the following notation.
Definition (Column Selection Matrix). k-column selection matrix Sy is matrix with k columns where each column has single non-zero element indicating the selected index. For example. Ss [[0, 0, 1, _]T, [0, 1, 0, 0]" [0, O, 0, [] a 3-column selection matrix selecting the third, second, and fourth columns. An important property that any matrix right-multiplied by column selection matrix will result in matrix consisting of the selected columns.
Next, we make an assumption regarding the nonlinear functions used in modules, which is crucial for validating our algorithms.
Assumption 1. Any column selection matrix S is commutative with the nonlinear functions under consideration. Specifically, the function satisfies the property that a (X)S = o(X for any X and any column selection matrix S.
Importantly, Assumption is met by any activation function that operates element-wise on the in- puts, as well as by widely used embedding functions, such as the rotary positional embedding (Su et al., 2024).
A.1 PROOF OF THEOREM 1 AND PROPOSITION 1: MLP COMPRESSION WITH NYSTRÖM APPROXIMATION
Theorem (MLP compression can be solved by Nyström approximation). Let Wu be searched over the matrix multiplication form Sy where S₁. is k-column selection matrix, and let WD be searched over Rixds, The optimal W can then be expressed (S) C, Sy CDWD Using Sy and W as the compressed matrices, the Type-l reconstruction error equation satisfies:
Vi S 1W (C,
(12)
where ENya ENya(Ca) denotes the Nyström approximation error, defined in Def. 1, relative to the activa- tion correlation matrix Co & EN O ( (X,Wo,) using the same Sy. in the compression of We.
Proof. Ideally, we want to seek low-rank matrices WU. WD without any constraint; however, the nonlinearity between the two matrices makes the optimal solution intractable. To overcome the nonlinearity between the two matrices, we instead restrict the compressed up matrix Wy be of the form St. S& Rd is k-column selection matrix and Wp is general Rexd matrix. Plug in this form, we can simplify equation 6 as
min St. WD
- (x
(a)
min
St. WD
N
Σ o (X.Wo) ( (WD S,,,)) (WD-S.W,) S,W))
min
Tr
Sg. Wp
= min
|C
WD Sa wD,) 1
(13)
W
where C. is the empirical correlation matrix of latent features C. = o C
(X,,
and (a) follows from Assumption 1. Setting the gradient of the last expression with respect Wp
17

---

Published as a conference paper at ICLR 2025
zero, we obtain the optimal down matrix W (s) C. St) back to the objective, we can further simplify the objective into,
S C. CD. After Plugging this
c. S (s) C.S)) STC,) woll (c) (c)-
min S.
I | 11 C. Sb (S Sx Sx Co 2 1W W (1 (C.).
(14)
Now, observe that the error on the right side of equation 14 proportional the Nyström matrix approximation to the matrix C, in Definition 1. Hence, the variable Sg can be optimized with any Nyström approximation algorithm (Gittens Mahoney, 2013). In this work, we adapt de- terministic Nyström algorithm, Algorithm 1, that has the theoretical guarantee proved in the next proposition.
Proposition 1. Suppose that the rank k and the scores S; in Algorithm I are chosen such that there exists an error & >> O satisfying e 2 Σis4+ 84. then the Type-1 modular reconstruction error in equation 6 is bounded by V1 S I d Eimrt of(C)), where dame and denote the intermediate dimension (i.e., the input dimension of Wp) and singular values, respectively.
Proof. Since column selection is equivalent to applying the deterministic ridge leverage score sampling (DRLS) to C. (McCurdy, 2018), Theorem in McCurdy (2018) () implies that
(1 C) h
CC
)M
I c S.Sk ct 3 C,
(15)
6
(C //
2
C.S.S.CC S.S. c². I 1-
C. 3
(16)
k(1 5)
Next, define P = Co (S C. Sx)' S C². We note that P is the projection matrix of the
column space of C. S. Now, we multiply I - P to both sides in the previous inequality to get
(I - PI -P
(17)
1 6 (
[C)) ) | (I (I-P)+ + 1-
(I- PP S.S. CI
(18)
k(1 - 5)
1 k(1 - [C )m 122
(19)
where in the last inequality we the fact that I- P-P < I and that - is the orthogonal
projection to the orthogonal complement of the column space C² S so that S ST.. (I P) 0. Now, we have
du O,,C.
|| (I - 1(1
(20)
1(
im
k++
du
+ -1
| P)C
Σ
(Ca)
(21)
k(1
Since PC. C.S₁ (S) C.S)) S C. the inequality is equivalent to
du
|C, - c,s C.s)* ST CTlll2 S 1(1
(C.)
(22)
1=
18

---

Published as a conference paper at ICLR 2025
Finally, complete the proof by,
(a)
Ey, (C,)
S dmllIs - C,S CST s C
(6)
d'nt
oi(Co)
1.211 =)²
1=k+1
(0)
E² d
dint
5
(C,),
(23)
11211 2)²
1=
where (a) follows from that||AA] - Adll for any matrix R (b) from equation 22, and (c) from Cauchy inequality that (2im) x;) I n > zi for any sequence {a:}i.
A.2 PROOF OF THEOREM 2 AND PROPOSITION KEY-QUERY COMPRESSION WITH CR APPROXIMATION
Theorem 2 (Key-Query compression can be solved by CR approximation). Let the compressed Wq. WK to be the form of WQ WK Sa, then Type-ll reconstruction evror in equation 6 has
Vi S Eo (C
(24)
where ECR denotes the CR approximation error, desined in Def. 1, relative to Cº C/2 utilizing the same Sy in the compression. Here, the matrices Co - o ( (X, WQ) and Cg - N En (X.WKK (X,Wk) denote the correlation matrices of query and key states, respectively.
Proof. Regarding two nonlinear functions satisfying Assumption 1, we propose to optimize the reconstruction error with compressed key query matrices of the form Wx k Sk. Sq where Sş is some column selection matrix. Now the reconstruction error of this module
- OX
(o (x,,)
Ir --5
(6) 1 TI Σ (1-S.:)) o, (x (x (1- Jx
Tr (CK (I - S.ST) Co) = |c c El (C
(25)
where Cx En= . aX WX
Co Co=Ei of X.W o
the correlation matrices associated with the outputs of Wa and W respectively. Here, (a) follows
from Assumption 1, (b) follows from that (I- S.SS. (X, or (X,Wo) (I
S.5 ) and
o, o,(X, (X, Wx) T or (X, Wx) are positive semidefinite, and (c) follows from that I- - < I. From last expression, we observe that the reconstruction is bounded by the CR approximation (Drineas
al., 2006) to the matrix patrix-product C c
Proposition If we adopt Algorithm 2 then Type-ll modular reconstruction error is bounded by Vn <
da-k 2
Σxen o<(Ck)) (eila)aCCo)) where as denotes the singular values.
19

---

Published as a conference paper at ICLR 2025
Proof. Our Algorithm 2 is deterministic variant of Drineas et al. (2006). Recall that
d
Ec = ||C cº - CRS calls =1
Σ
kgg.
le.,
(26)
1=++1
where k; and qi are the i-th column and i-th row of C and C². respectively. Then,
(a)
k
k
12111111112
1
(27)
A
Jamke R
(b) d k
d d k
k
Σ
11::13
(28)
i==1
(
d - k
Σ (C(Ck))
Σ
oi
(29)
where in (a) we use Cauchy-Schwarrs inequality and in (b) we use the fact that the column selection based on the norm product in Algorithm 2.
A.3 PROOF OF THEBREM 3 AND PROPOSITION 3: VALUE-OUTPUT COMPRESSION WITH SVD
Theorem 3 (Value-Output compression can be solved by SVD). If we search Wy and Wo over RRd xk and Rkxds, respectively, the optimum in equation Wv C- and Wo ΣVT. Here, UEVT and C & Em= X, X, are the SVD of C Wy Wo and input correlation matrix, respectively. The corresponding Type-Ill reconstruction error in equation 6 is the SVD approximation error, defined in Def. I, relative CiWyWa
VII E (C WyWo).
(30)
Proof. Wv E Raxk and Wo Rexd Plug in f(x) XW, Wo into equation 6 and simplify yields the objective
min Σ Tr xx-xx WWWW -WW ))wo))
Wv.Wo Wo 1=1
min Wv.Wo
- (C.
W. - C.W. E
(31)
where C = EN X X, is the input correlation matrix.
Proposition 3. Denote o; as the singular values, Algorithm 3 yields the optimal Type-Ill modular (C
reconstruction error Vm = o Wo).
Proof. As Ct WWWW Wo has low rank k, this reconstruction error is upper bounded by the residue of the spectrum of the matrix č Wy Wo. i... εSDD S / (C Wy Wo). In fact, the upper bound is achievable by Algorithm 2 since C.W,,, U.E,V. V.T. which the optimal rank & approximation to the matrix C W.W.
A.4 PROOF OF THBOREM GLOBAL SPARSITY ALLOCATION
Theorem 4. For sufficient large e, (11) is the optimal sparsity allocation the equation 10.
20

---

Published as a conference paper at ICLR 2025
Proof. Consider the relaxed optimization problem
8c(1 (1) +H(( s.t. Σ 6i
Dang.
(32)
Its associated Lagrangian is
C(dia, A) E>5 8c (1 (o.) +
1
Σ
di
Dwv
(33)
T.
To find the optimum, set the gradient of the Lagrangian zero, which yields
VoL(o1., A) = V
Σ 8/(1 0.) - -HH()))) + AVA
1
Di
Darg
(34)
T.
L
= 6
Σ s/(1 - 6i) -
Φi log Φ5
0 Davg
(35)
L
==1
This is equivalent to that, for any j 1,..., Lu
L
0 = 3o3 Σ 8:(1 - 6,) -
Σ
Di log
Σ
0i -
Davg
(36)
i=1
i=1
1
-j Elog Ø5 - + x
(37)
After rearrangement, we have 0j = C (xep(=s)/s) for some constant C. On the other hand, Φ
satisfies the constraint of Σ 1= 0; = Gavg, which implies
Σ
C
= Liding
(38)
C Lowa
Σ expp(ss]1
(39)
Φ: = Loav exp(-s; ss/EE/
Σ exp(-s;/E). j=1
(40)
Finally, we must verify that for any I...., L, the above expression of o; is a valid sparsity allocation, satisfying Φ; S 1 for sufficiently large E, to ensure it is also the optimum solution to the original optimization problem in equation 10. Since di Todkag exp(-x:/e)) 8s/e)/ Σ exp(-sj/e) is continuous function of & and lim Φ; Qavg < 1, there must exist some constant N; such
that when E > Nr. Φ: is less than 1. Hence, the sparsity allocation is valid optimal solution to equation 10 if max(N Nr), completing the proof.
21

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS
B.1 MODIFIED ALGORITHMS FOR GROUPED QUERY ATTENTION
Some modern LLMs such Gemma (Team al., 2024), Llama 3 (AI@ Meta, 2024) utilize shared key-query strategy to improve inference efficiency. This design adopts the grouped-query attention (GQA) mechanism (Ainslie al., 2023) that couples multiple queries and output matrices with shared keys and values, so compressed key and value matrices also have the constraints of sharing across different heads. Its mechanism is illustrated follows.
H/G
(GQA) Σ Σ
Softmax (XW)) (XW)) XWWW
(41)
2 JEG:
Type-11
Type-111
where G; denotes the set of each group and
G - IG,] is each of its size. We see that our compressed Wi. W must jointly optimized within each group. To address it, we modify Algorithm 2, by using projec- tion strategies. In line 3 of Algorithm 2 for Type-II module, we calculate the group score equal to the square root of sum of the scores of each head within group, i.e., 8 =
Grouped Query Attention
w
v Eneg ||C [I, e]l?, where h indicates the head. By doing in this way, sure that the column selection matrix for com- pressions remains equal within the group. For grouped Type-lll modification, in line of Al- gorithm 3, we caleulate the SVD of CWv UEVT and skip the calculation of W and the second SVD and then outputs Wv WyUks Wo,; U Wo.j. Vj E G; Since Wy is shared within a group, this ensures that the compressed Wv is also shared. In Table 14, we apply this modification Llama-3 8B compression.
W
0,6
Figure 7: Illustration of Type-1l and Type-Ill modi- fications GQA. In Type-I, the index selection matrix S is shared among different key projection matrices in the same group. Similarly, in Type-II, the eigenmatrix shared among different output matrices within same group.
B.2 IMPLEMENTATION DETAILS
Setup We utilize the HuggingFace generation library (Wolf et al., 2019) to implement our LLM models and adapt the SliceGPT (Ashkboos et al., 2024) GitHub repository for correlation matrix estimations. All compression experiments were conducted a single NVIDIA A100 80GB GPU, except for the 70B model compressions, which utilized 8 A100 GPUs. The models use the FP16 data format. Unless otherwise specified, the calibration set consists of random sample of 128 sequences, each of length 2048, from WikiText-2, following the common practice in the literature Ashkboos al. (2024); der Ouderaa al. (2023).
Datasets We consider multiple tasks in LM Evaluation Harness (Gao et al., 2021), including ARC- ARC-c (Clark et al., 2018), PIQA (Bisk al., 2020), WinoGrande (Sakaguchi al., 2021), and HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), MathQA (Amini et al., 2019), BoolQ (Clark et al., 2019), COPA (Roemmele al., 2011), MMLU (Hendrycks al., 2020), and LAMBADA (Paperno et al., 2016).
Conversion of Layernorm to RMSNorm We adapt SliceGPT (Ashkboos et al., 2024) official code to implement our compression. As shown in the work, this conversion is an invariant trans- formation that preserves the model output. SliceGPT uses this transformation and the orthogonal invariance property of RMSNor to slice the weight matrices. On the other hand, MoDeGPT does not reply on the invariance property. We use the transformation simply for easy adaptation from SliceGPT by avoiding building all from the scratch. A side benefit is that work is compatible
22

---

Published as a conference paper at ICLR 2025
with SliceGPT where slicing and our compression be applied independently. Although our experiments on OPT and LLAMA do not find clear improvement when incorporating the two (see Table 22), it might be beneficial for some other LLMs.
Correlation Matrix Estimations Our algorithms utilize various input correlation matrices as de- tailed in Algorithms 1, 2, and 3. Following the approach used in SliceGPT (Ashkboos al., 2024), we employ the Catcher function to gather empirical data from the calibration set. For matrix de- composition, we upcast correlation matrices from FP16 FP64 and then downcast the decomposed weights back to FP16. Our process sequentially compresses weights across all layers, mirroring SliceGPT's method. Additionally, our approach adaptable to parallel structural models like Phi-2, showcasing flexibility similar to that demonstrated by SliceGPT.
Matrix Operations We utilize torch. svd and torch. pinv in PyTorch for performing Sin- gular Value Decomposition (SVD) and computing the Moore-Penroo inverse on tensors of dtype FP64.
MLP Module Algorithm requires ridge leverage score parameter X. We find that the results are largely insensitive this parameter; therefore, we simply use À across all experiments.
Key-Query Module MoDeGPT reduces the feature dimension in the key-query module. In our current setup, we store head-dependent index selections, which specify the rows of cosine and sine matrices in the rotary embedding, using only (d,,) INT8 numbers, together with the reduced di- mension matrices. We've observed that this method may slow down generation speed at compression rates below 10%. A feasible modification is zeroing out columns associated with pruned indices; however, this increases the memory footprint because the matrices stored do not undergo dimension reduction. We think there is potential for improvements through enhanced engineering efforts that could better optimize the balance between memory savings and generation speed.
Value Value-Output Module Lines 3-5 in Algorithm 3 provide more computationally efficient imple- mentation of the SVD of C' Wv. Wo.j. Since Wv.,i and Wo,; are thin matrices, applying SVD directly their product incurs O(d)) complexity, while applying SVD to them sequentially incurs only Odd x (dn/Hl )) computations.
Global Sparsity Allocation To allocate global sparsity, first calculate the BI scores with single forward pass on the calibration set. We then set the sparsity according to a chosen temperature detailed in Section 3.3. A high leads to very uniform allocation, while low value introduces excessive sparsity in some layers. Empirically, we find that a simple rule of thumb is to choose a temperature that results in maximal layer sparsity around 80%.
Throughput Benchmark We use the official SliceGPT (Ashkboos et al., 2024) codebase to benchmark the throughput of all methods, with both sequence length and batch size set to 256 and utilizing KVCache.
B.3 ADDITIONAL GENERATION AND ZERO-SHOT EXPERIMENTS
Generation Performance In Table 10, we compare the perplexity of compressed OPT and LLAMA-2 7B models on WikiTexxt22 with other baselines that do not use gradient informa- tion. The rightmost column indicates their computational complexity per layer. We observe that MoDeGPT performs the best among all structured compression methods, and the 30-40% com- pressed MoDeGPT models outperform the 2:4 SparseGPT. Notably, our method shows better per- formance in LLAMA than in OPT models. We suspect this is due to the higher nonlinearity, such as RoPE and Gated MLP, adopted in LLAMA, and that our method favors more nonlinear structures. This table also shows that compression is effective on small language models.
Zero-Shot Task Performance In Table 11, we report the zero-shot performance of LLAMA-2 7B, calibrated with WikiText-2 and the Alpaca dataset, across various compression rates. We observe that MoDeGPT outperforms LLM Surgeon as the compression rate increases, and the benefits us- ing the Alpaca dataset also grow with higher compression rates. Notably, while ShortGPT performs
23

---

Published as a conference paper at ICLR 2025
Table 10: Perplexities of none madient-basee structured compression methods WikiText-2.
OPT
LLAMA-2
Method
Compression
Complexity
125M
1.3B
2.7B
6.7B
Dense
0%
27.65
14.62
12.47
10.86
5.12
Spure GPT 2:4 (Frantar & Alistarh, 2023)
50%
45.07
29.61
14.90
13.00
8.69
Of
767.2
894.4
1229
3464
861.76
Magnitude
20%
4685
1278
2788
16747
821.34
(f
30%
17970
3098
9255
17312
9623
10%
36.29
68.36
20.82
357.61
n/a
SVD
20%
55.48
1023.49
50.01
2387.39
u/a
O
30%
173.77
8851.45
707.17
9448.12
52719
33.3
20.76
17.69
27:2
14259
OBD (LeCun al., 1989)
20%
94.14
1392
3236
7570
15630
O diass))
30%
545.6
2147
7233
7628
21386
10%
34.48
16.58
13.86
11.6
6.46
20%
42.87
19.15
15.86
12.62
7.68
SliceGPT (Ashkboos al. 2024)
30%
59.87
23.87
19.91
14.19
10.47
(chasten)
40%
102.41
36.2
30.77
17.99
15.19
50%
185.52
66.
56.99
26.72
24.82
28.06
15.03
12.78
11.17
20%
29.62
15.98
13.56
11.79
6.16
MoDeGPT (ours)
30%
33.27
17.91
14.71
12.67
Ofdiiteen)
40%
38.37
21.92
17:43
14.79
8.41
50%
51.81
32.67
24.75
20.39
11.88
Table 11: Downstream zero-shot task performance of LLAMA-2 7B calibrated with 128 samples from Wiki- Tex12.
Compression
Method
ARC-e
ARC-e
PIQA
WinoGrande
HellaSwag
Average
74.58%
46.25%
(Men et al., 2024)
58.33%
38.05%
72.58%
65.51
SliceGPT (Ashkboos al., 2024)
51.47%
31.06%
64.25%
62.74
49.78%
20%
surgeon (van der
al., 2023)
71.36%
41.89%
27.09%
MoDeGPT (ours)
42.06%
74.05%
68.02%
69.05%
64.46%
MlDeGGPT-Alpacs (ours)
71.71%
41.89%
76.22%
69%
65.52%
al., 2024)
48.65%
32.85%
64.33%
56%
53.25%
SliceGPT (Ashkboos 2024)
29.27%
30%
surgeon (van der Ouderaa al., 2023)
36.69%
23.56%
61.09%
60.72%
MoDeGPT (ours)
63.26%
38.73%
70,40%
67.32%
63. 63.26%
60.78%
MoDeGPT-Alpace
65.49%
39.16%
73.34%
66.22%
65.90%
62 02.02%
SborGPT et al., 2024)
41.16%
29.94%
60.12%
60.46%
47.07%
SliceGPT (Ashkboos 2024)
36.49%
24.57%
54,90%
40%
LLM surgeon (van der Ouderaa al., 2023)
52.31%
30.29%
69.26%
54.38%
48.0 18.04%%
50.86%
MoDeGPT (ours)
49.45%
30.03%
64,96%
61.96%
53.01%
$1.88%
MoDeGPT-Alpacu (ours)
59.76%
34.73%
70.35%
64.40%
58.633%
57.58%
Table 12: Downstream zero-shot task performance of LLAMA-2 13B calibrated with 128 samples from Wiki- Tex12.
Method
Compression
ARC-e
ARC-e
PIQA
WinoGrande
HellaSwag
Average
Dense
77.48%
49.23%
80.47%
72.22%
79.39%
71.76%
20%
55.81%
35.84%
65.83%
67.17%
53.58%
55.6599
SticeGPT (Ashkboos et ul., 2024)
30%
45.96%
30,804;
59.63%
61 80%
44.09%
48.46%
40%
38.59%
27.05%
55.98
56.51
37.15%
43.06%
20%
74.07%
16%
74.53%
70.32%
68. 96.9
66.81%
MoDeGPT (ours)
30%
71.93%
43.60%
73.94%
71.90%
68.21%
65. .2%92%
40%
62.88%
38.40%
69.10%
67.72%
58.27%
59.27%
24

---

sensitive Published as a conference paper at ICLR 2025
Table 13: Downstream shot task performance of LLAMA-2 13B calibrated with 128 samples from Alpaca.
Method
Compression
ARC-e
ARC-e
PIQA
WinoGrande
HellaSwag
Average
Dense
0%
77.48%
49.23%
80.47%
72.22%
79.39%
71.76%
20%
69.36%
40.70%
74.97%
65.67%
61.01%
62.34%
SliceGPT (Ashkhoos et al., 2024)
30%
60.27%
36.18%
69.42%
64.09%
49.74%
55.94%
40%
48.99%
32.51%
63.17%
56.75%
39.85%
48.25%
20%
74.24%
45.90%
78.24%
72.53%
75.78%
69.34%
MoDeGPT (ours)
30%
70.24%
41.47%
77.15%
71.27%
71.84%
66.39%
40%
63.72%
38 82%
71.87%
66.30%
62.10%
60.56%
poorly in generation tasks, it significantly outperforms SliceGPT in zero-shot tasks. Both LLM Sur- geon and MoDeGPT maintain high performance in generation and zero-shot tasks, but our method requires only 3% of the computational resources compared LLM Surgeon.
We also test the performance on LLAMA-2 13B using the WikiText-2 calibration set, as shown in Table 12. Similar the 7B model, our method excels at higher compression rates (above 20%). However, at a 40% compression rate, we notice a performance drop in the HellaSwag task compared the 30% compression, likely due to inherent biases in our method. Nevertheless, with calibration from the Alpaca dataset, shown in Table 13, our method achieves high performance at 20% and 30% compression. Addressing these inherent biases and enhancing performance on the HellaSwar wag task is promising area for future research.
Table 14: Downstream zero-shot task performance of LLAMA-3 8B calibrated with 128 samples from Alpaca.
Method
Compression
Perplexity
ARC-
ARC-e
PIQA
WinoGrande
HellaSwag
Average
Dense
0%
2.98
77.89%
53.58%
80.63%
72.69%
79.16%
72.75%
25%
282.56
38.13%
31.40%
60.94%
54.22%
31.52%
43.24%
ShonGPT (Men et 2024)
30%
659,33
36.83%
30.72%
58 98%
54.62%
29.08%
42.04%
SliceGPTT (Ashkboos et al.,, 2024)
25%
3.87
58.88%
33.36%
68.55%
58.01%
49.86%
53.73%
4.52
52.02%
29.18%
54.62%
41.38%
48.41%
3.52
67.05%
75.52%
66.49%
MoDeGPT (ours)
62.75%
73.61%
In Table 14, we test our method on LLAMA-3 8B using our modified algorithm tailored for grouped query attention. As this is relatively model, could only compare results with SliceGPT and ShorGPT, which already support this model. We observe that compression has a more significant impact performance degradation compared to LLAMA-2 for all tested methods. We believe this due to denser information encoding in each parameter, making the model more sensitive to weight changes. However, MoDeGPT maintains approximately 90% performance with 25% compression. Another interesting observation is that the performance order of ShorGPT and SliceGPT is reversed for LLAMA-3 compared LLAMA-2, with ShortGPT's layer pruning strategy performing much worse. This suggests fundamental differences between the two models and indicates that LLAMA-3 may require more sophisticated compression techniques.
In Table 16, we evaluate MoDeGPT on a broader range of tasks to assess its generalizability. We compare MoDeGPT with two baseline methods: SliceGPT (a decomposition approach) and Short- GPT (a layer-pruning method), all using 30% compression of LLAMA-2 7B, with calibration per- formed on the Alpaca dataset. The top rows the table show the raw accuracies, while the bottom rows display the relative degradation compared to the original dense model.
MoDeGPT demonstrates the least degradation across all tasks, with an average drop of only 7.19%, while the other methods experience drops exceeding 17%. Notably, the degradation is generally consistent across tasks, except for Lambada and MMLU, which show more significant drops. These tasks also exhibit the largest degradations in the baseline methods, suggesting they to compression.
Lambada, in particular, exhibits extreme degradation in both SliceGPT and ShontGT (over 40% for both), making it the most challenging task to maintain accuracy after compression. In contrast, MoDeGPT shows relatively small degradation of just 16.8% almost 25% lower than the other
25

---

Published as a conference paper at ICLR 2025
Table 15: Generation samples of LLAMA-2 7B with 30% compression.
Prompt:
The universe is entretyy of space, time, matler, energy that exists.
sierpe 2018, The universe is everything that ever did or ever will exist. includes all the past, all the present, all the future. What
5 most important things you learned about life? do you describe a person words? Can you be huppy with nothing? there the meaning of universe? Who is the hest person talk when you need alvice? What the meaning of the word universe? What is the meuning universe according Godungsseite.
everything? What
ShortGPT (Mea et al., 2024)
According Wikipedin, the universe refers to everything that ever existed, continues existence, or could possibly exist. Thus it includes all the pasi, all the preseni, and all the future.
The universe contains many galaxies, stars, planets, other objects such astemidss, planetary systems. It has been estimated to contain 100 billion triliinn stars. The Sun is a star in the Milky Way galaxy. which is one of about 400 billion galaxies in the Milky Way's ss vicinity The Milky Way itself is part the Milk Way Group which includes several smaller groups of galaxies.
SliceGPT (Ashibous et al., 2024)
The universe is a vasi, dynumic, and complex structure that contains many different lypes of objects, including stars, galaxies, plancts, asteroids, and intergalactic gas clouds. It is also home to life-forms as humans, animals, plants, micmorganisms, and microscopic organisms. studied by astronomers for centuries, methods, Astronomical observations have revealed that the universe much larger than previously thought, with estimated diameter of approximately 10 billion light years. This means that the universe roughly 33 times larger than our solar system. The universe is composed of two main components: the cosmos and the cosmic void. The cosmos consists of galaxies, which are large collections of stars and other objects bound together by gravitational forces. The cosmic void, on the other hand, is a region of space hetween galaxies that is largely empty of matter.
universe and complexity have estimated using various
MoDeGPT (ouns)
Table 16: Zero-shot task performance degradation of LLAMA-2 7B, calibrated with 128 samples from the Alpaca dataset, evaluated across a broader set tasks.
Method
BoolQ
PIQA
HellaS,
WinoG.
ARC-e
ARC=
OBQA
COPA
Lamb.
MMLUmml
Average
Dense
77.68%
79.05%
76.00%
68.98%
74.58%
56.33%
44.22%
87.00%
73.86%
29.29%
66.70%
BlcetePTCAskkbboe (Asbkboes al., 2024)
61.99%
68.55%
48.09%
59.75%
39.69%
34.47%
31.40%
75,00%
21.02%
23.2118
48.08%
ShonGiP (Mon al., 2024)
62. 17%
64 48%
56. 15%
64.399
70%
32.59%
32,80%
79.00%
29.03%
24.11%
40. 34%
MoDeGGPT (ours)
69.76%
73.34%
65.90%
66.22%
65.49%
39.16%
39.00%
87.00%
57.07%
32.14%
59.51
A SheccIPT
-15.0%%
-10.50%
-27.31%
-17.80%
-11,86%
-12.00%
-16.08%
ShonGPT
-15.51%
-14.57%
-19885%
-25 88%
-13.74%
-11.40%
8.00%
18%
-17.66
McDeGPT (ouns)
-7.92%
-5.71%
-7.1%%
-5.20%
methods. This hints that MoDeGPT is better at preserving important information, which is crucial for excelling on more difficult tasks like Lambada.
Finally, we compare the generation quality using samples from the three methods' generations for 30% compressed ILAAM-22 7B, as shown in Table 15. ShorGGPT produces the lowest quality gen- eration, while both SliceGPT and MoDeGPT generate high-quality responses, with MoDeGPT pro- viding more detailed responses than SliceGPT.
B.4 ADDITIONAL BASELINE COMPARISONS: FEATURE-MIME AND SVD APPROACIESS
In Table 17 and 18, we compare our method against feature-mimic and SVD-based approaches, respectively. In the former case, we observe that alternative methods generally underperform com- pared state-of-the-ar gradient-based techniques like LLM Surgeon, while our approach achieves comparable or even superior results. In the latter comparison, our advantage is even more pro- nounced, which we attribute our more refined decomposition algorithms, tailored specifically to different components of the transformer architecture based on their levels of nonlinearity, rather than relying solely on SVD-based decompositions.
B.5 ADDITIONAL BASELINE COMPARISONS: UNSTRUCTURREDDSEM-SSRRRRRT COMPRESSION
In Table 19, We compare MoDeGPT to the ofate-of-the-art non-structured methods, Wanda, SparseGPT, and ZeroPruner. These methods generally outperform MoDeGPT at 50% compres-
26

---

Published as a conference paper at ICLR 2025
Table 17: Comparisons of feature-mintt mimic based methodd for 30% compression of LLAMA-2 7B and 13B mod-
Model
Method
ARC-e
ARC-c
PIQA
WinoG.
HellaS.
BoolQ
OBQA
Average.
Dense
74.58
46.25
79.11
69.06
75.99
77.74
44.20
66,70
LLM Pruner (Ma al., 2023)
61.41
33.96
71.93
58.72
59.49
61.41
36.60
53,52
LLAMA-2 7B
FLAP (An al.. 2024)
60.65
34.47
72.74
64.01
63.80
66.88
36,40
56,99
Bolaco (5 x 4) (Ji al., 2024)
65.87
34.30
71.27
64.48
57.85
73.85
37.80
57.92
MoDeGIIT (ours)
65.49
39.16
73.34
66.22
65.90
69.76
39,00
59.83
Dense
77.48
49.23
80.47
72.22
79.39
80,52
45.20
69.22
LLM Pruner (Ma al., 2023)
65,45
40.36
75.90
60,22
67.90
62.43
44,60
59.55
LLAMA-2 13B
FLAP (An al., 2024)
67.38
38.23
74.81
67.48
70.29
65.54
40.00
60.53
Bolaco (5 x 4) (Ji al., 2024)
71.76
40, 10
74.16
69.06
66.66
75.63
41.60
62.71
MoDeGPT (ours)
70.24
41.47
77.15
71.27
71.84
73.7
41.00
63.81
Table 18: Comparisons with SVD-based methods in LLAMA-1 1 7B.
Compress, Kate
Method
Wikflest-2
FIB 1
ARC=
ARC-e
PIQA
WinoG.
Hells
MathQA
OBQA
Axg.
0%
Dense
5.68
8.35
73
42
79
70
50
27
34
54
FWSVD (Hsu 2022)
1727
2152
23
50
26
21
15
32
20%
SVD-LLM (Wang of ol., 2024)
ASVD (uuan et al., 2023)
11.14
7.94
16.22
16.55
58
27
29
38
64
43
41
24
24
22
25
43
44
MoDeGPT (ours)
6.53
39.17
70
36
69
30
26
31
51
et 2022)
18156
21
40%
SVD-LLM (Wang al., 2024)
ASVD (Yuun et al., 2023)
13.11
1407
63.75
3292
42
28
22
25
60
48
58
26
33
21
19
19
12
37
30
(ours)
60.55
sion rate. However, MoDeGPT with 40% compression achieves a significantly better perplexity (8.41 versus 10.17).
The observation suggests that with small concession on compression rate, our struc- tured compression can be on par with the semi-structured method that requires spe- cial GPU support for efficient inference.
Table Comparisons with remi-structured pruning.
Method
Structure
40%
50%
SparseGPT (2:4)
Semi-structured
10.17
Wanda (2:4)
Semi-structured
11.02
ZeroPruner (2:4)
Semi-structured
10.52
B.6
RECOVERY FINE-TUNING
MoDeGPT (ours)
Structured
8.41
11.88
Table 20: Compression and recovery fine-tuning for LLAMA-2 7B using Alpaca dataset
Method
Compress.
ARC-
ARC-c
PIQA
WinoGrande
HellaSwag
Average
Denso
0%
74.58%
16.25%
79.11%
69.06%
75.99%
69.00%
20%
44.20% (.315)
76.99% (1 0.77%)
(1 1.58%)
0.36%)
65.36%
0.16%)
McDeGPT RCT-MLP
30%
61.94%
() 0.55%)
42.15% (1 2.99%)
73.83% (1 0.19%)
66.54% ( 0.32%)
67.08% (? 1.18)
62.91% (1 0.89%)
40%
59.26% () 0.50%)
37.12% (1 2.39%)
72,09% (1 1.74%)
64.33% (1 0.07%)
60.82% (? 199)
58.72%
1.14%)
McDeGPT RCT-ALL
20%
() 42.925 1.03%)
77.209% 0.98%)
(1
64.99% (.533))))
30%
63.35 63.38% () 2,11
41.47% (1 2.31%)
74.81% (1 1/479)
66.06/ (1 0.16%)
65.644% (. 0.58%)
62,27% (1 0.25%)
40%
58.425% () 1,349)
38.2356 3,5095)
72.0395 (1 1.68%)
(1 0.799)
59.55% (" 0.925)
58.34% 0.769))))
While MoDeGPT does not require recovery fine-tuning, in this section, we explore how RFT can further enhance performance. In Table 20, we present recovery fine-tuning results for our method on LLAMA-2 7B, following the same tuning setting as SliceGPT (Ashkboos al., 2024). We use calibration set of 128 random samples, each 2048 in length, from the Alpaca dataset, and a recovery fine-tuning set 8000 samples, each 1024 in length, employing LoRA (Hu et al., 2021). We use SliceGPT's 's hyperparameters for LoRA, except for the learning rate, which is set to 5 10-5. 5, The other primary hyperparameters used are lora ara__lppa and batch, size 3. We evaluate two scenarios: 1) fine-tuning all linear matrices, and 2) tuning only the MLP.
0.05, 10, lora_r = 32, lora cora_dropout
The green and red indicators in the table denote performance increases or decreases relative to the compressed model before fine-tuning. Notably, tuning exclusively within the MLP consistently yields better performance than tuning all parameters. Since we followed the same tuning setting as
27

---

Published as a conference paper at ICLR 2025
SliceGPT (Ashkboos et al., 2024) for fair comparison, it is likely that better configurations exist for our method, potentially enhancing performance further. Another key observation is that despite fine-tuning using 40 times more data than calibration and employing backpropagation, MoDeGGT without RFT achieves very similar performance. The percentage difference is minimal, suggesting that using local reconstruction compression technique.
the objective is effective and efficient method with our
The table demonstrates that fine-tuning can slightly improve performance for higher compression rates, with the most significant increase observed in the ARC-c task. Evaluating the full benefits of fine-tuning remains a subject for future research.
B.7 EXPERIMENTS WITH EQUAL COMPUTATIONAL BUDGETS
Table 21: Compression comparisons with approximately cqual computational budgets.
(Compress/ (une-tune)
PIQA
WinoG.
Average.
SExeGPT
26m 4hh5mm
2.59 (3.52)
56.82 (56.69)
38.48 (34.7)
71822 (68.55)
59.83 (59.75)
59.30 (18.69)
57.26 (53.63)
SLEB
9m 44h5mm
(4.36)
52.36 (52.36)
34.04 (31.91)
71.00 (69.58)
59.98
60.16 28)
55.51 (54.06)
MoDeGPT
(h09m/ 31m
(3.08)
67.42 (65.49)
40.96 (39.16)
74.10 (73.34)
65.98 (65.49)
66.57 (65.90)
63.01 (62.02)
We study the combined effect of compression and recovery fine-tuning for different approaches with equal computational cost, as shown in Table 21 In this experiment, we compress LLAMA-2 7B with 30% compression rate a single A100 GPU, The model is first compressed using 128 samples from the Alpaca dataset for calibration, followed by fine-tuning with LoRA on 5k Alpaca lora lora_alpple alpha = 10, lora_r 32,
samples. For fair comparisons, we fix the hyperparameters. and lora_ddaar derrruudroutttttt = 0.05. We compare MoDeGPT against SliceGPT and SLEB, which serve as baselines for decomposition-basee and layer-pruning-basee approaches, respectively.
Since the methods vary in compression time, we adjust the fine-tuning epochs to equalize the total time spent across methods. The table reports the time spent in each phase for different methods. Notably, MoDeGPT has the longest compression time and is therefore fine-tuned for only one epoch. The table presents zero-shot accuracies both before and after fine-tuning (after/before).
MoDeGPT achieves the highest zero-shot performance across all tasks, excluding perplexity, both before and after fine-tuning, with its performance advantage primarily arising from the compres- sion phase. The superior perplexity but lower zero-shot performance of SliceGPT compared to MoDeGPT underscores the pivotal role of the compression stage, suggesting that an excessive com- putational focus on fine-tuning may lead to overfitting.
Lastly, SLEB, despite having the longest fine-tuning time, exhibits smaller improvements than SliceGPT in zero-shot performances. further emphasizing the pivotal role of the compression phase determining the final model's performance. Moreover, MoDeGPT outperforms the baselines even without fine-tuning, demonstrating its effectiveness during the compression stage.
B.8 COMBINATION MODEGPT AND SLICEGPT
MoDeGPT is orthogonal SliceGPT as it reduces dimensions from different sides of weight ma- trix. Figures I (c) and (d) provide an illustrative comparison. Combining SliceGPT with MoDeGGT seems be natural extension. To demonstrate their compatibility, we experimented with various configurations as shown in Table 22. The numbers x-y-z in the leftmost column indicate 2% slicing rate of SliceGPT, and 5 and 2% compression rates of MoDeGPT in MLP and MHA modules, respectively. The two rightmost columns test the use of sparsity allocation in the MLP and/or MHA modules.
Notably, our tests show that applying sparsity allocation with SliceGPT barely improves perfor- mance, consistent with the findings in the SliceGPT paper (Ashkboos et al., 2024). Therefore, we do not use sparsity allocation for slicing. Compared to the results in Table 3, the combination of SliceGPT and MoDeGPT does not improve perplexity over pure MoDeGPT. We attribute this to two points: 1. the significant overhead induced by slicing: to achieve target compression rate, the
28

---

Published as a conference paper at ICLR 2025
Table 22: Perplexity performance of SliceGPT MoDeGPT on LLAMA-2 7B
Slice Slice-MLP-MHi MLP-MHA
Compression Rate
WikiText2 Perplexity
MLP Sparsity
MHA Sparsity
(%-6-%
Allocation
Allocation
Dense
0%
5.12
20-20-0
19.65%
7.38
/
x
20-20-0
19.65%
7.33
X
x
25-25-0
27.38%
8.42
x
30-30-0
34.93%
9.99
/
x
20-25-0
22.25%
7.70
X
x
15-30-0
11.77%
7.27
X
x
10-30-0
9.03%
6.83
x
10-25-25
28.00%
7.31
x
10-30-25
30.91%.
7.78
x
20-20-20
29.18%
8.00
X
x
model must slice higher rate. 2. the slicing and compression ratio might not be optimal, and might changes from layer to layer.
Although did not make exhaustive search, we believe there is efficient sparsity allocation for slicing, and better tuning of the slicing and compression ratios could enhance the performance of the combined method. We leave this topic for futurre research.
B.9 COMPRESSION TIME AND MEMORY CONSUMPTION
Table 23: Compression computations for calibration set of size 128 in WikiTex12.
Model
MoDeGPT
SliceGPT Ashkboos et (2024)
LI surgeon
al. (2023)
Time
GPUs
Time
Time
GPUs
LLAMA-2 7B
4h09m
1xA100 80GB
0h26m
1xA100 80GB
17h088m
4xH100 80GB
LLAMA-2 13B
8h26m
1xA100 80GB
0h45m
1xA100 S0GB
1d9h26m
8xH100
Table 24: Memory consumption and compute time of 30% compression for blocks in transformer layers tested single A100 80GB GPU.
LLAMA-7B (13.81 GiB)
LLAMA-13B (25.92 GiB)
Block
Peak Memory (GiB)
GPU hours
Peak Memory (GiB)
GPU hours
MHA
15.54 (+11.5%)
2h52m
28.60 (+9.4%)
5h04m
MLP
23.33 (+68.9%)
1h13m
41.40 (+54.1%)
3h22m
In Table 23, compare the compression times of MoDeGPT, SliceGPT, and LLM Surgeon. Since MoDeGPT and SliceGPT do not leverage gradients, they can compress a model size 13B us- ing single GPU. From previous tables, observe that while compute time is longer than
29

---

Published as a conference paper at ICLR 2025
Table 25: Downstream zero-shot task performance of 30% MoDeGPT LLAMA-2 7B for varying global rank temperature.
Method
Perplexity
ARC-e
ARC-c
PIQA
WinoGrande
HellaSwag
Average
Dense
5.12
74.58%
46.25%
79.11%
69.06%
75.99%
69.00%
0.075
7.44
59.72%
37.29%
68.50%
65.90%
61.55%
58.59%
0.1
7.46
63.43%
39.42%
70.78%
65.59%
63.24%
60.49%
0.5
7.03
56.14%
32.34%
67.68%
64.88%
58.01%
55.81%
MoDeGPT (ours)
7.25
53.20%
31.06%
66.16%
64.17%
56.66%
54.25%
7.35
53.62%
31.06%
65.83%
63.14%
55.98%
53.93%
9.06
52.36%
30.80%
65. 18%
63.69%
55.31%
53.47%
SliceGPT, MoDeGPT achieves significantly better performance. Conversely, our computation time is considerably shorter than LLM Surgeon, yet we achieve comparable performance. Even when equating H100 to A100, our method can save up to 97% of computations. the peak GPU memory usage when compressing LLAMA-2 7B and 13B models on single A100 GPU. The primary source of additional memory overhead, beyond the model itself, is the storage of intermediate activations required for correlation estimation in the MLP. The table shows that this overhead ranges from approximately 50% to 70%. However, for the 13B model, the peak memory usage remains under 50% of the total GPU memory capacity.
Table 24, we report
B.10 GLOBAL SPARSITY ALLOCATION
In Table 25, we report the perplexity and zero-shot performance as we vary the temperature pa- rameter in the global sparsity allocation. Initially, the uniform strategy, corresponding an infinite temperature, performs significantly worse than our sparsity allocation strategy.
0.5
100%
50%
0.4
50%
50%
70%
Spario0
70%
40%
60%
Importnes
50%
50%
40%-
02:
30%
0.1-
10%
10%
0%
0%
Layer ID
Figure 8: Dynamic sparsity allocation across layers for LLAMA-2 7B.
For 0.075. this results in extreme sparsity, as shown in Figure 8, and performance begins to drop. For & 2 1, the allocation becomes too similar to the uniform strategy. In practice, we find that the e value that yields minimum layer allocation around 20% performs exceptionally well. In Figure 8, left and middle panels, we also observe how the allocation shape for different values corresponds to the importance score in the left figure. For LLMs, both OPT and LLAMA show that the first and last few layers have significantly higher importance and, therefore, should allocate less sparsity, as depicted in the figure. On the right of Figure we also show the allocation for different compression levels. The shape remains similar across different levels using the same For higher compression rates, the maximum layer sparsity also increases, suggesting that we should increase to avoid extreme sparsity. We report the ranks of the QKV projection matrices across various layers of compressed LLAMA-2 7B and 70B models, as determined by the global sparsity allocation used in this study (equation 11), with their distributions visualized in Figure 10.
The ranks were computed using equation 11 with 128 samples from WikiText-2 and values of 0.1 and 0.02 for the 7B and 70B models, respectively. These values were selected to ensure the maximum layer sparsity remains around 70-80%, as a 90% sparsity level is often too extreme, based on experience from experiments.
Interestingly. we found that the rank distributions exhibit similar shapes across the models, suggest- ing deep connection between the allocation strategy and the L.LA MA-2 family architectures.
30

---

Published as a conference paper at ICLR 2025
Table 26: Layer ranks for various models.
Model
Layer Rank
LLAMA-2 7B
3989, 3886, 3813, 3889, 3750, 3616, 3598, 3612
3625, 3393, 3546, 3660, 3654, 3568, 3575, 3544
3453, 3241, 2997, 2703, 2413, 1741, 1620, 1217
1129, 1254, 1054, 741, 1203, 1363, 2640, 4060
LLAMA-2 70B
8192, 8183, 8186, 8169, 8143, 8103, 8130, 8088
8134, 7983, 7908, 7873, 7957, 8018, 7932, 7968
7772, 8000, 7858, 7784, 7486, 7419, 7079, 7016
7090, 7596, 7214, 6784, 6620, 6556, 6204, 6384
6366, 6762, 6719, 6411, 6472, 6356, 6651, 6918
7138, 6839, 6872, 6112, 6620, 5467, 5042, 5328
4402, 3940, 3563, 3745, 3632, 3076, 2814, 3051
2814, 2622, 3025, 2395, 2189, 2128, 2158, 2128
2248. 2037, 2760, 2947, 2453, 3051, 3152, 3609
3446, 3540, 4148, 4694, 5548, 5994, 7355, 8187
Global Allocation (Ours)
100%
100%
80%
90%
80%
80%
70%
70%
80%
60%
50%
50%
40%
40%
30%
20%
20%
10%
0%
15
20
25
Figure 9: Layer sparsity distribution comparisons.
Table 27:
Global sparsity allocation comparisons.
Sparsity Mean
Sparsity
Perplexity
PIQA
HellaS,
ARC-E
ARC-C
Average
$2.36
53.47
Spursity Allocation (Ours)
26.72%
Yin (2023)
4.46%
59.12
56.64
also compare our global allocation strategy in equation with a state-of-the-art alternative, OWL (Yin et al., 2023), shown in Table 27. In this experiment, compress LLAMA-2 7B using MoDeGPT with different global sparsity strategies. Despite our method having a higher perplexity, consistently achieves better zero-shot performance across all reported tasks. Figure 9 visualizes the layer sparsity distributions of the two approaches. Unlike OWL, our distribution exhibits much greater heterogeneity across layers, showing less sparsity in the first and last layers. This figure suggests that heterogeneity may play crucial role in structured compression.
B.11 REFINED SPARSITY ALLOCATION: DIFFERENTIATING MLP AND MHA BLOCKS WITH FINER-GRAINED SCORES
In this subsection, we refined our global sparsity allocation strategy by introducing different sparsity levels for the MLP and MHA blocks within each transformer layer. A similar approach layer pruning, which employs distinct sparsity levels for MLP and MHA, has been explored by Finercut (Zhang al., 2024). This strategic refinement has significantly improved both compression accuracy and inference throughput, particularly enhancing inference speed. Notably, in our 30% compression experiments LLAMA-2 7B, illustrated in Table 28, achieved the highest throughput among
31

---

Published as a conference paper at ICLR 2025
Global sparisly allocation in Liama2 7B
Global sparisty allocalion in Llama2 70B
8000
3500
2000
5000
2000
4000
3000
1000
10
15
20
25
30
0
10
20
30
40
50
60
70
Layer
Layer
Figure 10: Layer Ranks of LLAMA-2 7B and 70B.
all baselines, surpassing even dedicated layer pruning strategies, while maintaining exceptional ac- curacy.
Instead of computing single score per layer. we now calculate distinct scores- one for MLP and another for MHA-appllee the correlation methodology outlined in Section 3.3. This dual- score system enables a more nuanced sparsity allocation that aligns better with the unique com- putational and structural characteristics of each block type, thereby optimizing performance with- out incurring significant computational overhead. The updated global sparsity allocation Equation equation 10 is follows:
Wy (5/(1 (-) + (HH+))(())).
max PIL
d=
jEmlp.mha
such that
1
Σ
Σ
wj = Oavg 0 0 050551 S
(42)
L miipp + Wmhn)
<= jEmlp.mha
where of and of represent the sparsity and score for the j-th block in layer i respectively, and the weights Wmlp 2, Wima are applied to preserve the average sparsity, consistent with the parameter size ratio in transformer blocks. The solution has a similar closed-form expression:
L(Wmip + Wmhn 2avg Softmax s/e).
(43)
Importantly, these updates come with minimal computational overhead. Although we now calcu- late scores per layer (instead of one), the computational is negligible as score calculation remains lightweight and does not increase compression time.
Table 28: Enhanced Compression through Nonuniform Sparsity in MLP and MHA Blocks.
Method
MLP Sparsity Mean
MILA Sporsity Mean
Throughput (tokeni))
PIQA
HellaS.
WinoG.
ARC-e
ARC-c
AAvrrage
SLEB Song al. (2024)
30.0%
30.0%
2539.39 (1.49x)
69.58
58.28
58.17
$2.36
31.91
$4.06
SheeUPT Ashiboos et al. (2024)
30.0%
30.0%
1515.67 (1,070)
68.35
48.69
99.75
56.69
34.47
53.03
MaDeGRT (untfoe module sporsity
90.0%
30,07
2490 15 (1 (1,66)
73.34
68.90
66.22
65.49
39.16
62.00
MoDeGPT mncuniioon module sparsity)
26.5%
36.4%
2722.96 (1.60x)
73.78
65.14
68.03
66.79
38.40
62.43
B. 12
ABLATION STUDY ON COMPRESSION EACH MODULE
Impact of Module-Wiss Compression on Perplexity. Table 29 presents the perplexity changes the LLAMA-2 7B model when compressing each module individually. The rightmost column shows the normalized slope of perplexity change relative to the parameter size in each module. The results reveal that the MLP module has the most significant impact on overall performance, likely due to its containing 66% of the model's parameters. On the other hand, the slope indicates that the compression algorithms for Type I and Type III modules perform comparably, while Type II performs the worst. This finding aligns with our theoretical results, which suggest that the recon- struction bounds are weakest in Type III. From a decomposition perspective, the CR approximation the most coarse-grained, leading the least effective compression outcomes.
32

---

Published as a conference paper at ICLR 2025
Table 29: Perplexity of compressed LLAMA-2 7B in each module.
Compression Rate
0%
20%
30%
40%
50%
Normalized Slope
Module
Type MLP
5.12
5.34
5.68
6.71
7.12
8.24
0.094
Type II: Query, Key
5.12
5.14
5.23
5.43
5,58
6.33
0.121
Type Value, Output
5.12
5.16
5.24
5.37
5.62
5.92
0.095
Table 31: Heterogeneous sparsity allocation in modules.
Sparsity (MLP, MHA)
Perplexity
ARC-e
ARC-c
PIQA
WinoGrande
HellaSwag
Average
30%, 30%
7.51
65.49
39.16%
73.34%
66.22%
65.90 %
62.02%
35%, 20%
7.79
60.52%
38.48%
68.82%
65.98%
61.34%
59.03%
25%, 40%
7.14
57.03%
35.15%
70.89%
65.27%
61.63%
57.99%
Impact of Module-Wise Compression on Throughput. Table 30 presents the throughputs for the 30% compressed LLAMA-2 7B across different modules.
Table 30: Module-Wise Throughputs of 30% Compressed LLAMA-2 7B
Module
Throughputs (tokens/s)
Type MLP
1585
The results indicate that the compression yields similar speedups for both Type-1l and Type-III modules. This sharp differ- ence in speedups highlights the potential for uniform compression across modules, which we leave as a direction for future research.
Type II: Query, Key
2136
Type III: Value, Output
2121
Heterogeneous Sparsity Across Modules In our work, we apply nonuniform sparsity across lay- ers while maintaining uniform sparsity across modules within the same layer. To investigate whether heterogeneity in module sparsity enhance performance, ing LLAA A-2 7B by 30%, with varying sparsity levels in the MI.P and MHA blocks. The sparsity levels were adjusted ensure the average compression rate remained at 30%.
conducted experiment compress-
We tested three configurations: equal sparsity for MLP and MHA, higher sparsity in MLP, and higher sparsity in MHA. The results are presented in Table 31 From the table, we observe that while lower sparsity in MLP yields the best perplexity. it results in the worst zero-shot performance among the three configurations. Conversely, uniform sparsity across modules outperforms the other configurations in all tasks, while high and low MLP sparsity each demonstrate strengths in specific tasks compared to one another. These findings underscore the sensitivity of compression perfor- mance to variations in module sparsity, suggesting that more sophisticated allocation method may be necessary to surpass the performance of uniform allocation.
B. SCALABILITY TO LARGER MODELS
While our work is tested on a single GPU, it can be extended to multi-GPU setups to compress larger models, such as those with 70B parameters. To apply our method larger models, the model must fit within the GPU memory to perform the forward pass. As shown in Table 24. memory utilization is less than twice the model size, so approximately double the model's size in GPU memory is expected for running intensive part is the compression of the value- palue-output module, as highlighted in Table Since the computational complexity of this module scales cubically with the hidden dimension (due to the SVD in the value-output compression) and is proportional to the number of layers being compressed, the time required compress 70B model using multi-GPts can be estimated using the following
method. In our compression process, the most computationally
33

---

Published as a conference paper at ICLR 2025
formula:
Compute Time (70B)
Compute Time (7B) hidden dim(70B) 3
layer num(70B)
hidden dim(7B)
layer num(7B)
= hours (8192/ 4096) x (80/32) = 80 hours
For sanity check, we applied the same formula to estimate the compression time for 13B model and obtained an estimate of 9 hours, which aligns closely with our empirical result of 8 hours and 26 minutes, shown in Table 7.
B.14 HIGH COMPRESSION RATE EXPERIMENTS
Table 32: Perplexity of LLAMA-2 7B Across 10% to 80% Compressions
Compression Rate
0%
10%
20%
30%
40%
50%
60%
70%
80%
Perplexity
5.12
5.48
6.16
7.51
8.41
11.88
26.59
84.22
245.84
We analyzed the perplexity LLAMA-2 7B at high compression rates, using 128 samples from WikiText2 for calibration. We observed significant breakdown point at 50% compression, where the perplexity increased sharply from 41% to 123%. This indicates the compression limit of our method.
B.15 SENSITIVITY ANALYSIS OF DFFERENT CALIBRATION SETS
In Table 33, we evaluate in-domain and out-of-domain perplexity using different calibration sets: WikiText2, PTB (Marcus et al., 1993), and Alpaca. Our results indicate that perplexity is minimized when the model is calibrated with the same dataset as the test set. Notably, when calibrated with different datasets, the results on Alpaca demonstrate the most consistent performance with the least variance, while PTB shows the highest variance. Nevertheless, calibration with PTB provides the most robust results across all three datasets.
B.16 ADDITIONAL SPEEDUP EXPERIMENTS
SliceGPT
MoDeGPT
Dense
Throughput (Tokensis)
2400
2000
1600
1200
800
400
batch size 64
batch size 128
batch size 256
batch size 512
Figure 11: Throughput benchmarks of compressed LLAMA-2 7B single A100 80GB GPU.
Table 34 reports the throughputs and latency on fast and slow parallel computing environments using NVIDIA A100 and Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50 GHz with 20 cores. The results indicate that the reduction computational complexity proportional to the compression percentage. While the speedup for single-batch inference is comparable that of the original model, we observe significant improvements in speedup for multi-batch inference on the GPU and on the CPU. Therefore, method performs optimally when the parallel computing capabilities of the environment are fully utilized.
In Figure 11, explored various batch sizes, comparing the throughput of 30% compressed MoDeGPT with 30% sliced SliceGPT (Ashkboos et al., 2024) and the dense model. We found that throughput surpassed that of the dense model for batch sizes over 64. Particularly, at batch
34

---

Published as a conference paper at ICLR 2025
Table 33: Perplexity results under different calibration datasets.
Test Set
WikiText2
PTB 4
Alpaca
Calibration Set
WikiText2
6.16
27.69 (+22%)
3.12 (+11%)
PTB
6.99 (+13%)
22.75
3.14 (+12%)
Alpaca
7.64 (+24%)
40.71 (+79%)
2.80
Table 34: Inference speed and computational complexity of the pruned LLAMA-2 7B model.
Method
Parameter
Memory (GiB)
Compute Complexity
Latency CPU (s/token)
Latency GPU
256-Batch Throughputs
(GMACs)
(shtoken)
(tokens/s)
Dense
12.92
425.12 (1.00x)
32.41 (1.00x)
(1.00x)
1700 (1.00x)
20% SliceGPT
5.45
10.45
339.04 (0.80x)
26.46 (0.82x)
0.037 (1.06x)
1802 (1.06x)
20% MoDeGPT
5.44
10.43
339 34 (0.80x)
22.66 (0.70x)
0.034 (0.97x)
2168 (1.28x)
30% SliceGPT
4.73
9.07
298.36 (0.70x)
25.28 (0.78x)
0.037 (1.06x)
1830 (1.08x)
30% MoDeGGP
4.79
9.07
297.91 (0.70%)
19.20 (0.59x)
0.034 (0.97
2521 (1.48x)
40% SliceGPT
4.11
7.88
262.12 (0.62x)
22.68 (0.70x)
0.037 (1.06x)
1839 (1.08
40% MoDeGIT
4.14
7.94
256.34 (0.60x)
18.57 (0.57x)
0.036 (1.03x)
2568 (1.51x)
2600
Dense
x1:58
Throughput (Tokens/s)
MoDeGPT SliceGPT
x1.50
2400
x1.46
2200
SVD
x1.29
2000
x1.07
x1.09
x1.10
1800
x1.03
1600
x1.00
x0.96
x0.91
1400
x0.72
x0.79
x0.79
1200
0%
10%
20%
30%
40%
50%
Figure 12: Speedup vs. compression.
sizes exceeding 256, MoDeGPT's throughput 1.46 times greater than the dense model, while SliceGPT only achieved 1.07 times the throughput. MoDeGPT's increased throughput stems from its reduced matrix size and avoidance of extra adapters in residual paths.
Finally, we benchmark throughput. We set the sequence length to 256 and recorded the average generation time of LLAMA-2 7B on single A100 GPU with batch size 256. In Figure 12, SVD exhibits lower throughput than the uncompressed model due to the doubled amount of matrices decomposed form which makes computation less parallelizable. SliceGPT, while achieving greater throughput, sees less than 10% speedup, hindered by additional computations in residual paths. In contrast, MoDeGPT achieves non-trivial speedups that increase with compression rates; at 50% compression, it achieves a 58% increase in throughput, significantly surpassing both SVD and SliceGPT. However, at compression rates below 10%, throughput drops below that of the uncom- pressed model. This decrease is attributed to the implementation of the compressed Type-Il module, which needs an optimized kernel to better parallelize the computation of the pruned attention heads. We leave the implementation of such an optimized computation kernel as future work to address the corner case.
C LIMITATIONS AND BROADER IMPACTS
Intrinsic Bias Our experiments on zero-shot tasks show that MoDeGPT excels certain zero- shot tasks while underperforming in others, indicating intrinsic bias toward specific tasks. Our
35

---

Published as a conference paper at ICLR 2025
current method does not offer definitive solution to eliminate this bias. Addressing bias removal will be a critical area for future research.
Overfitting the Reconstruction Loss While MoDeGPT excels in zero-shot tasks by minimizing local reconstruction error, we noted instances where compressed models, despite achieving lower perplexity, underperformed in zero-shot tasks. This discrepancy may stem from the models overfit- ting local reconstructions to calibration data. Addressing this overfitting remains a challenge for our method.
Broader Impacts The introduction of Modular Decomposition (MoDeGPT) significantly impacts the ethical deployment and broader adoption of Large Language Models (LLMs). By minimizing computational demands, MoDeGPT enables effective deployment resource-constrainee devices, democratizing access to cutting-edge Al and potentially reducing the technological divide between large and small entities.
Additionally, MoDeGPT 's efficiency in using computational resources can decrease energy con- sumption during AI training and inference, promoting sustainable Al practices and reducing the environmental impact of large-scale computations. However, the potential for increased misuse of AI technologies, such surveillance and disinformation, highlights the need for robust governance and ethical frameworks.
Ultimately, by maintaining high accuracy while reducing model size, MoDeGPT ensures the reli- ability of AI applications in critical domains such as healthcare. The development of MoDeGPT thus promises greater Al accessibility and sustainability, but it also introduces new challenges in governance and ethical technology use.
36


Reviews:
[{"review_id": "rJ2rfXKAAO", "reviewer": "Reviewer_ZuW6", "summary": "This paper proposes a novel model compression method by applying three different matrix decomposition algorithms to three distinct types of computations within Transformers. Compared to previous model compression algorithms, this approach achieves a significant improvement in performance.", "questions": "1. Why does Table 3 include only 50% compression results for models like SparseGPT but lack results for 40% compression? Why is a 40% compression result of MoDeGPT compared to a 50% compression result of SparseGPT?\n2. I am curious why magnitude-based and SVD-based compression methods seem to cause model collapse in Table 1, performing worse than random compression (Uniform).\n3. The authors applied different compression rates to different layers, but are the compression rates for the three types of computations identical? Based on the analysis in Figure 4, it might be better to allocate a higher compression rate for Attention computations.\n4. Why is MoDeGPT more efficient than the baseline at the same compression rate (Figure 3)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose the interesting idea of using three different matrix decomposition algorithms to compress computations in both MLP and Attention.\n2. Experimental results demonstrate that the proposed method offers advantages in terms of both performance and efficiency compared to prior pruning and matrix decomposition algorithms.\n3. The Appendix includes additional methods and experiments related to group-query attention.", "weaknesses": "1. The authors suggest using three different types of matrix decompositions for three different types of computations within Transformers, but they do not provide motivation for this choice. For example, why is CR decomposition more suitable for Type-2 computation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vrs3l0rNI0", "reviewer": "Reviewer_Z3HT", "summary": "This paper proposes MoDeGPT, which compresses transformers by applying structure decompositions on operations that span *two* weight matrices. The parameter subgroups targeted are the MLP weights, key and query projections, and value and attention output projections. Experimental results show that MoDeGPT is the best no-gradient structured method, and also comparable to the best structured and gradient-based method.", "questions": "1. In Table 3, is the main claim that although semi-structured methods may outperform MoDeGPT, they are held back by custom GPU support which hinders research velocity?\n2. It would be nice to see a throughput versus perplexity graph as well, as opposed to just sparsity vs ppl/throughput, e.g. merge tables 2 and 3.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "To the best of my knowledge, the method of structured approximations across multiple matrices is novel and the results are strong. For the most part, the paper is well-written.", "weaknesses": "One weakness is the lack of justification for the approximation methods for each weight group. Could you give more intuition behind why each method was chosen? For example, the sentence \"Since $W_U$ is inside a nonlinear function $\\sigma_s$, we constrain the search space for its approximation to a matrix multiplication $W_U S_k$ for tractability, where $S_k$ is the $k$-column selection matrix\" (line 244) only describes the approximation, whereas a justification would explain why Nystrom is a better fit for this problem than other methods.\n\nAnother weakness is the relative lack of analysis on the global sparsity allocation. However, this is orthogonal to the main contribution of structured multi-weight approximations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "so9aryOAVL", "reviewer": "Reviewer_XvJF", "summary": "This paper proposes MoDeGPT, an accurate structured pruning algorithm for LLMs. \nThe main idea of MoDeGPT is to define \"modules\", a novel pruning structure, and apply tailored decomposition algorithms for three different types of modules.\nThe main strengths of this paper are (1) introducing decomposition algorithms that are not previously used in this domain, (2) proposing a new global sparsity allocation algorithm, and (3) exhaustive experiments and theoretical analysis in Appendix.\nHowever, I have concerns regarding the following: (1) overclaiming regarding the efficiency of MoDeGPT,  (2) lack of experiments regarding large models, e.g., Llama 3 70B, and (3) too simplified proof of Theorem 4.\nTherefore, I summarized my concerns in \"Weaknesses\" and \"Questions\" and I need to discuss them with the authors.\nThe score can be increased according to the author's response.", "questions": "1. Can MoDeGPT outperform \"efficient\" competitors, such as SliceGPT [2], SLEB, if the competitors perform fine-tuning on the sample dataset to have the same pruning cost as MoDeGPT?\n\n2. Could you elaborate on the detailed explanation of the proof for Theorem 4? Is it permissible to assume that epsilon is large enough to simplify the problem?\n\n3. Does the proposed Global Sparsity Allocation outperform OWL [3]'s strategy?\n\n4.  Does MoDeGPT outperform competitors when pruning gigantic models, e.g., Llama-3 70B?\n\n5. What are the characteristics of Nystrom approximation, CR decomposition, and SVD, and why do we have to use them as proposed in this paper?\n\n### References\n\n[1] Song, Jiwon, et al. \"SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks.\" arXiv preprint arXiv:2402.09025 (2024).\n\n[2] Ashkboos, Saleh, et al. \"Slicegpt: Compress large language models by deleting rows and columns.\" arXiv preprint arXiv:2401.15024 (2024).\n\n[3] Yin, Lu, et al. \"Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity.\" arXiv preprint arXiv:2310.05175 (2023).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper has diverse strengths and I summarize them as follows:\n\n### Method\n1. The authors introduce Nystrom approximation, CR decomposition, and SVD to pruning row-column pairs in LLMs. To the best of my knowledge, this is the first work to use Nystrom approximation and CR decomposition to prune LLMs. The authors carefully use them to prune different types of modules.\n\n2. The authors propose a novel global sparsity allocation algorithm with entropic regularization. If this algorithm contributes a lot to improving the accuracy of the pruned models, then this algorithm can be broadly used in pruning.\n\n### Experiments\n3. The authors conduct exhaustive experiments to show the superiority of MoDeGPT. Their experiments not only covers accuracies, but also inference speed and pruning cost.\n\n4. The authors analyze the effect of MoDeGPT in a detailed way. They also analyze the sparsity patterns.\n\n### Writing\n\n5. The contents are well-organized and easy to read. Specifically, the authors assign unique colors for each module type and consistently use them. This was very helpful to understand this paper.", "weaknesses": "### Method\n\n1. In the caption of Figure 1, the authors insist that their new pruning structure avoids the need for extra adapters. However, SliceGPT's adapters are introduced to improve accuracy and can be removed for inferencing without (dimensional) errors. Therefore, that statement should be modified.\n\n2. The main contribution of this paper is introducing diverse decomposition algorithms and applying them to the proper modules. However, there are lack of explanations of the characteristics of these decomposition algorithms and justification for using them for each type of module.\n\n3. The proof of Theorem 4 is too simplified and hard to understand. There are lack of explanations to get Equation 33. The authors impose a strong assumption that epsilon becomes infinity which indicates the uniformness of phis.\n\n### Experiment\n\n4. The authors emphasize that MoDeGPT is an efficient pruning algorithm, for example, in Lines 475-477, but MoDeGPT requires expensive pruning costs more than 8 hours for pruning Llama-2 13B models. According to SELB [1], most of pruning algorithms requires less than 16 minutes for pruning Llama-2 13B models. Therefore, it is an overclaiming to insist that MoDeGPT is an efficient algorithm. \n\n5. There are lack of competitors. The authors should compare their results with state-of-the-art pruning algorithms, especially layer (or block) pruning algorithms, such as SLEB [1]. Layer pruning algorithms provide significant inference speedup and should be included in Figure 3.\n\n### Writing\n\n6. The second paragraph of the Introduction is too detailed and hard to find the main point. It is hard to capture \"these challenges\" in the third paragraph after reading.\n\n7. The criteria of Table 1 are ambiguous. (1) \"No backward propagation\" seems like an indirect criteria of pruning efficiency, but MoDeGPT is slow without requiring backpropagation. (2) What is the threshold of maintaining accuracy? (3) SparseGPT supports 2:4 pruning which is treated as a (semi-)structured pruning algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kUzTJuJ4nv", "reviewer": "Reviewer_dBU8", "summary": "This paper introduces MoDeGPT, a novel training-free compression method for large language models.  It presents a systematic framework for categorizing approximation challenges in Transformer compression, complete with error guarantees. MoDeGPT demonstrates significant performance gains. This method outperforms prior approaches in compression, and achieves a 46% increase in inference throughput.", "questions": "The specific questions and suggestions are listed below:\n\n(1)Do you consider evaluating on more diverse tasks to verify the method's generalizability?\n\n(2)In the specific experiments, could you provide the chosen rank size for the matrix decomposition or an analysis of the related experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has the following strengths：\n\n(1) The paper presents a novel training-free compression method called MoDeGPT, applies matrix decomposition at the module level for the first time, and extends the theoretical foundation for weight decomposition in language models.\n\n(2) The paper offers a comprehensive literature review and theoretical analysis, demonstrates significant performance improvements through experimental results, and provides error guarantees along with a theoretical framework.\n\n(3) The method outperforms previous approaches in compression performance, achieves a 46% increase in inference throughput, and enhances the practical value of large language models.", "weaknesses": "The Weaknesses of the paper are listed as follows：\n(1) MoDeGPT shows intrinsic bias, performing well on some zero-shot tasks but poorly on others, and currently lacks a solution for bias removal.\n(2) Overfitting of the model to calibration data prevents the compression method from generalizing across most tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)