MoDeGPT: Modular Decomposition for Large Language Model Compression

Authors: Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
Primary area: foundation or frontier models, including LLMs
Keywords: llm, model compression, matrix decomposition

Abstract:
Large Language Models (LLMs) have significantly advanced AI with their exceptional performance across a wide range of tasks. However, their extensive computational requirements restrict their use on devices with limited resources.
While recent compression methods based on low-rank matrices show potential
solutions, they often suffer from significant loss of accuracy or introduce substantial
overhead in parameters and inference time. In this paper, we introduce Modular De-
composition (MoDeGPT), a new, efficient, and structured compression framework
that overcomes these limitations. MoDeGPT jointly decomposes pairs of consecu-
tive subcomponents within Transformer blocks, reduces hidden dimensions through
output reconstruction on a larger structural scale than conventional low-rank meth-
ods, and repurposes three classical matrix decomposition algorithms—Nyström
approximation, CR decomposition, and SVD—to ensure bounded errors in our
novel decomposition approach. Our experiments show that MoDeGPT, without
relying on backward propagation, consistently matches or surpasses the performance of prior techniques that depend on gradient information, while achieving a
98% reduction in compute costs when compressing a 13B-parameter model. On
LLaMA-2/3 and OPT models, MoDeGPT retains 90-95% of zero-shot performance
with compression rates of 25-30%. The compression process can be completed on
a single GPU in a few hours, boosting inference throughput by up to 46%.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730114849236

thread_type:
official_review

<thread_object>
creator: reviewer_dbu8

<content>
summary:
This paper introduces MoDeGPT, a novel training-free compression method for large language models.  It presents a systematic framework for categorizing approximation challenges in Transformer compression, complete with error guarantees. MoDeGPT demonstrates significant performance gains. This method outperforms prior approaches in compression, and achieves a 46% increase in inference throughput.

strengths:
The paper has the following strengths：

(1) The paper presents a novel training-free compression method called MoDeGPT, applies matrix decomposition at the module level for the first time, and extends the theoretical foundation for weight decomposition in language models.

(2) The paper offers a comprehensive literature review and theoretical analysis, demonstrates significant performance improvements through experimental results, and provides error guarantees along with a theoretical framework.

(3) The method outperforms previous approaches in compression performance, achieves a 46% increase in inference throughput, and enhances the practical value of large language models.

weaknesses:
The Weaknesses of the paper are listed as follows：
(1) MoDeGPT shows intrinsic bias, performing well on some zero-shot tasks but poorly on others, and currently lacks a solution for bias removal.
(2) Overfitting of the model to calibration data prevents the compression method from generalizing across most tasks.

questions:
The specific questions and suggestions are listed below:

(1)Do you consider evaluating on more diverse tasks to verify the method's generalizability?

(2)In the specific experiments, could you provide the chosen rank size for the matrix decomposition or an analysis of the related experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730436213695

thread_type:
official_review

<thread_object>
creator: reviewer_xvjf

<content>
summary:
This paper proposes MoDeGPT, an accurate structured pruning algorithm for LLMs. 
The main idea of MoDeGPT is to define "modules", a novel pruning structure, and apply tailored decomposition algorithms for three different types of modules.
The main strengths of this paper are (1) introducing decomposition algorithms that are not previously used in this domain, (2) proposing a new global sparsity allocation algorithm, and (3) exhaustive experiments and theoretical analysis in Appendix.
However, I have concerns regarding the following: (1) overclaiming regarding the efficiency of MoDeGPT,  (2) lack of experiments regarding large models, e.g., Llama 3 70B, and (3) too simplified proof of Theorem 4.
Therefore, I summarized my concerns in "Weaknesses" and "Questions" and I need to discuss them with the authors.
The score can be increased according to the author's response.

strengths:
This paper has diverse strengths and I summarize them as follows:

### Method
1. The authors introduce Nystrom approximation, CR decomposition, and SVD to pruning row-column pairs in LLMs. To the best of my knowledge, this is the first work to use Nystrom approximation and CR decomposition to prune LLMs. The authors carefully use them to prune different types of modules.

2. The authors propose a novel global sparsity allocation algorithm with entropic regularization. If this algorithm contributes a lot to improving the accuracy of the pruned models, then this algorithm can be broadly used in pruning.

### Experiments
3. The authors conduct exhaustive experiments to show the superiority of MoDeGPT. Their experiments not only covers accuracies, but also inference speed and pruning cost.

4. The authors analyze the effect of MoDeGPT in a detailed way. They also analyze the sparsity patterns.

### Writing

5. The contents are well-organized and easy to read. Specifically, the authors assign unique colors for each module type and consistently use them. This was very helpful to understand this paper.

weaknesses:
### Method

1. In the caption of Figure 1, the authors insist that their new pruning structure avoids the need for extra adapters. However, SliceGPT's adapters are introduced to improve accuracy and can be removed for inferencing without (dimensional) errors. Therefore, that statement should be modified.

2. The main contribution of this paper is introducing diverse decomposition algorithms and applying them to the proper modules. However, there are lack of explanations of the characteristics of these decomposition algorithms and justification for using them for each type of module.

3. The proof of Theorem 4 is too simplified and hard to understand. There are lack of explanations to get Equation 33. The authors impose a strong assumption that epsilon becomes infinity which indicates the uniformness of phis.

### Experiment

4. The authors emphasize that MoDeGPT is an efficient pruning algorithm, for example, in Lines 475-477, but MoDeGPT requires expensive pruning costs more than 8 hours for pruning Llama-2 13B models. According to SELB [1], most of pruning algorithms requires less than 16 minutes for pruning Llama-2 13B models. Therefore, it is an overclaiming to insist that MoDeGPT is an efficient algorithm. 

5. There are lack of competitors. The authors should compare their results with state-of-the-art pruning algorithms, especially layer (or block) pruning algorithms, such as SLEB [1]. Layer pruning algorithms provide significant inference speedup and should be included in Figure 3.

### Writing

6. The second paragraph of the Introduction is too detailed and hard to find the main point. It is hard to capture "these challenges" in the third paragraph after reading.

7. The criteria of Table 1 are ambiguous. (1) "No backward propagation" seems like an indirect criteria of pruning efficiency, but MoDeGPT is slow without requiring backpropagation. (2) What is the threshold of maintaining accuracy? (3) SparseGPT supports 2:4 pruning which is treated as a (semi-)structured pruning algorithm.

questions:
1. Can MoDeGPT outperform "efficient" competitors, such as SliceGPT [2], SLEB, if the competitors perform fine-tuning on the sample dataset to have the same pruning cost as MoDeGPT?

2. Could you elaborate on the detailed explanation of the proof for Theorem 4? Is it permissible to assume that epsilon is large enough to simplify the problem?

3. Does the proposed Global Sparsity Allocation outperform OWL [3]'s strategy?

4.  Does MoDeGPT outperform competitors when pruning gigantic models, e.g., Llama-3 70B?

5. What are the characteristics of Nystrom approximation, CR decomposition, and SVD, and why do we have to use them as proposed in this paper?

### References

[1] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

[2] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

[3] Yin, Lu, et al. "Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity." arXiv preprint arXiv:2310.05175 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706546680

thread_type:
official_review

<thread_object>
creator: reviewer_z3ht

<content>
summary:
This paper proposes MoDeGPT, which compresses transformers by applying structure decompositions on operations that span *two* weight matrices. The parameter subgroups targeted are the MLP weights, key and query projections, and value and attention output projections. Experimental results show that MoDeGPT is the best no-gradient structured method, and also comparable to the best structured and gradient-based method.

strengths:
To the best of my knowledge, the method of structured approximations across multiple matrices is novel and the results are strong. For the most part, the paper is well-written.

weaknesses:
One weakness is the lack of justification for the approximation methods for each weight group. Could you give more intuition behind why each method was chosen? For example, the sentence "Since $W_U$ is inside a nonlinear function $\sigma_s$, we constrain the search space for its approximation to a matrix multiplication $W_U S_k$ for tractability, where $S_k$ is the $k$-column selection matrix" (line 244) only describes the approximation, whereas a justification would explain why Nystrom is a better fit for this problem than other methods.

Another weakness is the relative lack of analysis on the global sparsity allocation. However, this is orthogonal to the main contribution of structured multi-weight approximations.

questions:
1. In Table 3, is the main claim that although semi-structured methods may outperform MoDeGPT, they are held back by custom GPU support which hinders research velocity?
2. It would be nice to see a throughput versus perplexity graph as well, as opposed to just sparsity vs ppl/throughput, e.g. merge tables 2 and 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707154414

thread_type:
official_review

<thread_object>
creator: reviewer_zuw6

<content>
summary:
This paper proposes a novel model compression method by applying three different matrix decomposition algorithms to three distinct types of computations within Transformers. Compared to previous model compression algorithms, this approach achieves a significant improvement in performance.

strengths:
1. The authors propose the interesting idea of using three different matrix decomposition algorithms to compress computations in both MLP and Attention.
2. Experimental results demonstrate that the proposed method offers advantages in terms of both performance and efficiency compared to prior pruning and matrix decomposition algorithms.
3. The Appendix includes additional methods and experiments related to group-query attention.

weaknesses:
1. The authors suggest using three different types of matrix decompositions for three different types of computations within Transformers, but they do not provide motivation for this choice. For example, why is CR decomposition more suitable for Type-2 computation?

questions:
1. Why does Table 3 include only 50% compression results for models like SparseGPT but lack results for 40% compression? Why is a 40% compression result of MoDeGPT compared to a 50% compression result of SparseGPT?
2. I am curious why magnitude-based and SVD-based compression methods seem to cause model collapse in Table 1, performing worse than random compression (Uniform).
3. The authors applied different compression rates to different layers, but are the compression rates for the three types of computations identical? Based on the analysis in Figure 4, it might be better to allocate a higher compression rate for Attention computations.
4. Why is MoDeGPT more efficient than the baseline at the same compression rate (Figure 3)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732238914964

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/2)

comment:
We would like to thank all reviewers for their encouraging and constructive feedback. Specifically, we sincerely thank **all reviewers** for recognizing the **novelty** of our work.

The insightful comments and suggestions from the reviewers have **significantly enhanced the quality of our submission**. In response, we have conducted additional experiments, provided further clarification on key aspects of our methodology, and strengthened the justification for the MoDeGPT framework.
While we will address each reviewer’s feedback in detail, we summarize the major revisions and additions to the manuscript below:

---

### **Characterization of Modules and Justification of the Proposed Method**

- We have provided more intuitive explanations to justify the use of different decomposition strategies for various modules. Specifically:  
  - Each module is characterized by the **number of nonlinear functions** it contains. 
  - For each matrix within a nonlinear function, we constrain the compressed matrix to be in the form of a multiplication with a column selection matrix to be optimized. This form is essential for ensuring the **tractable optimization** of our modular decomposition objective.
  - The column selection matrix is sparse, with only one non-zero element per column, and therefore cannot be optimized using traditional SVD, which generally outputs dense matrices.  
  - Depending on the number of nonlinear functions, the optimal compression solutions correspond to different types of matrix decompositions. These solutions are formally presented in Theorems 1, 2, and 3, which coincide with various existing matrix decomposition techniques.

---

### **Large Model Experiments (70B)**

- We conducted new experiments on the **Llama2-70B** model, yielding even more promising results than smaller models. Notably, we achieved:  
  - **4.5% and 3.2% drops in performance with 30% compression**, using only 128 calibration samples from WikiText-2 and Alpaca, respectively, without recovery fine-tuning.
  - These results outperform decomposition and layer pruning baselines, including SliceGPT [1], ShortGPT [2], and SLEB [3].

| **Method**                       |  **WikitText-2 ↓** | **ARC-e ↑** | **ARC-c ↑** | **PIQA ↑** | **WinoG. ↑** | **HellaS. ↑** | **BoolQ ↑** | **OBQA ↑** | **MathQA ↑** | **MMLU-ml ↑** | **COPA ↑** | **Lamb. ↑** | **Average ↑** |
|-----------------------------------|------------------|-------------|-----------|----------|------------|-------------|-----------|-----------|------------|-------------|----------|----------|---------------|
| Dense Llama-2 70B            | 3.12             | 80.98       | 57.25     | 82.75    | 77.90      | 83.83       | 83.79     | 48.80     | 38.42      | 42.86       | 94.00    | 79.60    | 70.02         |
| SliceGPT                      | 5.76             | 67.05       | 42.06     | 67.52    | 71.11      | 55.57       | 41.56     | 40.20     | 27.87      | 32.14       | 82.00    | 52.03    | 52.65         |
| ShortGPT                      | 66.33            | 60.65       | 34.47     | 72.74    | 64.01      | 63.80       | 66.88     | 34.40     | 23.05      | 31.25       | 75.00    | 27.01    | 48.06         |
| SLEB                          | 5.54             | 71.97       | 44.20     | *77.74*  | 69.38      | *73.54*     | *67.25*   | 41.80     | 27.47      | *32.15*     | *88.00*  | 64.22    | 59.79         |
| MoDeGPT + OWL Sparsity        | **4.67**         | 76.01       | 50.34     | 74.70    | 72.85      | 72.43       | 69.88     | 44.20     | 32.26      | **44.64**   | 87.00    | 69.61    | 63.08         |
| MoDeGPT + Our Sparsity        | *4.89*           | *77.69*     | *50.94*   | 77.53    | *76.87*    | *78.16*     | *74.71*   | *45.60*   | **35.04**  | *42.86*     | *89.00*  | **72.17**| *65.51*       |
| MoDeGPT + Our Sparsity + Alpaca| 5.73            | **78.57**   | **51.54** | **80.85**| **77.19**  | **79.60**   | **82.81** | **46.40** | *32.83*    | 40.18       | **94.00**| *70.72*  | **66.79**     |



---
</content>
</thread_object>

</thread>
<thread>
creation date:
1732239076631

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update of Manuscript

comment:
Dear reviewers,  

We sincerely thank you for your encouraging and constructive feedback on our manuscript. In response to your suggestions, we have revised the manuscript and **highlighted all changes in blue** for ease of review. Below is a summary of the key updates:

---

###  **Main Text**
- **Characterization of Modules:**  
  **[Section 3.2]** We provided a clear roadmap of our algorithms, detailing the characteristics of each module and justifying the selection of specific matrix decompositions over alternatives.  
- **Large Model (70B) Experiments:**  
  **[Section 4.3]** We added results from experiments on the large-scale Llama2-70B model, demonstrating even more promising outcomes than smaller models, with only a 3.2% performance drop for 30% compression, achieved without fine-tuning.  
- **Enhanced Accuracy-Throughput Trade-Off Plot:**  
  **[Section 4.4]** We provided an improved plot to better illustrate the trade-off between throughput and perplexity.

---

### **Appendix**
- **Theory Part:**  
  - **[Section A.4]** We refined the proof of Theorem 4, including detailed explanations to improve clarity and rigor.  

- **Experiment Part:**  
  - **[Section B.3]** Added experiments evaluating the compressed model on a more diverse set of tasks to assess task generalizability.  
  - **[Section B.6]** Included baseline comparisons under equal computational cost constraints.  
  - **[Section B.9]** Compared our global sparsity allocation approach with the state-of-the-art OWL method and reported layer-wise ranks.  
  - **[Section B.10]** Added experiments evaluating the effect of using nonuniform sparsity across modules within the same layer.  

---
</content>
</thread_object>

</thread>
<thread>
creation date:
1733207259143

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
12/2 Update

comment:
Dear Reviewers,

Thank you for your thoughtful feedback and for taking the time to review our paper. We hope you had a wonderful Thanksgiving holiday. During the rebuttal period, we have worked diligently to refine our methods and are excited to share **the latest improvements** to our methodology, particularly in **inference speed**.

---

### **Improved Speed and Accuracy with Nonuniform Module Sparsity**
Following **Reviewer ZuW6's** suggestion and drawing insights from prior strategies [3], we refined our global sparsity allocation strategy by introducing distinct sparsity levels for the MLP and MHA blocks within each transformer layer. Instead of calculating a single score per layer, we now compute two scores—one for MLP and one for MHA using the same correlation as described in Section 3.3. The updated global sparsity allocation in Equation 10 is as follows:

$$
\max_{\phi_{1:L}}\sum_{i=1}^L\sum_{j \in \{\text{mlp}, \text{mha}\}} w_j (s^j_i (1-\phi^j_i) + \varepsilon H(\phi_i)) \quad \text{such that} \quad \frac{1}{L(w_{\text{mlp}} + w_{\text{mha}})} \sum_{i=1}^L \sum_{j \in \{\text{mlp}, \text{mha}\}} w_j \phi^j_i = \phi_{\text{avg}}, \quad 0 \leq \phi_i \leq 1,
$$

where $\phi^j_i$ and $s^j_i$ represent the sparsity and score for the $j$-th block in layer $i$, respectively, and the weights $w_{\text{mlp}}=2, w_{\text{mha}}=1$ are applied to preserve the average sparsity, consistent with the parameter size ratio in transformer blocks. The solution has a similar closed-form solution:

$$
    \phi = L(w_{\text{mlp}} + w_{\text{mha}})\phi_{\text{avg}}\times\text{Softmax}(-s\odot w/\varepsilon).
$$

This updated strategy has enhanced both compression accuracy and inference throughput, especially in the inference speed. Notably, in our 30% compression experiments on Llama2-7B (as shown in the table below), we achieved **the fastest throughput among all baselines (even faster than layer pruning strategies!) while maintaining superior accuracy**. We are grateful to Reviewer ZuW6 for pointing out this direction and highlighting that our method can benefit from higher sparsity in MHA module.

Importantly, these updates come with minimal computational overhead. Although we now calculate two scores per layer (instead of one), the computational cost is negligible as score calculation remains lightweight and does not increase compression time.

We are thrilled to share these findings and will include comprehensive experiments in the revised paper. Thank you for your insightful feedback and the time spent during the rebuttal period, which have greatly enhanced this research.



| Method                                                 | MLP mean sparsity | MHA mean sparsity | ↑ Throughput  (tokens/s)     | ↑ PIQA | ↑ HellaS. | ↑ WinoG. | ↑ ARC-e | ↑ ARC-c | ↑ Average |
|--------------------------------------------------------|-------------------|-------------------|---------------------|--------|-----------|----------|---------|---------|-----------|
| SLEB [1]                                                 | 30%              | 30%              | 2539.39 (1.49x)    | 69.58  | 58.28     | 58.17    | 52.36   | 31.91   | 54.06     |
| SliceGPT [2]                                             | 30%              | 30%              | 1815.67 (1.07x)    | 68.55  | 48.69     | 59.75    | 56.69   | 34.47   | 53.63     |
| MoDeGPT                                               | 30%              | 30%              | 2490.15 (1.46x)    | 73.34  | **65.90**     | 66.22    | 65.49   | **39.16**   | 62.02     |
| MoDeGPT w/ nonuniform module sparsity | 26.80%           | 36.43%           | **2722.98 (1.60x)**    | **73.78**  | 65.14     | **68.03**    | **66.79**   | 38.40   | **62.43**     |


---

### **References**

[1] Song, Y., et al. "SLEB: Sparsity-aware learning for efficient bandwidth in language models," 2024.  
[2] Ashkboos, S., et al. "SliceGPT: Efficient fine-tuning of large language models by slicing and pruning," 2024.  
[3] Zhang, X., et al. "FINERCUT: Finer-grained Interpretable Layer Pruning for Large Language Models," 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new approach to compressing Transformer-based models. The idea is to use a set of particular forms of low-rank matrix factorization for the weight matrices. The authors’ strategy is fairly sophisticated, as it seeks to associate various component operations inside a Transformer with different matrix approximation approaches. 

There’s a bunch of strengths here: the overall approach is creative, the empirical results are pretty strong. The authors have provided extensive details on the approach. 

This is a good paper that is worth accepting.