Structured Mixture-of-Experts LLMs Compression  via Singular Value Decomposition

Authors: Wei Li, Lujun Li, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, Yike Guo
Primary area: foundation or frontier models, including LLMs
Keywords: Mixture of Experts, Efficient Large Language Models, Low-Rank Decomposition, Network Sparsity

Abstract:
Mixture of Experts (MoE) architecture has emerged as a powerful paradigm in the development of Large Language Models (LLMs), offering superior scaling capabilities and reduced computational costs. However, the increased parameter budgets and memory overhead associated with MoE LLMs pose significant challenges to their efficiency and widespread deployment. In this paper, we present MoE-SVD, the first decomposition-based compression framework tailored for MoE LLMs without any extra training. By harnessing the power of Singular Value Decomposition (SVD), MoE-SVD addresses the critical issues of decomposition collapse and matrix redundancy in MoE architectures.   Specifically, we first decompose experts into compact low-rank matrices, resulting in accelerated inference and memory optimization. In particular, we propose selective decomposition strategy by measuring sensitivity metrics based on weight singular values and activation statistics to automatically identify decomposable expert layers. Then, we share a single V-matrix across all experts and employ a top-k selection for U-matrices. This low-rank matrix sharing and trimming scheme allows for significant parameter reduction while preserving diversity among experts.  Comprehensive experiments conducted on Mixtral-8×7B|22B, Phi-3.5-MoE and DeepSeekMoE across multiple datasets reveal that MoE-SVD consistently outperforms existing compression methods in terms of performance-efficiency tradeoffs. Notably, we achieve a remarkable 60\% compression ratio on Mixtral-7x8B and Phi-3.5-MoE, resulting in a 1.5$\times$ inference acceleration with minimal performance degradation. Codes are available in the supplementary materials.

Decision: reject
Reviewer scores: 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730363520375

thread_type:
official_review

<thread_object>
creator: reviewer_nyc5

<content>
summary:
This paper proposes a compression framework called MoE-SVD, which uses a low-rank matrix decomposition of experts and a selective decomposition strategy based on weight singular values and activation statistics to identify the factorable expert layer, while sharing the V-matrix and implementing top-k selection in the U-matrix to reduce the number of parameters while maintaining the diversity among experts. The experimental results show that MoE-SVD performs better than existing compression methods on multiple datasets in terms of performance and efficiency tradeoff.

strengths:
1. The method seems technically sound and straightforward in principle.
2. Good written and easy to follow.

weaknesses:
1. The accuracy degradation of the compressed model is too obvious. With a 20% compression ratio, the average accuracy is generally reduced by 5%-10%. Even with the LoRA fine-tuning, the compression model cannot be improved to lossless, which greatly limits the practicability of the method.
2. Once it comes to fine-tuning, the author needs to compare it with some fine-tuning compression methods, such as MC-SMoE [1], which also uses low-rank decomposition + sparse technique.
3. For matrix multiplication, Rank(AB) ≤ min(Rank(A), Rank(B)), which also means that for an MLP layer, the rank of its output y is often less than W and x. Maybe using AFM [2] instead of SVD here is better, i.e., perform SVD decomposition of MLP output y, and then merge UV into the MLP weight. According to paper [2], AFM seems to be consistently superior to SVD.
4. The authors only test the throughput in Mixtral-8x7B and Phi-3.5-MoE, lacking speed on DeepSeekMoE. This is a little strange, because some MoE LLMs (such as DeepSeekMoE and Qwen2-57B-A14B) have a very heavy share-expert, which is a key bottleneck in LLM inference. Can the author discuss the actual speed performance of MOE-SVD on this type of model?
5. For current LLM inference, to achieve higher throughput, tensor parallelism is generally adopted. However, low-rank decomposition will split one MLP layer into two layers, which may result in relatively large communication overhead when combined with tensor parallelism. At present, the throughput measurement has only been done on an H100 GPU. Can authors further discuss the tensor parallelism (like TP4) inference with  MoE-SVD in detail?

[1]: Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy, In ICLR2024.
[2]: Compressing transformers: features are low-rank, but weights are not! In AAAI 2023.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730409830596

thread_type:
official_review

<thread_object>
creator: reviewer_efn1

<content>
summary:
This paper proposed MoE-SVD, which is a training-free decomposition-based compression framework. Specifically, they first decompose experts into low-rank matrices via SVD. In particular, they selectively decompose the expert layers based on sensitivity metrics. Then, they share a single V-matrix across all experts and use a top-k selection for U-matrices for parameter reduction.

strengths:
1. This paper explores SVD-based methods for MoE LLMs and identifies key factors that degrade performance: certain layers are sensitive to decomposition, and activation patterns differ in MoE, revealing expert redundancy. 
2. the authors propose a selective decomposition strategy along with low-rank matrix sharing and trimming, which are well-justified approaches. 
3. Overall, the paper is clear and easy to follow.

weaknesses:
1. In line 53, could you elaborate on "Hardware Dependency" in the context of semi-structured sparse methods? 

2. In Eq. (4),  $r_i$ represents the rank, but I believe the matrix should be of full rank, so the purpose of using $r_i$ here is unclear. Additionally, what does $f_i$ represent? Is it related to sampling frequency? 

3. Figure 4 is unclear to interpret. It appears that the green grid denotes the decomposed experts, but could you clarify what the compression ratios, also shown on the y-axis, represent? While the overall motivation, that the approach seeks to identify layers with varying decomposition sensitivity, is understandable, the details require further clarification.

4. For the experiments, including Qwen in the experiments could strengthen the results.

questions:
Please clarify more on Section 3.2
</content>
</thread_object>

</thread>
<thread>
creation date:
1730585988226

thread_type:
official_review

<thread_object>
creator: reviewer_ghaz

<content>
summary:
This paper introduces MoE-SVD, a novel framework leveraging SVD for compressing Mixture-of-Experts architectures in large language models. MoE-SVD addresses key challenges of decomposition collapse and parameter redundancy in MoE models, utilizing techniques such as selective decomposition, matrix sharing, and matrix trimming. Experiments conducted on Mixtral-8x7B, Phi-3.5-MoE, and DeepSeekMoE demonstrate its ability and inference acceleration. Further evaluations with LoRA fine-tuning and quantization suggest MoE-SVD’s adaptability across different MoE backbones.

strengths:
- The paper thoroughly evaluates the proposed method across various MoE architectures, including Mixtral-8x7B, Phi-3.5-MoE, and DeepSeekMoE.
- Additional experiments with LoRA fine-tuning and quantization validate the efficacy of MoE-SVD.
- MoE-SVD introduces a unique method for MoE compression. The matrix-sharing techniques can be interesting.

weaknesses:
- The paper does not include comparisons with structured compression methods directly applicable to MoE models, such as those presented in [1, 2, 3]. Additionally, it overlooks methods that specifically target expert layer compression or simultaneously address both expert layer and expert number compression, as found in [4, 5, 6]. These comparisons are crucial for a comprehensive evaluation of MoE-SVD's performance.
- Key metrics such as model size reduction, TFLOPS, and runtime are missing. These metrics are critical for assessing the practical efficiency of the proposed method.
-  MoE-SVD shows a notable decline in performance even with a 20% parameter reduction, indicating possible limitations in maintaining model quality during compression. Furthermore, the absence of significance testing for performance claims weakens the robustness of the results.
- The representation and decomposition process of the expert matrix is ambiguous. In the expert decomposition subsection, the paper describes the expert matrix as $\mathbb{R}^{m \times n}$, but experts are typically two- or three-layer neural networks. It is unclear whether the matrix is decomposed individually for each layer or concatenated before decomposition, and if so, how the concatenation is done. Clarifying these points is important for the reader's understanding.
- The calculation of the sensitivity score in the selective decomposition strategy lacks clarity. It is unclear how activations are computed, how the dataset for expert frequency calculation is chosen, and whether the frequency values obtained from one dataset can be generalized to others. Providing pseudo-code would enhance comprehensibility.
- The reasoning behind sharing the V-matrix to maintain performance is insufficiently explained. Further elaboration on the properties that enable the effective shared use of this matrix would be beneficial.
-  The rationale for each expert storing information from two other experts, as indicated in equation (8), requires clearer justification. Specifically, an explanation is needed on why each expert needs information from other experts and how this contributes to enhancing the model’s overall performance. Although the paper mentions diversity as a factor, it is unclear why simply combining the U-matrix suffices can work, especially in the context of zero-shot MoE-LLMs.

[1] A simple and effective pruning approach for large language models. ICLR 2024

[2] Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML 2023

[3] LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation. ICML 2023

[4] Merge, then compress: Demystify efficient SMoe with hints from its routing policy. ICLR 2024

[5] STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning.

[6] Demystifying the Compression of Mixture-of-Experts Through a Unified Framework.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732243888591

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
**Dear Reviewers, Area Chairs, Senior Area Chairs and Program Chairs,**

We sincerely thank all reviewers for their positive feedback and constructive comments. Reviewers positively acknowledge **the novelty of the idea, the methodology employed, the extensive experiments conducted, the superior performance, and the good presentation of the paper**. More encouragingly, **Reviewer ghaz and eFn1** think our **novel framework tackles an important problem in MoE LLM efficiency** for the community: 

**[Novelty]:**

- **Reviewer ghaz:** *"MoE-SVD introduces a unique method for MoE compression; the matrix-sharing techniques can be interesting."*
- **Reviewer eFn1:** *"Explores SVD-based methods for MoE LLMs and identifies key factors that degrade performance."*

**[Theoretical Soundness]:**

- **Reviewer eFn1:** *"Proposes a selective decomposition strategy along with low-rank matrix sharing and trimming, which are well-justified approaches."*
- **Reviewer nYc5:** *"The method seems technically sound and straightforward in principle."*

**[Comprehensive Evaluation]:**

- **Reviewer ghaz:** *"The paper thoroughly evaluates the proposed method across various MoE architectures, including Mixtral-8x7B, Phi-3.5-MoE, and DeepSeekMoE."*
- **Reviewer eFn1:** *"Additional experiments with LoRA fine-tuning and quantization validate the efficacy of MoE-SVD."*

**[Clear Presentation]:**

- **Reviewer eFn1:** *"Overall, the paper is clear and easy to follow."*
- **Reviewer nYc5:** *"Good written and easy to follow."*

**[Efficacy & Practicality]:**

- **Reviewer ghaz:** *"Additional experiments with LoRA fine-tuning and quantization validate the efficacy of MoE-SVD."*
- **Reviewer nYc5:** *"The method seems technically sound and straightforward in principle."*



In the past two weeks, we carefully improved the experiments (using all computational resources we have), the clarifications and the discussions of our work to address the concerns, the questions and the requests by all four reviewers. Summarily, we made the following improvements:

**(1)** To enhance the empirical validation of our MoE-SVD methodology, following Reviewers ghaz, eFn1, and nYc5, we have restructured our experimental results through six quantitative analyses in 6 tables: **(a)** Table 1 presents a rigorous representation of our decomposition strategy, replacing  the previous Figure 4; **(b)** Tables 11 and 12 demonstrate MoE-SVD's performance improvements (performance degradation of only 3% and 5%) on Mixtral-8×7B and Phi3.5 MoE under 20% compression after incorporating more calibration data, with statistical significance measurements.; **(c)** Tables 9 and 10 provide comparative analyses against established methods (Wanda, SparseGPT, LoSparse, MC-SMoE, and Unified-MoE) with key performance metrics (TFLOPs, runtime, model size) ; **(d)** Table 13 extends our evaluation to the Qwen 57B model; **(e)** A new analysis of Decomposed Layer/CKA distance justifies our matrix sharing strategy. 

**(2)** To strengthen our methodological framework, responding to all reviewers, we have: **(a)** formalized our sensitivity score computation; **(b)** Clarify theoretical foundations for our V matrix sharing strategy through CKA distance analysis; **(c)** enhanced the methodology sections (lines 226-230, 243, 257) and appendices (B.2-B.4, C, D.1-D.2) with detailed explanations of sensitivity metrics, decomposition criteria, and implementation protocols.

**(3)** To elucidate the clarity of our approach, we have introduced three new algorithms tables that provide: **(a)** detailed implementations with PyTorch code for core components (sensitive scores and sampling frequency collection, experts decomposition, selection strategy); **(b)** refined pseudocode for the SVD-MOE framework and MakePositiveDefinite function (Algorithms 4-5).

**(4)** Following Reviewers nYc5 and eFn1, we have conducted runtime performance analyses on DeepSeekMoE and Qwen models in Table 10 and 13, validating our approach across diverse MoE architectures and shared expert configurations.





**Finally, we have carefully integrated all these improvements into a cohesive revision that we believe enhances the quality of our work.** We look forward to the reviewers' feedback on these comprehensive updates.





Best regards,

Paper 88 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission proposes an SVD based method for mixture-of-experts compression.  As pointed out in the reviews, mixture-of-experts model compression has received a lot of attention with many competing methods.  The submission provides some improvements across the provided metrics and settings considered.  Two reviewers felt that the submission could be marginally acceptable, while one reviewer argued for rejection.  Due to some remaining concerns, a large number of competing methods, and weak support for acceptance, the submission falls below the threshold for acceptance to ICLR in its current form.