MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Authors: Wei Li
Lujun Li
Hao Gu
You-Liang Huang
Mark G. Lee
Shengjie Sun
Wei Xue
Yike Guo
Venue: ICML 2025 poster (2025)
Keywords: Mixture of Experts
Efficient Large Language Models
Low-Rank Decomposition
Network Sparsity
Primary area: deep_learning->large_language_models
TL;DR: In this paper, we present a novel training-free compressor for MoE LLMs that uses SVD to break experts into smaller matrices, sharing and trimming them to save memory, speed up inference.

Abstract: Mixture of Experts (MoE) architecture improves Large Language Models (LLMs) with better scaling, but its higher parameter counts and memory demands create challenges for deployment. In this paper, we present MoE-SVD, a new decomposition-based compression framework tailored for MoE LLMs without any extra training. By harnessing the power of Singular Value Decomposition (SVD), MoE-SVD addresses the critical issues of decomposition collapse and matrix redundancy in MoE architectures.   Specifically, we first decompose experts into compact low-rank matrices, resulting in accelerated inference and memory optimization. In particular, we propose selective decomposition strategy by measuring sensitivity metrics based on weight singular values and activation statistics to automatically identify decomposable expert layers. Then, we share a single V-matrix across all experts and employ a top-k selection for U-matrices. This low-rank matrix sharing and trimming scheme allows for significant parameter reduction while preserving diversity among experts.  Comprehensive experiments on Mixtral, Phi-3.5, DeepSeek, and Qwen2 MoE LLMs show MoE-SVD outperforms other compression methods, achieving a 60\% compression ratio and 1.5× faster inference with minimal performance loss.

MoE-SVD: Structured Mfxture-of-Experts LLMs Compression via Singular Value Decomposition
WeiLi"
Lujun Li
Hao Gu
You-Liang Huang²
Mark Lee'
Shengjie Sun
Wei Xue?
Yike Guo
Abstract
language models (LLMs) (e.g., MiniMax-01 (MiniMax et al., 2025), Qwen2.5-Max (Team, 2024) and DeepSeek- V3 (DeepSeek-Al et al., 2024)). These architectures incor- porate multiple experts and employ a sparse gating mech- anism, enabling efficient computation and facilitating the scaling of LLMs within the constraints of limited compu- tational resources (Dai et al., 2024). Despite these advan- tages, MoE LLMs still face several challenges: (1) Im- mense Parameter Sizes: MoE models generally have a larger number of parameters than dense models (Gu et al., 2025), which can make them difficult to train and deploy, es- pecially in resource-constrained environments. (2) Memory Overhead: MoE models can suffer from memory ineffi- ciency due to the need store and access multiple expert weights and biases, potentially hindering their deployment devices with limited memory (Song et al., 2023).
Mixture of Experts (MoE) architecture improves Large Language Models (LLMs) with better scal- ing, but its higher parameter counts and mem- ory demands create challenges for deployment. In this paper, we present MoE-SVD, a new decomposition compression framework tai- lored for MoE LLMs without any extra training. By harnessing the power of Singular Value De- composition (SVD), MoE-SVD addresses the crit- ical issues of decomposition collapse and matrix redundancy in MoE architectures. Specifically, we first decompose experts into compact low-rank matrices, resulting in accelerated inference and memory optimization. In particular, we propose selective decomposition strategy by measuring sensitivity metries based on weight singular val- ues and activation statistics automatically iden- tify decomposable expert layers. Then, we share single V-matrix across all experts and employ top-k selection for U-matrices. This low-rank matrix sharing and trimming scheme allows for significant parameter reduction while preserving diversity among experts. Comprehensive experi- ments on Mixtral, Phi-3.5, DeepSeck, and Qwen2 MoE LLMs show MoE-SVD outperforms other compression methods, achieving a 60% compres- sion ratio and 1 1.5x faster inference with minimal performance loss.
Limitations of Traditional MoE Compressions: To ad- dress the challenges of large parameter size and memory overhead, some MoE-specific compression methods have been proposed to prune unimportant experts or weights, For example, NAEE (Lu et al., 2024) proposes task- specific expert pruning and dynamic skipping, while MoE- Compression (He et al., 2024) evaluates various types of sparse schemes across multiple MoE components. Although these techniques show promise, they suffer from certain lim- itations: (1) Performance Degradation, especially under high compression ratios, often necessitating costly and time- consuming retraining. For instance, pruning 25% of experts in Mixtral-8xi0 results in 23% performance drop (He et al., 2024). (2) Hardware Dependency: Some semi- structured sparse methods only gain speedup on NVIDIA Ampere and Hopper architecture GPUs, limiting their gen- eral applicability. (3) Limited Acceleration: Some MoE compression methods only reduce the number of experts without significantly reducing the size of the activated ex- perts (Liu al., 2024a), resulting in minimal speedup dur- ing inference. MoE- Moo-Compression (He et al., 2024) report that eliminating 12.5% of experts yields less than 1% speed boost. In contrast, recent Singular Value Decompo- sition (SVD) techniques (Hsu et al., 2022) are hardware- independent and successfully compact LLM to high com- pression ratios without additional training. These facts en- courage us to explore SVD alternative pruning.
1. Introduction
Mixture of Experts (MoE) (Cai et al., 2024b) have demon- strated promising advancmments in the realm of recent large
'Equal contribution 'University of Birmingham "The Hong Kong University of Science and Technology 'The Hong Kong University of Science and Technology (Guangzhou) 'AlSpeech Co., Ltd.. Correspondence to: Mark Lee <M.G.Lee@ tham.ac.uk>. Yike Guo eyikcguo@ust...m hkk.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Structured Centurreof-Expeere LLMs Compression via Singular Value Decomposition
Comperionn of PPL across Different Lavers flog Scnle)
Comparison of OWL and Our metrie scrses Different Lavers
Average CKA Similarity in Layer Decomposition
1.3
Our mert
WL. (E.g Soak)
(1.299)
0.
Aenage lary ke Zariago lary
Avcroge CEA
(1:
[71,
(2% 14)
0.6
(12.
0.4
0.4
Figure 1: Perplexity of 50% per-layer SVD decomposition (left), per-layer values of OWL & our metric (middle), mean CKA similarity (Kornblith et al., 2019) of decomposed V & U matrix and original matrix of each expert layer (right). These results are obtained for Mixtral-8x7B on WikiText-2.
However, we directly apply these SVD-based methods to compress MoE models, resulting in serious performance collapse. For example, Mixtral-8x7B with ASVD (Yuan al., 2023) and SVD-LLM (Wang et al., 2024) reach over 1000 perplexity
that apply uniform compression across different layers, our method introduces a sensitivity metric derived from matrix singular values and activation statistics, facilitating adap- tive decomposition. As illustrated in Figure I (middle), this metric accurately identifies the sensitive expert layers, allow- ing for more targeted compression. (2) Low-rank Matrix Sharing and Trimming: To further minimize parameter redundancy, we introduce V-matrix sharing, where the most frequently sampled V-matrix is retained and shared across all experts. In addition, we apply U-matrix trimming by selecting the top-k U-matrices based on sampling frequency, while discarding the less frequently used matrices. This strategy significantly minimizes the number of parameters, while ensuring the diversity for effective MoE function- ing. With these innovative schemes, our MoF-SVD offers substantial parameter reduction, creates a naturally sparse expert structure for faster inference, and can be deployed on standard computing infrastructure without requiring addi- tional training phases. This flexible framework allows high compression ratios and strikes an optimal balance between computational efficiency and model performance.
WikiText-2. This naturally raises key questions: Why SVD-based methods fail on MoE LLMs and how to solve this?
To answer these questions, we individually decompose each expert layer in Figure 1 and un- cover observations: (1) Decomposition Sensitivity: Some expert layers more sensitive to SVD decomposition than others. For example, initial and final layers of decompo- sition can lead to drastic performance loss. This indicates that layer-wise non-umt uniform decomposition is important for MoE L.Mss. (2) Model Statistic Disparities: Activation outliers in OWL (Yin et al., 2023) are an effective pre-layer importance statistic and metric for dense LLM. However, notice that their values MoF in Figure (middle) do not match the pre-layer decomposition result in (1). This could be attributed to biases derived from multi-expert de- sign and dynamic activation in MoE LLMs. (3) Expert Redundancy: Expert merging methods (Liu et al., 2024a) show various experts are similar in the weight space and contain significant redundancy. Our Figure 1 (right) indi- cates high similarity of decomposed V-matrices, which can further share weights or trim redundant matrices.
Our New Observations:
Validation and Results: Extensive experiments demon- strate Our MoE-SVD method achieves sfte-oof-the-at per- formance compressing MoE models while maintaining their performance on various language modeling and com- mon sense reasoning datasets. The results show that MoE- SVD outperforms other methods such SVD, ASVD, and SVD-LLMM across all compression ratios from 20% to 60%, on both Mixtral-8x7B and Phi-3.5-MoE models. For ex- ample, on the Mixtral-8x -xBB model, MoE-SVD achieves a 20% compression ratio with only a 2% drop in performance, while the other methods experience a significant drop in performance. Similarly, on the Phi-3.5-MoE, MoE-SVD achieves 40% compression ratio with only 5% drop in performance. These results demonstrate the effectiveness of MoE-SVD in compressing MoE models while maintaining their performance. In addition, our MoE-SVD can general- ize well to other MoE LLMs such as DeepSeekMoE- 16B and Mix2,, and can be further improved in perfor- mance by LoRA fine-tuning and efficiency with quantiza-
"Different problems require different solutions."
Albert Einstein
Our Novel Framework: As this well-said quote goes, the sparse-activated MoE dynamic architecture differs from common LLMs and deserves customized decomposition schemes based on the above observations. To this end, we develop MoE-SVD, novel compression framework specif- ically designed for MoE LLMs. Our MoE-SVD leverages SVD to decompose expert layers in a structured manner, creating a naturally sparse expert structure that reduces com- putational costs while maintaining model expressiveness. The core innovation of MoE-SVD lies in: (1) Selective De- composition Strategy: Unlike previous SVD approaches
2

---

Structured Mixture-of- -f--reetssssssss---e LLMs Compression via Singular Value Decomposition
Table 1: Comparison of our MoE-SVD to general LLM compressors (e.g., Wanda, SparseGPT, LoSparse) and MoB-specific compressors (e.g., MC-S, MoE- I?, Cooe-eomprssi.n)
Methhdd
Pipeline
Decomposition
Training
Mot-specilic Strategy
Exaluated Models
Wanda (3023)
Pruning without weight updates
Nout
Note
None
LLaMA, LLaMA-2 138, 708)
SpaneGiP
Pruning and weight updates
None
ADMM
None
OPT-1758, BLOOM- 1768
LoSparse (20236)
Decomipose -spusse
Low rank sparse
uning
None
base (110M), DeBERTAV3 large (131M)
MC Shdlli (2024g)
Merge -compessssssss
Low rank sparse
Fine tning cdhsall)
Frequency based merging
13 base (2200), Switch base 32 (2.08)
MoE-I- (2024)
Seurch-reagen drocmpuuse
Low-rank sporse Nene
Fine uning
Genesic Starch. KT KT-Recrption Field
Qwen. Mixtral-xx27
MoF-Cumgensson (2004)
Prining and loyer dropping
None
Layerdreping, sparetty
Mistral-Sc7B, MorPSo
McF-SVD
Decompose mecotppse-mmtricc share-trim
Low-rank
Selective SVD, shane-trim
Mistral-Sx7B/226, Pla-3 5-43,, DeepSeeliQenMMei
tion. We summarize our contribution as follows:
methods (Lu et al., 2024; He et al., 2024) requiring fine- tuning and drop inactive experts without acceleration, MoE- SVD primarily exploits SVD to reduce the size of activated experts for acceleration without extensive retraining. (3) Our MoE-SVD enhances the SVD for large-scale MoEs without expert merging and fine-tuning, in contrast to MC- SMoE (Li et al., 2024g), which only addresses small-scale MoE by first merging experts and then applying vanilla de- composition before fine-tuning. (4) MoE-SVD performs low-rank matrix sharing and trimming, avoiding the direct dropping of entire experts like MoE-1² (Yang et al., 2024) and MoE-Compression (He et al., 2024), which prevents drastic performance loss. (5) Our method is search-free, avoiding the huge search overhead of in MoE-I² (Yang et al., 2024) and EEP (Liu et al., 2024a). (6) Other approaches include post-training quantization (Li et al., 2024f) and system optimization (e.g., expert offloading (Xue et al., 2024a), parallelism (Cai et al., 2024a), and switching (Liu et al., 2024b)). Our MoE-SVD focuses solely on expert compression and remains orthogonal to these methods.
To overcome limitations of existing methods, we open new doors for MoE compression from the SVD tech- nical route. We derive series of important findings about decomposition collapse, statistic discrepancies, and redundancy, providing insights into this new area.
We introduce MoE-SVD, new structured compressor for MoE LLMss Our MoE-SVD enjoys benefits: high compression ratios, clear inference accelcration, free from specialized hardware and extra training
We propose a selective decomposition that adaptively applies SVD and develop low-rank matrix sharing and trimming techniques. By sharing V-matrices across ex- perts and trimming redundant U-matrices, we achieve significant parameter reduction while maintaining ex- pert diversity and model performance.
Extensive experiments demonstrate the effectiveness of MoE-SVD on Mixtral, Phi-3.5 and DeepSeek MoE. Our MoE-SVD consistently outperforms other SVD- based methods across 20% ~ 60% compression ratios and achieves 1.2x 1.5x inference speedups.
Singular Value Decomposition. Recently, several SVD- based methods have been proposed for compressing LLMs (Golub et al., 1987). FWSVD (Hsu et al., 2022) introduces a weighted low-rank factorization. while ASVD (Yuan et al., 2023) proposes an activation-aware SVD method that considers the activation patterns of the model's lay- ers to improve compression efficiency. Meanwhile, SVD- LLM (Wang et al., 2024) adopts truncation- data whitening and layer-wise parameter update strategies to achieve better compression ratios. In contrast to these gen- eral SVD-based methods, our MoE-SVD is specifically de- signed for MoE LLMs, addressing their unique challenges (e.g., decomposition sensitivity and expert redundancy). Ad- ditionally, while these methods typically uniformly decom- pose every layer in LLMs, our method employs adaptive decomposition across various expert layers in MoE LLMs.
2. Related Work
Mixture of Experts. architecture's ability to achieve supe- rior scaling laws at reduced costs (Clark et al., 2022) has led to its widespread adoption in recent Large Language Models (LLMs) (Jiang al., 2024; Dai et al., 2024). Recent advancements in MoE focus refining expert structures (Dai et al., 2024), enhancing router designs (Zhou et al., 2022), and developing training strategies (Liu et al., 2023). However, MoE L.Mss still face challenges, including in- creased parameter budgets due to expert replication (He et al., 2023), communication costs that enhance latency (Xue et al., 2024b), and significant memory overhead issues (Li et al., 2024g), posing challenges to their efficiency.
3. Methodology
MoF Compression. Table highlights the key differences between our method and other compressor: (1) Rather than LLM compressors like Wanda (Sun et al., 2023) (mostly un- structured), we propose new MoE-specific structured com- pression paradigms. (2) Compared to other expert pruning
Our MoE-SVD introduces SVD expert decomposition, se- lective decomposition strategy, low-rank matrix sharing and trimming to reduce model parameters while maintaining performance. The main process of MoB-SVD is illustrated in Figure 2. More algorithm details are in Appendix C.
3

---

Structured coxture-off-xperty LLMs Compression via Singular Value Decomposition
Router
Router
Router
Router
U.
Us E
Un Z
U+
U222
Un
U+ZI
U.
FFN:
FFN:
FFN.
Vi
V
V.
Sum
Sum
Sum
Sum
Expert Layer
SVD Expert Decomposition
V-matrix Sharing
U-matrix Trimming
Figure 2: Pipeline of MoE-SVD. We first selectively decompose expert layers with SVD, dividing them into U and V matrices. Then, we present V-matrix sharing and U-matrix trimming steps. For V-matrix sharing, we retain only a single V-matrix V. and share it across experts. For U-matrix trimming, we perform frequency- -baed top-k selection of U-matrices and trim out unselected ones.
3.1. SVD Expert Decomposition in MoE LLMs
expert decomposition creates a naturally sparse expert struc- ture, potentially reducing computational costs. The number of experts can be easily adjusted, allowing for fine-grained control over the MoE's capacity and computational require- ments, avenues for compress MoE LLMs.
Recap of MoE Formulation. MoE architectures in LLM enhance model capacity and efficiency by using expert- based Feed-Forward Network (FFNN layers for different input tokens. The output y of the MoE-FFN layer for input is computed as:
3.2. Selective Decomposition Strategy
Σ G(z). Ex(x),
(1)
To determine the sensitivity of expert layers in the MoE to decomposition, we employ selective decomposition strat- egy. This approach based on a carefully crafted sensitivity metric that considers both the singular value decomposition of expert weight matrices and the activation patterns of these experts during inference. For a layer with N experts, we nor- malize these sensitivities using expert sampling frequency to obtain the layer-wise sensitivity metric Sy:
where N is the number of experts, G(x)) is the gating func- tion, The gating function G(x) typically employs top-k selection mechanism, where only the top-k experts are acti- vated G" (2xx) = (Gz)) k)), resulting in sparse output. and (x) is the output of the i-th expert. Each expert E, a standard FFN with two or three fully-connected layers.
Our method and other MoE compressors are focused on compressing FFN experts, which account for the majority of parameters and memory overhead in MoE models.
N
S1.
fc ps-as
(4)
Activation- SVD for Expert Decomposition. In our SVD-based framework, we upply SVD to decompose the weights of expert layers. To capture outliers and low- rank features in activations, we scale the original matrix W matrix W S before SVD, where S is obtained via cholesky decomposition of activation gram matrix (see more details in Appendix D.I). Con- sider an MoE model with N experts, each fully-connected layer represented by the srtivation-weighter
where fi represents the sampling frequency of the i-th ex- pert during router selection, p; denotes the principal rank (number of large value components) of singular vectors Σ, = diag(".,1, ..., .i,a) obtained from the SVD of the i-th expert's 's weight matrix, and a; measures the pro- portion of activation outliers of the i-th expert exceeding the mean absolute activation value (see more details in Ap- pendix D.1). To apply selective decomposition, we set threshold r based on the desired compression ratio. Ex- pert layer with sensitivity S, 2 are preserved without decomposition, while those below the threshold undergo SVD decomposition. This process is repeated for each layer in the network. This selective decomposition strategy allows for a nuanced approach to MoE compression, preserving the most important experts while reducing the computational footprint of less critical components.
to activation-weighee
weight matrix W; E m where i E {1, ..., J). We begin by applying activation-weighte SVD to each expert matrix:
W. = U,E.V",
(2)
where U3 E R and Vi E R=** are orthogonal matrices containing the left and right singular vectors, respectively, and Σ, E mRx is a diagonal matrix containing the sin- gular values in descending order, respectively. To create sparse MoE structure, we first factorize each expert W using the SVD decomposition and then trunca the top-k singular values and their corresponding singular vectors and finally reconstruct an approximated weight matrix:
Decomposable Expert Layer Analysis. To better under- stand our selective decomposition, we show layer decompo- sition results for Mixtral-8x: -8 and Mhi-B.5-MoE in Figure 3. As the compression increases from 20% to 60%, both mod- els exhibit a gradual increase in decomposed layers, albeit with distinct characteristics. Mixtral-8x7B displays a more
(E. = {u. oi Jim
(3)
where ut and U2 are the i-th columns of U and V respec- tively, and a; the i-th singular value. Our SVD-based
4

---

Structured Mixture-of-Expeee LLMs Compression via Singular Value Decomposition
Layer Decomposition Heatmap
is selected follows:
Mixtral 8-7B
20%
V arg max f(Vi),
Compression
50%
40%
Not Desenppooe Decooppo
50%
S(V.) Exex Ui (G))(
(5)
60%
X
Layers
Phi MoE
where X' represents the set of all input tokens, [[-] is the indicator function, and k denotes the number of experts selected by the top-k gating mechanism. After selecting V,, we update all expert matrices to use this shared V-matrix:
20%
Compresson
30%
40%
50%
60%
Layens
~ UUEV"
i= N,
(6)
Figure 3: Decomposed expert layers in MoB-SVD.
The shared V matrix encapsulates common output space transformations across all experts. For the expert matrix W; using the shared V-matrix V,, its expected reconstruction error is B[I[W; W. B[IIW; - V[ Mini- mizing this error is equivalent to maximizing the correlation between W; and U,E,V?" Given that V, is chosen based on the highest router sampling frequency, it represents the most commonly used output transformation. Thus, sharing V, minimizes the expected reconstruction error across all experts.
aggressive decomposition pattern, with approximately half of its layers decomposed at 60% compression, whereas Phi- 3.5-MoE demonstrates greater resilience, maintaining more undecomposed layers at higher compression ratios. Notably, both models consistently undecompose their initial and final layers across all compression levels, suggesting the critical nature of these layers for maintaining model performance. In contrast, certain middle layers in both architectures show remarkable tendencies to decomposition. Mixtral-8x7B exhibits a block-like decomposition pattern, while Phi-3.5- MoE showcases a more uniform distribution of decomposed layers. These observations reveal intriguing patterns and present insights for our selective decomposition of expert layers under layer-wise MoE compression.
U-matrix Trimming:
For the remaining U-matrices, we employ a top-k selection strategy based on router sampling frequency. The diversity among experts is primarily main- tained through the unique U.S, components. Typically, we set k = 2 to balance parameter efficiency and expert diver- sity. The selected U-matrices for each expert are determined as follows:
(U.,,, U12] = Topk (1U, (f())) > f(Vi), k = 2), (7)
3.3. Low-rank Matrix Sharing and Trimming
where TopK selects the k U-matrices with the highest router sampling frequencies among those experts more frequently sampled than expert i. The final expert function for expert i becomes:
Motivation: For MoE models, different expert matrices con- tain some similarities and can be merged (Liu et al., 2024a). As shown in Figure (right), decomposed V-matrices share certain similarities. Consider two expert matrices W1 and W2 with SVD decompositions W1 = Un V and W2 = U2' The similarity in their output transfor- mations can be quantified by the Frobenius inner product of their V-matrices (V1, 122 trained similar tasks, this inner product is often close to high values, indicating high similarity in output transforma- tions. Consequently, we can perform a more fine- ine-grained matrix selection and sharing, achieving a superior trade-off between performance and the number of parameters.
E1(x) = (U.1 + Ur.2
(8)
where Σn,1 and Σg are the corresponding singular value matrices for the selected U-matrices.
(2) For experts
Our method achieves significant parameter reduction compared to the original MoE model with N x m x n parameters. After applying our low-rank decomposition, each expert matrix W. is decomposed into U,E. E mxr, Σg E Rrrx", and V/2 E R"x", resulting in N X (m x + r X n) parameters per expert. With V-matrix sharing. we retain only one V, E R matrix shared across all experts, reducing the parameter count by a factor of N for the V-matrices. Furthermore, with U-matrix selection, we typically select k = 2 U-matrices, reducing the parameter count by factor of N for the U-matrices. 2
Parameter Reduction:
We compress MoE by retaining only the V-matrix with the highest router sampling frequency and sharing this matrix across all experts. This method significantly reduces the model's memory footprint while preserving crucial directional information in the feature space. The router sampling frequency f(V,) for each expert i is computed based on the routing decisions made by the gating network G(x). The shared V-matrix, denoted as V,,
V-matrix Sharing:
Thus, the total number of parameters in our compressed MoE model is m x x 2 n 2mr + rn. Comparing this to the original N m x n parameters, we achieve a
5

---

Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Table 2: Results of MoE-SVD and MoE-SVDi (with LoRA fine-tuning) on three language modeling datasets (measured by perplexity (4)) and seven common sense reasoning datasets (measured by both individual and average accuracy (1)). Ratio means model parameter size reduction ratio. Runtime denotes runtime throughput (Tokens/sec) on a single H800 GPU.
Rano
Method
Runtime
WikeTest 2..
PTB
C4)
Opent.
ARC_
WinoG
HellaS.
ARC_
PIQA
MamQA
Averagel
Mixtral-S- Sx7B
0% Osigiial
X7.7
3.95
12.99
5.76
0.36
0..84
0.76
0.65
0.57
0.52
0:43
0.63
Wands (2:43)
91.1 (1.04%)
4.73
18.8
8:43
0.32
0.775
0.72
0.55
0:47
0.79
0.36
0.57
SpaneCaPT (2123) (2:4)
90.1 (1.000)
4.61
21:11
8.19
0.5
0.77
0.74
0.56
0.45
0.27
0.35
0.56
LaSpane (200
92.9 (1.00)
963.51
505
1225.12
0.2
0.27
(149
0:38
0:26
0.53
0.20
0.32
ASVD (2023)
100. (1.10
47.29
20.30
0.71
0.48
0.35
0.51
SVD-LEM (2024)
(1.66(1.2))
13.45
42.72
17.36
0.42
0.44
20%
Moli- (2024)
(023(112x)
80.04
0.57
0/43
0.25
0.44
MC-SMol: (2004g)
05.3 (1.00x)
134135
131652
1478.13
0:36
0.28
0.51
0.39
0:25
0.54
0.19
0.33
MoE Compessioon (2024)
95.1 (1.13x)
6.12
14.67
11.61
0.3
0.73
0.70
0.54
0.46
0.73
0.33
0.54
MoE-SVD (Ours)
(04.
4.85
19.42
8.98
0.35
0.79
0.74
0.56
0.49
0.78
0.37
0.58
MoR-SVD) (Ours)
(04.(1.25)
4.31
14.94
7.82
0.35
0.80
0.75
0.61
0.55
0.81
0.38
0.50
(202)2202)
(078(1122))
30.57
87.24
0.34
0.23
0.36
SVD-LLM (2024)
10
254.76
252.25
79.40
0.45
0.35
0.25
0.36
40%
MoE-SVD
(093
6.74
27.73
12.41
0.72
0.67
0.43
0.71
0.32
0.50
ASVD-22021)
4909
86.61
164.57
0.42
0.40
0.23
0.35
SVD-LLM (2024)
1220(1.46)
1325.51
1856.2
439.20
0.33
0.38
0:21
0.56
0.23
0.32
McE-SVD (Ours)
(239
9.03
42.93
16.15
0.61
0.37
0.63
0.27
0.43
MoE-SVD: (Oars)
(159
8.73
25.36
12:13
0:25
0.67
0.64
0.50
0.37
0.73
0.26
0.49
ASVD (2023)
(502
12524.91
14702.02
11591.72
0.13
0.25
0.51
0.26
0:21
0.53
0.21
0.30
SVD-LLM (2024)
(48.51)77))
10181.25
9284-95
10967.80
0.34
0.26
0.51
0.36
0.22
0.54
0.21
0.30
MoE-SVD (Ound)
(1.8)
13.52
130.25
39.54
0.19
0.45
0.55
0.38
0.25
0.62
0.25
0.37
Phi-a5-MMl:
06
Original
98.2
3.48
8:43
8:22
0.40
0.77
0.76
0.68
0.56
0.79
0.38
0.62
ASVD (3023)
(LLD
7.23
10.65
9.58
0.35
0.75
0.73
0.57
0.49
0.75
0.34
0.56
SVD-LLM (2024)
(045(111))
8.34
14:77
12.89
0.31
0.67
0.66
0.53
0.45
0.72
0.22
0.51
MoE-SVD (Ourd)
(086(118)
4.55
10.13
9.93
0.39
0.77
0.73
0.63
0.55
0.78
0.35
0.50
MoE-SVD! (Ours)
(0
4.29
10.99
8.81
0.66
0.79
0.36
0,61
ASVD (2023)
(21,51133)
14.51
22.14
21.92
0.69
0.41
0.68
0.25
0.48
SVD-LLM (2024)
(227(1.3k)
38.83
68.52
43.81
0.36
0.39
0.43
40%
MoE-SVD (Ours)
(248(1.3))
11.73
11.94
0.72
0.58
0.31
0.56
(202)
(30.4(135))
20.55
33.53
30.26
0.62
0.36
0.24
0.44
SVD (2024)
129661)3
6194.87
6451.79
9345.47
0.29
0.27
0.20
0.31
50%
MoE-SVD (Ours)
(37
6.24
13.99
14.99
0.65
0.09
0.53
0.72
0.30
0.53
1388(1440))
107.71
208.69
161.40
0.40
0.30
0.59
0.23
0.35
SVD-LLM (2024)
(40.3(144))
7168.00
7101 49
7119.43
0.28
0.51
0.36
0.34
0.21
0.31
60%
MoF-SVD (Ouns)
(487(1)5)
7.46
20.5
21.85
0.60
0.68
0.46
0.40
0.71
0.25
0.49
Table 3: Results under 20% compression ratio on language modeling and reasoning datasets.
Model
Rustime
WikiTza-21
PTB.
C4L
Opents
ARC_e
Winoi.
HellaS.
ABC_
PIQA
MatlQAA
Averaget
DeepSeekMoll- 16B Original
$2.53
6.38
9.47
9.82
0.33
0.76
0.71
0.58
0.44
0.79
0.31
0.56
DrerSeokMeE-100 MoF-SVD
(1.21x)
10:48
0.31
0.35
0.53
0.31
0.54
Mistral-&s22B Original
16.34
2.95
10.1
6.14
0.37
0.36
0.80
0.67
0.59
0.83
0.50
0.66
Mistral-se338 MoF-SVD
78.18 (1.188)
4.0
15:02
8.73
0.37
0.54
0.79
0.1
0.54
0.81
0.44
0.63
OwenZ 57B A14B Original
42,70
4.32
11.66
9.23
0.33
0.75
0.74
0.65
0.47
0.8
0.39
0.58
Owen2 570 774- Mol. SVD
53.16 (1.25x)
5.41
13.25
11.63
0.30
0.74
0.73
0.6
0.45
0.78
0.35
0,56
substantial reduction in the number of parameters, especially when r < min(m, n). The parameter reduction ratio is 2mr+rn approximately Nmn which can be significant for LLMs with high input and output dimensions.
4.1. Experimental Setups
To showcase the versatility of our MoE-SVD method, we assess its effectiveness on Mixtral models (8x7B and 8x22B). Phi-3.5-MoE. and DeepSeek- MoE. Mixtral variations employ 8 experts, achieving re- markable language modeling capabilities. Phi-3.5-MoE excels with 16x3.8 B parameters, while DeepSeek-Mofr with 16 B parameters utilizing fine-grained experts, also ex- hibits superior performance. We evaluate our method across 10 datasets, encompassing 3 language modeling datasets (WikiText-2 (Merity et al., 2017), PTB (Mareus et al., 1993), and C4 (Raffel al., 2020)), along with common sense reasoning datasets (OpenbookQA (Mihaylov et al., 2018), WinoGrande (Sakaguchi et al., 2020). HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), MathQA (Amini et al., 2019), ARC-e, and ARC-c (Clark et al., 2018)) in a
Models and Datasets.
4. Experiments
In this section, we first compare MoE-SVD against other compressors on Mixtral-8x7P and Phi-3.5-Mod. at differ- ent compression ratios. Then, we conduct evaluation on diverse MoE LLMs, inference efficiency, ablation studies and extend MoE-SVD with quantization. All experiments are conducted on NVIDIA H800 GPUs.
6

---

Structured Mixture-of- LLMs Compression via Singular Value Decomposition
60%
50%
40%
30%
20%
Original
(a)
Phi 3.5 MoE on GPU
(b)
Mixtral 8x7B on GPU (c) Phi 3.5 MoE on GPU
(d) Mixtral 8x7B on GPU
175
225
Tokens/Sev seens/see
Tokens/see secc
150
Tokens/ sec
Tokens/scc
200
150
125
175
100
100
150
75
125
100
50
16
256
512 Sequence Length
Batch Size
Batch Size
Sequence Length
Figure 4: Throughput (Tokens/sec) of Mixtral-8x7B and Phi-3.5-Ma compressed by MoE-SVD at 20%~60%% ratios on single H800 GPU is compared Figures (a) & (b) for various batch sizes at sequence length = 32, and in Figures (c) & (d) for varying sequence lengths at batch size = 64.
Total Memory (GBB
Weight Memory (GB)
4.2. Performance and Acceleration Results
Phi 3.5 MoE
(b) Mixtral 8x7B
Memory Usage (GB)
Mcmory Usage (GB)
Performance Comparisons. Results Table 2 demon- strate that our MoE-SVD method maintains performance close to the original model across various compression tios, with minimal degradation in perplexity and common- sense reasoning accuracy. For instance, at a 20% com- pression ratio, MoF-SVD achieves comparable results to the Original, particularly common-senee reasoning tasks. This highlights the robustness of our method in preserv- ing model performance even under significant compres- sion. Compared to other compression methods, including Wanda, SparseGPT, MC-SMoE, and SVD-based approaches (ASVD, SVD-LLM), MoE-SVD consistently achieves bet- ter performance across both language modeling and reason- ing datasets. For example, at 20% compression, MoE-SVD outperforms SparseGPT and Wanda both perplexity and average accuracy. Additionally, MoE-SVD achieves sig- nificant improvement over ASVD and SVD-LLM in all evaluated metrics, with lower perplexity and higher accu- racy, particularly in challenging reasoning tasks such as ARC-e and HellaSwag, At a 40% compression ratio, MoE- SVD achieves 0.50 average accuracy of, significantly out- performing SVD-LLM (0.36) and ASVD (0.36), while at higher 60% compression, MoE-SVD still achieves 0.37 average accuracy, maintaining clear performance margin compared to SVD-LLM (0.30) and ASVD (0.30). This con- sistent superiority across varying sparsity levels highlights the robustness and adaptability of MoE-SVD in handling large-scale compression challenges. These results under- score the effectiveness of our method delivering superior compression performance compared to existing approaches. Furthermore, by leveraging LoRA fine-tuning, MoB-SVDi further improves upon the already strong results of MoE- SVD. At 20% compression ratio, MoE-SVD achieves an average accuracy of 0.60, notable gain over MoE-SVDI 0.58, while also reducing perplexity across language model-
173
140
1.50
120
100
100
80
60%
50% 40% 30% Origini Compression Ratio
605 Compression Ratio
50%
40%
30%
20% Original
Figure 5: Memory usage (GB) of MoE-SVD at varying compression ratios.
[a] Change Number
(b) Change Seed
4.
4.
widerd Partesty
Mumber data M4048 sokend))
Beod Random Sampling
Figure 6: Perplexity of 20% compressed Mixtral-8x22B via calibration data with varying number (a) and seeds (b) from WikiText-2 and C4.
zero-shot setting using the LM-Evaluation - frame- work (Gao et al., 2023).
Implementation Details. For fair comparisons, we fol- lowed the same settings as ASVD and SVD-LLM and used 256 random samples from WikiText-2 as calibration data. We focus on compressing the model without retraining the full model parameters. See Appendix for more details.

---

Structured Mixture-of- Expee-rrssx--sssss--e- LLMs Compression via Singular Value Decomposition
Table 4: Performance of different decomposition settings on Mixtral-8x7B.
Method
WikiText-21
PTBL
C4|
Openb.
ARC_e
WinoG.
HellaS.
ARC_e
PIQA
MathQA
Averaget
Original
3,98
12,99
6.78
0.36
0.84
0.76
0.65
0.57
0.82
0.43
0.63
SVD (Uniform)
18.80
80.04
23.92
0.24
0.57
0.58
0.43
0.33
0.68
0.23
0.44
SVD (OWL)
16.57
62.13
30.82
0.26
0.61
0.64
0.45
0.32
0.68
0.29
SVD (Our Selective)
8.67
26.72
12.05
0.24
0.67
0.66
0.48
0.35
0.72
0.28
0.49
MVD
4,86
19.42
8.98
0,33
0.79
0.74
0.56
0.49
0.78
0.37
0.58
Table 5: Perplexity (4) of our MoE-SVD with various selec- tive metries for Mixtral-8x7B on WikiText-2.
Table 7: Performance of Mixtral-8x7B with MoE-SVD der 20% compression ratios using calibration data randomly sampled from WikiText-2 (by default in our paper) and C4.
Metrics
f;
P:
Perplexity 12.65
9.85
10.23
9.27
9.31
8.67
Chamany
Wale Bourz1
FIB
19:33
506
Ogen
ABC
Winki
6104
0.6
E. 6.55 ARC
0.20
PIIA
638
19:38
679
672
0.6
2.99
Table 6: Perplexity (1) of our MVE-SVD with various num- bers of trimmed matrices for Mixtral-8xl on WikiText-2.
Table 8: Perplexity (L) of Mixtral 8x7B and Phi-3.5-MoF compressed with GPTQ and MoE-SVD on WikiText-2.
U trimming
2
3
Mistral 8x7B
GPTQ (4bi))
MoB-SVD (4bit
Mob-SVD (Sbit)
Perplexity
4.86
5.21
5.98
6.47
7.11
8.67
9.81
10.25
Memory
44.5
33.4
35.6
26.7
Perpixiity
4.35
6.22
6.93
11:53
Phi-3.5-Mot:
GPTQ (4bin
GPTQ (3bmn
Mob-SVD
MoB-SVD
Memory
39.0
29.3
27.3
20.5
ing datasets. This enhancement demonstrates the potential of fine-tuning techniques to refine and optimize compressed models, further bridging the gap with the original model.
Perlexitt
4.59
6.71
6.64
benefits of faster inference and reduced memory usage, mak- ing MoE LLMs more efficient and accessible.
Generalizability of Across Diverse MoE. Architectures. To demonstrate the broad applicability of MoE-SVD, we evaluate its performance on two distinct MoE models, DeepSeck-MoE-1663 and Mixtral-8x22 -x2rB, under 20% com- pression ratio, as shown in Table 3. The results indicate that compressed models retain competitive performance, with MoE-SVD achieving an 0.54 average accuracy on DeepSeek-Moie MMoSecckMME-168 16B and 0.63 on Mixtral-8x221 compared to 0.56 and 0.66 for their respective original versions. These results highlight the ability of MoE-SVD to preserve sub- stantial portion of the original model's reasoning capabilities across diverse datasets and architectures. Similarly, for the Qwen2-57BB 5-22-577-AI A14B model, MoE-SVD achieves 0.56 aver- age accuracy, closely matching the original model. This demonstrates the robustness and generalizability of MoE- SVD across varying MoE architectures, enabling efficient compression without significant loss in performance.
4.3. Ablation study
A Ablation of Selective Decomposition. Table 4 delves into the performance of different selective decomposition meth- ods. our non-uniform decomposition metric outperforms both uniform SVD and the OWL-based non-unifomn SVD method for compressing MoE. Table 5 our metrics can ob- tain better performance than vanilla ag T; and fs, respec- tively, demonstrating the importance of ensemble designs. In addition, our matrix sharing and trimming (MoE-SVD in Table 4) can further reduce the parameter redundancy allowing us to retain more sensitive expert layers, which leads to significant performance gains based on our selective decomposition.
Varying Numbers for Low-rank Matrix Trimming, Our U-matrix trimming serves the same purpose of performance- efficiency tradeoff as other expert pruning. Note that our trimmed U-matrix is derived from decomposable layer, which intrinsically contains more parameter redundancy and contributes slightly smaller to overall performance com- pared to the other layers. Ablation in Table 6 shows that retaining the Top-2 U-matrix with the highest frequency and trimming the rest 6 U-matrices results in an ideal trade-off and greatly reduces the number of parameters.
Inference Speed Acceleration and Memory Reduction. Figure 4 shows significant inference speed up for Phi-3.5- MoE and Mixtral-8x7B models as compression ratios in- crease, with Phi-3.5-MoE achieving up to 1.52x acceleration at 60% compression and Mixtral-8x7B reaching 1.53x at the same compression level, demonstrating consistent gains across batch sizes and sequence lengths. Meanwhile, Fig- ure 9 highlights MoE-SVD's memory reduction capabilities, with Phi-3.5-MofF
weight memory decreasing to 67.78 78 GB (43,45% of original) and Mixtral-8x7B to 7 GB (40.41% % of original) at 60% compression, enabling deploy- ment on memory- -imited devices and broadening real-world applicability. These results underscore MoE-SVD's dual
Impact of Calibration Data. Table 7 examines the impact of different calibration data sources and results indicate that the choice between WikiText-2 and C4 has minimal impact on the overall performance across various tasks. Figure
8

---

Structured Mixture-of-Expers LLMs Compression via Singular Value Decomposition
explores the effects of varying the number of calibration samples and random seed. Results indicate that increasing the number of data samples generally leads to a decrease in perplexity, suggesting improved performance with more samples. Additionally, the choice of random seed shows minimal effect on perplexity across both datasets, demon- strating that our MoE-SVD relatively robust to sampling variability.
and the research funding under HKUST-DXM Al for Fi-
nance Joint Laboratory (DXM25EG01).
Impact Statement
This work presents MoE-SVD, a technical framework fo- cused on compressing Mixture of Experts (MoE) models to improve inference speed and reduce memory usage. The research is conducted using open-source benchmarks and does not involve sensitive data, ethical concerns, societal risks. The primary impact lies in enhancing the efficiency of large language models, enabling their deployment on pnoourc--oostrainge devices and broadening their accessi- bility. As such, this work is purely technical and does not pose ethical or societal challenges.
Expanding MoE-SVD via Quantization. The results in Table 8 demonstrate the results of combining MoE-SVD with GPTQ (Frantar et al., 2022) to achieve significant mem- ory savings. Comparing 4-bit and 3-bit quantization levels, our MoE-SVD (4-bit) proves to be on par with direct 3-bit quantization (e.g., GPTQ (3bit) in terms of both memory ef- ficiency and performance. These findings underscore the ef- fectiveness of the quantization method when combined with MoE-SVD, showcasing its potential for creating memory- efficient models without compromising performance quality.
References
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziores R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation based formalisms. In NAACI-HI.T ()), pp. 2357-2367. Association for Com- putational Linguistics, 2019.
5. Conclusion
In this paper, we introduce MoE-SVD, a novel SVD-based compression framework tailored for MoE LLMs, effectively streamlining parameter expenses, and memory usage while upholding performance. To combat decomposition collapse stemming from matrix redundancy, we propose innovative solutions, including the selective decomposition strategy and a low-rank matrix sharing and trimming mechanism. The former utilizes a sensitivity metric for automated iden- tification of decomposable layers, while the latter harmo- nizes parameter efficiency and expert specialization through V-matrix sharing and U-matrix trimming. Our extensive as- sessments on Mixtral-8x7B -x7B and Phi-3.5-MoE models show- case the method's superiority over existing compressors in preserving model capabilities across diverse tasks. These promising results, encompassing preserved performance, accelerated inference speed, and substantial memory reduc- tion, position MoE-SVD as a significant stride forward in making MoE LLMs more accessible and efficient for real- world applications, paving the way for widespread adoption and deployment of these powerful models.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In AAAI, pp. 1432-7439, AAAI Press, 2020.
Cai, W., Jiang, J., Qin, L.., Cui, J., Kim, S., and Huang. J. Shorteut-conneeted expert parallelism for accelerating ofxture-of-experts. arXiv preprint arXiv:2404.05019, 2024a.
Cai, W., Jiang, J., Wang. F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts. arXiv preprint arXiv:240..06204 2024b.
Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed lan- guage models. In International conference on machine learning, pp. 4086-4086. PMLR, 2022.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, 0bs//880300545 2018.
Limitations: As a common phenomenon. SVD-based meth- ods (e.g., ASVD and SVD-LLM) add communication over- head during tensor parallelism. Nevertheless, our substantial reduction in computation time ultimately optimizes the total computation time, as detailed in the Appendix B.4.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of- lan- guage models. arXiv preprint arXiv:2401 1.6606666666,,.. 2024.
Acknowledgements
This work is funded in part by the HKUST Start-up Fund (R9911), Thsme-based Research Scheme grant (T45- 205/21-N), the InnoHK funding for Hong Kong Generative AI Research and Development Center, Hong Kong SAR,
DeepSeek-Al et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19453 2024.
Dong, P., Li, L., and Wei, Z. Diswot: Student architecture
search for distillation without training. In CVPR, 2023a.

---

Structured Mixture-of-Expert LLMs Compression via Singular Value Decomposition
Dong, P., Li, L., Wei, Z., Niu, X., Tian, Z., and Pan, H. Emq: Evolving training-free proxies for automated mixed precision quantization. arXiv preprint arXiv:2307.10554 2023b.
He, S., Dong, D., Ding, L.,, and Li, A. Demystifying the com- pression of mixture-of-expert through unified frame- work. arXiv preprint wrXiv:2446.025000 2024.
Hsu, Y., Hua, T., Chang, S., Lou, Q.. Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. ICLR. OpenReview.net, 2022.
Dong, P., Li, L.,, Tang, Z., Liu, X., Pan, X., Wang, Q., and Chu, X. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In ICML, 2024.
Huang. W., Liao, Y., Liu, J., He, R., Tan, H., Zhang, S., Li, H., Liu, S., and Qi, X. Mc-moe: Mixture compressor for mixture-of-expere Ilms gains more. arXiv preprint ar arXiv:2410002270.00 2024.
Dong, P., L., Tang, Z., Liu, X., Wei, Z., Wang, Q., and Chu, X. Parze: Parametric zero-cost proxies for efficient nas. In AAAI, 2025a.
Dong, P., Li, L., Zhong, Du, D., Fan, R., Chen, Y., Tang, Z., Wang, Q. Xue, W., Guo, Y., et al. Stbllm: Breaking the 1-bit barrier with structured binary Ilms. In ICLR, 2025b.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C.. Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Seao, L., Gervet, T., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICMI., 2023.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Lee, J., Qiao, A., Campos, F., Yao, Z., He, Y., et Stun: Structured- seeeeedtthn--usssseeee iiiennunstruttrred pruning for scalable moe pruning. arXiv preprint wrxXiv22499062// 2024.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf. H.. Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 12 2023. URL. https::////. /tt////eoodddo// arggrreoods///// 10256836.
Li, L. Self-regulated feature learning via teacher- free feature distillation. In ECCV, 2022.
Li, L. and Jin, Z. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeulPS, 2022.
Golub, G., Hoffman, A., and Stewart, G. gen- eralization of the eckart-young-irsky matrix approximation theorem. Linear Algebra and its Applications, 88 88-89:317-327, 1987. ISSN 0024-3795. doi: http://bb..r////0000000444779999 URL
Li, L., Dong, P., Wei, Z., and Yang, Y. Automated knowl- edge distillation via monte carlo tree search. In ICCV, 2023a.
Li, L., Bao, Y., Dong, P., Yang, C., A., Luo, W., Liu, Q. Xue, W., and Guo, Y. Detkds: Knowledge distillation search for object detectors. In ICML, 2024a.
/om/ scieeneeaariicee///////////////1
Gu, H., Li, W., Li, I., Qiyuan, 7., Lee, M., Sun, S., Xue, W. and Guo, Y. Delta decompression for moe-based Ilms compression. In ICML, 2025.
Li, L., Dong, P., Li, A., Wei, Z., and Yang, Y. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. NeulPS, 2024b.
He, S., Ding, L., Dong, D., Liu, B., Yu, F,, and Tao, D. PAD-net: An efficient framework for dynamic net- works. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14354-14366, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https:
Li, L., Peijie, Tang. Z., Liu, X., Wang, Q.. Luo, W., Xue, W., Liu, Q. Chu, X., and Guo, Y. Discovering sparsity al- location for layer-wise pruning of large language models. In NeulPS, 2024c.
Li, L., Sun, H., Li, S., Dong, P., Luo, W., Xue, W., Liu, Q., and Guo, Y. Auto-gas: Automated proxy discovery for training-free generative architecture search. ECCV, 2024d.
doi: 10.18653.1/20233aallooomo //aclantho lgg.ollgo............ acl hwwp://22...111nnnoooooe 803.
10

---

Structured Mixture-of-Expers LLMs Compression via Singular Value Decomposition
Li, L., Wei, Z., Dong, P., Luo, W., Xue, W., Liu, Q. and Guo, Y. Attnzero: efficient attention discovery for vision transformers. In ECCV, 2024e.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In ICLR (Poster). OpenRe- view.net, 2017.
Li, P., Jin, X., Cheng, Y., and Chen, T. Examining post- training quantization for mixture-of-experts: A bench- mark. arXiv preprint 1rXiv:2406.08155 2024f.
Mihaylov, T.,, Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? A new dataset for open book question answering. In EMNLP, pp. 2381-2391. Association for Computational Linguistics, 2018.
Li, P., Zhang, Z., Yadav, P., Sung, Y.-L., Cheng, Y., Bansal, M., and Chen, T. Merge, then compress: Demystify ef- ficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Represen- tations, 2024g. URL httph: /ate:://pppneeee /pwp//ppee.ew....///// nett/ forum? weeiideeeFeeyy 3WK.
MiniMax, Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., Jiao, E., Li, G., Zhang, G., Sun, H., Dong, H., Zhu, J., Zhuang, J., Song, J., Zhu, J., Han, J., Li, J., Xie, J., Xu, J., Yan, J.. Zhang. K., Xiao, K., Kang, K., Han, L.,, Wang, L.,, Yu, L., Feng, L.. Zheng, L.,, Chai, L.,, Xing, L.,, Ju, M., Chi, M.. Zhang, M., Huang, P., Niu, P., Li, P., Zhao, P., Yang, Q. Xu, Q. Wang. Q. Wang, Q., Li, Q. Leng, R., Shi, S., Yu, S., Li. S., Zhu, S., Huang, T., Liang. T.. Sun, W., Sun,
Li, W., Li, L., Lee, M., and Sun, S. Als: Adaptive layer spar- sity for large language models via activation correlation assessment. In NeulPS, 2024h.
Li, Y., Yu, Y., Zhang, Q., Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large language models based on low-rank and sparse approxi- mation. In ICML, 2023b.
W., Cheng, W. Li, W., Song, X., Su, X., Han, X., Zhang, X., Hou, X., Min, X., Zou, X., Shen, X., Gong, Y., Zhu, Y., Zhou, Y., Zhong, Y., Hu, Y., Fan, Y., Yu, Y., Yang.
Y., Li, Y., Huang, Y., Li, Y., Huang, Y., Xu, Y., Mao, Y.. Li, Z., Li, Z., Tao, Z., Ying. Z., Cong, Z., Qin, Z., Fan, Z. Yu, Z., Jiang, Z., and Wu, Z. Minimax-
Li, Y., Yu. Y., Zhang, Q., Liang, C., He, P.. Chen, W., and Zhao, T. LoSparse: Structured compression of large language models based on low-rank and sparse approximation. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 20336- PMLR, 23-
Scaling foundation models with lightning attention, 2025. URL https://arxies httaaaxivv.ogggeeeee /2500 2551.08313...
Raffel, C.,, Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-tex trans- former. J. Mach. Learn. Res., 21: 21:140:1111: 1-10::67, 2020.
29 Jul 2023c. URL /ttttt::/ppooeeeeeeeeeeee http///ooceeenn..../// mlrr press/vee wwtpss/v222/10 htt////22221/2///// html.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, pp. 8732-8740. AAAI Press, 2020.
Liu, B., Ding, L.., Shen, L.,, Peng, K., Cao, Y., Cheng, D., and Tao, D. Diversifying the mixture- ofaeee-of--eee of-experts repre- sentation for language models with orthogonal optimizer. arXiv preprint 2:X002210 2023.
Song, Y., Mi, Z., Xie, H., and Chen, H. Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023.
Liu, E., Zhu, J., Lin, Z., Ning, X., Blaschko, M. B., Yan, S., Dai, G., Yang, H., and Wang. Y. Efficient expert pruning for sparse mixture-of-experts language models: Enhanc- ing performance and reducing inference costs. arXiv preprint #xXiv22407.00945 2024a.
Sun, M.. Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1et::2.,..169,,, 2023.
Team, Q. Qwen2.5 technical report. arXiv preprint arX1v:2412.15115, 2024.
Liu, J., Gong, R., Zhang, M., He, Y., Cai, J., and Zhuang, B. Me-switch: memory-effictte expert switching framework for large language models. arXiv preprint Tttl:24006090441 2024b.
Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd- I1m: Truncation-aware singular value decomposition for large language model compression. arXiv preprint trXiv:2403.07378 2024.
Lu, x., Liu, Q. Xu, Y., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H. Not all experts are equal: Efficient expert pruning and skipping for coxture-offexperns large language models, 2024.
Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. Moe- infinity: Activation aware expert offloading for efficient moe serving. arXiv preprini arXiv:2401. 143611 2024a.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of english: The penn treebank. Comput. Linguistics, 19(2):3333330, 1993.
Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. Moe- infinity: Activation- aware expert offloading for efficient moe serving, 2024b.

---

Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Yang, C.,, Sui, Y., Xiao, J., Huang, L.., Gong, Y., Duan, Y., Jia, W., Yin, M., Cheng, Y., and Yuan, B. Moe-i2: Compressing mixture of experts models through inter- expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv: 1.01016, 2024.
Yin, I.,, Wu, Y., Zhang, Z.,, Hsieh, C.-Y., Wang, Y., Jia, Y., Pechenizkiy, M., Liang, Y., Wang, Z., and Liu, S. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05175, 2023.
Yuan, Z., Shang, Y., Song, Y., Wu, Q. Yan, Y., and Sun, G. ASVD: activation-awar singular value decompo- sition for compressing large language models. CoRR, abs/2312.05821, 2023.
Zellers, R., Holtzman, A.. Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? In ACL (1), pp. 4791-4800. Association for Computational Linguistics, 2019.
Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A. M., Le, Q. V., Laudon, J., et al. Mixture-of- experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
12

---

Structured Mixture-of-Experss LLMs Compression via Singular Value Decomposition
Appendix
Our appendix provides additional information and in-depth analysis to supplement the main content of the paper on MoE-SVD. is organized into two main sections: further discussions and implementation details. The discussion section covers innovation, advantages and implications. disadvantages, and social implications of our proposed method. The implementation details section includes algorithm table and specific implementation considerations.
A. Further Discussions
A.1. Ethics Statement
We focus solely on developing efficient techniques for Large Language Models (LLMs), utilizing publicly available datasets and models. Our research not designed to address human ethics or privacy concerns directly, Instead, we concentrate on improving the computational efficiency and deployment capabilities of existing MoE LLMs, which may indirectly contribute to broader accessibility and utilization of these powerful models.
A.2. Reproducibility
We affirm the solid reproducibility of our results and provide specific code implementations in the appendix. Our main experiments represent average outcomes from multiple repetitions, ensuring reliability. MoE LLMs, being very large models, exhibit relatively small variances in experimental results and evaluations. To further demonstrate the robustness and repeatability of our method, we present detailed results for different initial seeds, showcasing consistent performance across various conditions.
A.3. Detailed Comparison with Other Work
As shown in Table 1, our MoE-SVD method focuses on large-scale MoE architectures and avoids fine-tuning while employing advanced strategies like activation SVD and selective decomposition. Other methods either target dense model (Dong et al., 2024; Li et al., 2024c;h), smaller architectures (Li et al., 2024d;e), require retraining, or address different aspects of model compression (e.g., quantization (Dong et al., 2025b; 2023b), AutoML (Dong et al., 2025a; 2023a), distillation (Li et al., 2024a;b; 2023a; Li & Jin, 2022; Li, 2022)). In addition, we conducted comparisons with these methods where feasible and included the results in the revised manuscript. However, our primary comparisons are with SVD-based methods (ASVD, and SVD-LLM), which are directly applicable to MoE architectures and closely aligned with our methodology. These comparisons demonstrate the clear advantages of our approach. Finally, we extra individually discuss and contrast other methods primarily target different architectures. employ distinct strategies, or address challenges unrelated to those in our study:
STUN (Lee et al., 2024) lacks available code implementation details, it recently published on arXiv. Without the ability to replicate its results, it remains inaccessible for direct comparison.
Wanda and SparseGPT are sparsity- based pruning methods designed for dense architectures, not Mixture-of offetrrr----xpprets (MoE) models. These methods do not utilize Singular Value Decomposition (SVD) address the unique challenges of large-scale MoE architectures, such as expert diversity and weight redundancy.
LoSparse and MC-MoE (Huang et al., 2024) focus on small-scale models, such as BERT-base (1 parameters) and Switch-base-32 (2.0B parameters), whereas our method is designed for large-scale MoE models (16B-57B parameters). Additionally, these methods rely fine-tuning and knowledge distillation, whereas our approach avoids such computationally expensive steps.
MoE-Compres CoEprsppresionn (He et al., 2024) employs sparsity and layer-dropping techniques, which are fundamentally different from our SVD-based strategy. This framework focuses on compressing small medium-sized models, making it less relevant to our focus large-scale MoE architectures.
A.4. Detailed Comparison with Fine- Tine-Tuning Compression Methods
We also compare our method with Aine-tuning-buut based approaches like MC-SMoE, Table 9 highlights the key differences between our method and MC-SMoF.
13

---

Structured Mixture-of-Exxe Experts LLMs Compression via Singular Value Decomposition
Table 9: Comparison of MoB-SVD with MC-SMoE (Li et al., 2024g).
MU-ASMAR Bu 304g)
Maaz sege recine
Kegg: Rin segros
Lnneres spoe ea01s
Noeh Exe-sang
Prog youoll merge
DSleue 10000,, bestcchee de ae coube
Models
Phi 3.5 MoE and Mixtral Sensitivity Across Layers
0.5
Phi Mixtral
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
28
27
28
23
89
31
Layer
Figure 8: Sensitivity scores of Mixtral-8x7B and Phi-3.5-MoE accrossing layers
Our approach differs significantly from MC-SMoE in terms of methodology and application. While MC-SMoE relies on merging experts and fine-tuning, MoE-SVD employs selective decomposition tailored for large-scale MoE architectures without requiring fine-tuning, As shown in Table 9, our method achieves superior performance and scalability, handling models with up to 57B parameters.
A.5. Selective Decomposition Results
We detail the results of our selective decomposition strategy in Table 10. These results show clear trends: as the compression ratio increases, more layers are selectively decomposed. Middle layers are more likely to be decomposed, while the first and last layers tend to retained, aligning with empirical observations and sensitivity analyses.
Table 10: Decomposed layers of our selective decomposition strategy by varying compression ratios.
Mixtral-&xIB
Phi3.5 MoB
Ratios
Selected Decomposition Expert Layers
Ratios
Selected Decompasitinn Expert Layers
20
[3.566,7,9912222222
12.23,24252
20
[11.12,15.00.2]] 23.
[35.6,799 11..7,99.22213... 12,.222,,2..
30
[11,12, 23,22,2,,7222
[3.566.7.9 12, 2,,,222,,2444 25,22,,4,,2,266,
40
16. 20, 21,21 26,2222,,72228
50
50
156.10..1. 16,11,,,,,,,1111111... 211.1/.........22222222222222222
60
156.10.11.12111155 28..0.1.1222311511.88.2222222222
B. More results
B.1. More Results Per-layer Decomposition and Compression Ratios
Our extended experimental investigations provide deeper insights into the efficacy and behavior of the MoE-SVD compression technique. Figure 8 presents detailed distribution of sensitivity scores across layers for both Mixtral-8x7B and Phi-3.5-MoF: mod- els. This analysis elucidates the varying impact of compression on different layers within the network architecture. Furthermore, we conduct an in-depth examination of perplexity results for each decomposed block of Mixtral-8x77
Comparison of PPL across Different Layers (Leg Scale)
(25, (06)
Wild PPL (Log Sealo)
(IK, 24]
sv (1
20% compression, as illus- trated in Figure These results offer valuable insights into the relationship between compression ratios and model performance. To further optimize our layer selection process, we develop and implement& sophisticated heatmap-based approach, visualized in Figure 10. This method provides more intuitive and data-driven way to identify layers most suitable for compression, enhancing the overall efficiency and effectiveness of our MoE-SVD technique.
Layer
Figure 7: Perplexity of Mixtral-8x7B via 20% per- layer SVD decomposition on WikiText-2.
B.2. More Comparison with Structured Compression Methods
In Table 2, we compare our MoE-SVD against several structured compression methods applied to Mixtral-8x7B. as well as methods specifically targeting expert layer compression (Li et al., 2024g; He al., 2024). Our MoE-SVD achieves
14

---

Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Table 11: Metrics (model size, TFLOPs, runtime) of MoE-SVD. Runtime denotes runtime throughput (Tokens/sec) on a single H800 GPU.
Mixtral-8x7B
Dense
20%
30%
40%
50%
60%
Model-size
46.7B
37.1B
32.2B
28.3B
23B
17.6B
TFLOPs
5.27E+14
4.92E+14
4,40E+14
4.26E+14
3,97E+14
3.67E+14
Runtime
87.73
104.66
106.03
108.83
123.88
156.1
Phi-3.5-MoE
Dense
20%
30%
40%
50%
60%
Model-size
41.9B
33.2B
29B
25.2B
20.6B
16.4B
TFLOPs
2.72E+14
2,46E+14
2.27E+14
2.00E+14
1,82E+14
1.71E+14
Runtime
98.2
108.63
114.8
124.79
137.17
148.7
DeepSeekMoE
Dense
20%
30%
40%
50%
60%
Model-size
6.4B
13.2B
11.4B
9.7B
6.4B
TFLOPs
1.10E+14
1.02E+14
9.88E+13
9.29E+13
9.25E+13
8.82E+13
Runtime
52.53
62.79
94.71
118.93
119.81
128.71
Table 12: Significance test for three repeated experiments of Mixtral-8x7B and Phi-3.5-MoE compressed by MoE-SVD
under 20% compression ratios.
Mixtral-8x7B
Openb.
ARC_e
WinoG.
HellaS.
ARC_e
PIQA
MathQA
MoE-SVD (512)
0.32+0.0199
0.78=0
0.73=0.0129
0.57=0.0050
0.48:0.0146
0.79=0.0096
0.37=0.0087
MoE-SVD (512) +LoRA
0.33+0.0205
0.80=0.0086
0.73=0.0128
0.61=0.0049
0.5520.0145
0.81=0.0095
0.38=0.0084
Phi-3.5-MoE
Open
ARC_e
WinoG.
HellaS.
ARC_e
PIQA
MatiQA
MvF.SVD (512)
0.38+0.0215
0,76=0.0090
0.72=0.0127
0.6340.0049
0.53=0.0146
0.77=0.0099
0.35=0.0081
McE-SVD (512) +LoRA
0.39#0.0213
0.81=0.0080
0.74=0.0123
0.65e0.0041
0.54=0.0142
0. 0.79=0.0095
0.36=0.0084
runtime speedup of 1.2x while maintaining performance across various benchmarks. Specifically, MoE-SVD records lower perplexities on language modeling tasks such as WikiText-2 (4.44) and PTB (15.21) compared to Wanda (4.72 and 18.8) and SparseGPT (4.61 and 21.11). ). On downstream tasks, our method attains the highest average score of 0.58, outperforming (He et al., 2024)'s 0.54 and significantly surpassing LoSparse (Li et al., 2023c) and MC-SMoE (Li al., 2024g), which exhibit substantial performance drops. These results highlight that MoE-SVD not only accelerates inference but also preserves or improves accuracy relative to other methods.
B.3. More Results on Real-time and Significance test
In Tables 11 and 12, present the experimental results of our MoE-SVD method applied Mixtral-8x7B and Phi-3.5-MoE models at 20% ratios. Our approach achieves substantial reductions in model size and computational overhead while maintaining competitive performance. Specifically, MoE-SVD reduces the model size of Mixtral-8x7B from 46.7B to 37.1B IB and improves runtime throughput from 87.73 to 104.66 Tokens/Sec. The significance tests in Table 12 indicate that these improvements are statistically meaningful across multiple runs.
B.4. Detailed Tensor Parallelism and Potential Communication Overhead
Our MoE-SVD method significantly reduces computation time for weight matrices through low-rank decomposition, achieving a 1.2x-1.5x speedup during inference. These results outperform existing structured compression methods on an algorithmic level, confirming the effectiveness of our approach. When combined with hardware optimization techniques such as tensor parallelism, all low-rank decomposition methods, including ours, MC-SMoE [1], ASVD, and SVD-LLM. inherently introduce communication overhead. This trade-off is common phenomenon for such techniques in tensor- parallel scenarios. However, the total optimization between computation time and communication time depends specific hardware configurations.
Our SVD-MoE framework optimizes the trade-off between computation and communication by leveraging low-rank matrix decomposition. The expert weight matrix W E " where = 4096, is decomposed into U E "x and VT H with the reduced rank 1024 representing 25% of the hidden dimension (r H/4). For input activations
15

---

Structured Mixture-of-Expere LLMs Compression via Singular Value Decomposition
X E BR with batch size B 128, the matrix multiplication Y XW replaced by a two-stage process, Y XUVT,
introducing intermediate activation Z XU Bxr
The computational efficiency is evident in the following metrics. The original computation requires:
ComputationnSscfor = B x H = 128 x 40962 = 2.15 x 10 operations.
After decomposition, the computation is reduced to:
Computations,,n after 2 B H x = 2 128 x 4096 1024 1.07 x 10 operations.
This results in a 50% reduction in computational cost. However, the decomposition increases communication volume from B x H = 128 x 4096 524, 288 elements to B x (r + H) 128 x (1024 + 4096) - 655, 360 elements, representng a 25% increase.
To analyze this trade-off, we evaluate performance on NVIDIA A100 GPUs, which feature 600 GB/s NVLink bandwidth and 19.5 TFLOPS peak performance (FP32). Communication time is calculated
Total Data (bytes)
Communication Time
Bandwidth (bytes/s)"
The communication time increases from 3.5 10-6 seconds 4.37 10-6 seconds. Meanwhile, computation time decreases from:
Computations 11 x 10 seconds, 19.5 x 1012
to:
Computations 5.5 10 seconds.
19.5 1012
The total processing time improves from 11.0035 x 10-3 seconds to 5.50437 x 10-3 seconds, yielding a net optimization of 5.49913 10-3 seconds.
The marginal increase in communication overhead:
ACommunication Time 0.87 10-6 seconds,
is negligible compared to the substantial reduction in computation time:
AComputation Time = 5.5 x 10-3 seconds.
This demonstrates that the SVD-MoE architecture effectively navigates the trade-off between computational efficiency and communication overhead. The high-bandwidth infrastructure ensures that increased communication does negate the performance gains achieved through decomposition.
We considered conducting additional experiments to further validate these findings. However, performing such experiments was infeasible within the rebuttal period due to the limitations of PyTorch's tensor parallelism, which only supports official models. Adapting tensor parallelism for non-official models like ours would require substantial modifications to the library. Nonetheless, our analysis strongly suggests that the computational benefits of SVD-MoE outweigh the communication costs. We plan to explore these aspects in future work further optimize the performance of decomposed MoE architectures.
C. Pseudocode
In our experimental implementation, we present a detailed algorithmic procedure for compressing MoE-based large language models using the proposed MoE-SVD method. Algorithm 4 outlines the main steps of this approach. The process begins by collecting scaling matrices through forward hooks during inference, as shown in Algorithm (Step 1). This step crucial for capturing activation patterns and computing the sensitivity metric for each expert. Subsequently, we perform singular value decomposition (SVD) on the scaled weight matrices, followed by truncation for effecttvve compression, as detailed in Algorithm 2 (Step 2). Our method introduces V-matrix sharing mechanism, where the most frequently used V-matrix is selected and shared among all experts, described in Algorithm 3 (Step 3). Additionally, we employ U-matrix trimming by retaining the top-k U-matrices based on expert sampling frequencies to refine the expert functions (Step 4), To ensure numerical stability, we apply the adjustment function provided in Algorithm 5, which modifies matrices to be positive definite when necessary. This comprehensive approach enables significant model compression while maintaining performance, effectively addressing the need for efficient large-scale language models.
16

---

Structured Mixture-of- LLMs Compression via Singular Value Decomposition
D. Implementation Details
MoB-SVD optimizes MoE models by selectively decomposing less critical experts to reduce computational complexity while maintaining performance. consists of two main phases: computing a sensitivity metric Sr. for each expert layer during calibration data inference, and decomposing experts.
D.1. Calculation of Sensitivity Score
During calibration data inference. both the sensitivity metric S2 = Σラ= fi Pe - aa and the activation matrices for each expert are collected. The sensitivity metric integrates utilization frequency fe. principal Rank pt, and activation outliers at, where each component is described in detail below:
Sampling Frequency (fi): The variable Ji represents the utilization frequency of the i-th expert, quantifying how often this expert is selected by the router during inference. It is calculated over calibration dataset A' as:
f: Exex Ili E [op)
|
where G(x) is the output of the gating network for input x, [op k) returns the indices of the top k selected experts, [[|] is the indicator function, and X denotes the total number of samples in the dataset. This metric reflects the relative importance of each expert based on its selection frequency.
Principal Rank (pt): The variable p; denotes the principal rank of the i-th expert, which is the number of dominant singular values in the diagonal matrix Σ, obtained from the SVD of the expert's weight matrix W. pi is defined as the number of singular values in E, that exceed a given threshold, effectively capturing the dimensionality of the weight matrix's 's significant components. This rank reflects the structural complexity of the expert's weight representation, with higher values of pa indicating more complex and information-rich weights.
Activation Outliers (a,): The variable o; measures the proportion of activations in the i-th expert that exceed a certain threshold relative to the mean absolute activation value. For a set of activations A, in the i-th expert, a; is computed as:
0g EoEA, T(aa > T Mean((A:1))
(10)
[A
where |A,] denotes the total number of activations for the i-th expert, Mean(|A,)) is the mean absolute value of these activations, and r is a user-defined threshold. This metric highlights the presence of outlier activations indicative of the expert's contribution to the model's capacity. The overall sensitivity metric S aggregates these factors across all N experts in layer, providing a comprehensive measure of the layer's importance.
D.2. Decomposition Process of Expert Matrix
Following ASVD (Yuan et al., 2023) and SVD-LLM (Wang et al., 2024), our framework employs an setivation-weighted SVD that enhances the vanilla SVD by incorporating activation statisties to improve decomposition accuracy. With activation matrix X and original weight W nriginal, we compute the activation-weightee matrix by scaling the original weight matrix based on the activation statistics:
W W originall S,
(11)
where Wn, E R m represents the activation-weigheeee matrix and matrix is obtained through cholesky decomposition of activation gram matrix XX"". We then perform SVD on Waw and final compressed weight matrix is obtained by truncating the smallest singular values:
W aw U Trunc (E) VT. s-
(12)
This activation weighted approach effectively mitigates reconstruction loss from outliers during matrix decomposition while maintaining the essential characteristics of the original weight distribution.
D.3. Model-Specific Configurations
In our experimental realization, we develop tailored post-mccomnonition strategies for various large language models, each with unique architectures. For Mixtral-8x77 which employs experts per layer, we implement a novel approach of sharing
17

---

Structured ceature-of-Experre LLMs Compression via Singular Value Decomposition
Mixtral 877B: Parameter Retention Ratio Heatmap
Phi-a.5-MoE: Parameter Retention Ratio Heatmap
0.1
Retention Ratio @
0.2
S02
0.8
0.3
0.3
0.7
0.4
0.8
0.4
0.8
0.5
0.5
0.5
0.8
044
03
0.3
0.9
0.8
02
0.8
0.9
02
0.1
000
2011000 Number Layers
00000000000000
Number of Layers
Figure 9: Retained parameters calculation for Hix2ral-xx22B.
Figure 10: Retained parameters calculation for Phi-3.5-MoE.
components (v1, v2, v3) from the most frequently activated expert while retaining U components (ul, u2, u3) from the top two most frequent experts. We extend this methodology to Phi-3.5-MoE, featuring 16 experts per layer, by broadening the retention scope. In this instance, we share components from the most frequently selected expert and preserve components from the top four most frequent experts. For the more complex Deepseek 16B, which utilizes 64 experts per block, we innovate further by partitioning the experts into 8 distinct groups. Within each group, we share the most frequent V component and strategically trim the 6 lowest frequency U components. This carefully crafted selective retention and sharing approach enables us to maintain model performance while achieving substantial reductions in both parameter count and computational requirements.
D.4. Computational Efficiency
We rigorously assess the computational efficiency of our LLM compression techniques, recognizing its paramount importance for practical deployment scenarios. Our MoE-SVD compression methodology comprises two distinct phases: activation data collection and SVD decomposition with expert trimming. Through extensive experimentation on the Phi 3.5 MoE model, we meticulously quantify time requirements for various layer counts. Our findings reveal that single-layer processing consumes 266 seconds for activation collection and 107 seconds for SVD and trimming. These durations exhibit non-linear increase. reaching 489 and 209 seconds for two layers, and 689 and 320 seconds for three layers, respectively. In comprehensive 20% layer compression test, observe total time requirement of 44 minutes, with 30 minutes allocated to data collection and 14 minutes to SVD and trimming. These results provide crucial insights into the scalability and efficiency of our approach across different model configurations.
D.5. Scalability Analysis
Our in-depth scalability analysis unveils intriguing patterns in the computational behavior of our compression technique. The activation collection phase demonstrates near-linear time growth with respect to layer count, indicating limited parallelization potential due to the inherent sequential nature of data propagation. However, we emphasize that this collection process one-time operation per model, with the resulting data being storable and reusable for various compression ratios. In contrast, the SVD and trimming phase exhibits promising sub-linear scaling, suggesting enhanced opportunities for parallelization. While our current implementation relies on sequential Python loops, we identify significant potential for efficiency improvements through parallel processing of experts across layers. This aligns seamlessly with the independent operation of experts in different layers of MoE. models, indicating promising scalability prospects for MoE-SVD, particularly in the computationally intensive SVD and trimming phase when applied to large-scale models.
D.6. Potential Parallelization Strategies
To further optimize our approach, we explore a range of potential parallelization strategies. We consider leveraging Python's multiprocessing modules and GPU acceleration frameworks such as PyTorch or TensorFlow to exploit parallel computing capabilities. For models exceeding 100B parameters, we propose the utilization of distributed computing frameworks like Dask or Ray to efficiently scale computation across multiple machines. We hypothesize that this approach could potentially reduce SVD phase time complexity from Omm?? line-r relative to processor count, with the potential scale with
18

---

Structured Mixture-of-Experee LLMs Compression via Singular Value Decomposition
the maximum expert count per layer rather than the total layer count. However, we acknowledge that the effectiveness of these strategies may vary based available computational resources, inter-process communication overhead, and the challenges of expert load balancing in distributed environments. In addressing the research challenges associated with our proposed parallelization strategies, we encounter several non- con-trivial technical hurdles. These include the need to fundamentally redesign algorithms for efficient concurrent processing, develop robust mechanisms for managing complex data dependencies, and optimize resource utilization across heterogeneous computing environments. We emphasize the critical importance conducting comprehensive empirical studies to quantify potential performance improvements across a diverse range of model sizes and hardware configurations. While we anticipate that parallel processing may significantly enhance MoE-SVD's scalability for large language models, maintain cautious stance regarding its effectiveness when combined with our existing matrix sharing and trimming optimizations. We assert that rigorous experimentation and thorough analysis are essential to verify these potential benefits and to fully understand the implications of our proposed parallelization strategies in real-worid, large-scale language model compression scenarios.
Algorithm PyTorch code for sensitivity metric MoE-SVD.
import torch
import torch
def compute layer tau=2.0):
l sew.stiessstttyeeeeeeeeee (experts ( activations, gating_outputa. calibration data,
Compute layer-wise sensitivity metric S for MoE compression
Args:
experts weights (list of torch. Tensor): Weight matrices each expert activations (list of torch. Tensor) Activation values for each expert Beeiing_outpute (torch. Tensor) Router outputs for calibration data calibration data (torch. Tensor) Calibration dataset top_k (int): Number of experts to select per token
tau (float): Threshold for activation outliers
Returns:
float: Layer sensitivity score S_L
experts len (experts weights)
device experts Seiestssweigght[000 device
# Compute sampling frequency (f_i) top_k k k_indices torch, topk torch, (gating outputs, k, dim=-1) (oue-1).indion indices device=device)
torch. zeros
for indices in top k indices:
aopert_counte [indices] +=
expert_coune
# Compute principal rank (r_i) using SVD
F i torch. zeros (num experts, devicesdevice) for i, weight in enumerate (experts_ weights): U, s, V torch. torch.linalg... henalggh Count singular values above threshold
(S) * 1e-2 # Example threshold rill] torch. sum (S threshold)
threshold =
Compute activation outliers torch. (num_ experts, deviccedevvcce) for activation enumerate (activations):
mean abs act torch. mean (torch. abs (activation))
outliers torch. sum (torch. abs (activation) > tau mean abt_act)
a_i outliers activation. numell
# Compute final sensitivity metric S.L.
L torch. sum i ri i))
19

---

Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Algorithm 2 PyTorch code for SVD Expert Decomposition of MoE-SVD.
class MoBSYDCoopressioon:
def init (self, Sruncate_k-Norde top k_ experts==):
Initialize Activation-Neighten SVD with Matrix Sharing and Trimming
Args:
truncate_F (int): : Number singular values keep
exxperts (int): Number of top experts select U-matrix trimming
self. truncate k truncate_K k
self self.top top_kop__ko___o top_k_experteee experts = top_k_experts
def compute wewpihtttiioowwggtttsssssssssss X)
Compute activation-weighted scaling matrix s using Cholesky decomposition
Args:
X (torch. Tensor) : Activation matrix [batch size, feature dim] torch. mm (X, x)t()) is Cumulative activation matrix, representing the of processed activation data.
Compute Gram matrix gram torch.m. mm (X, ())
Cholesky decomposition
torch. www.lli.lggg...... choleskyygrraaeee
def decompose eepee_eepppeeeeeeeeeeeeeeeeeeeee (self, originall, X)
Perform activatiion-weighted SVD single expert
Args:
W original (torch. Tensor) Original weight matrix (torch, Tensor) (torch.Tensor): Activation matrix
**"
Compute activattion-weighted matrix
self. compute. activation wet.iaattoonnoeiighttet weights (X)
aawww torch. mm (W_ (N_originsl, S)
Perform SVD U sigma, v = torch. torch.linalg. www.ll. svd aw, full matrices=False)
Truncate specified truncateuueeekk U(: :self. truncate
not None:
sigma sigmal:self trancttt
v v[:self. trrucattttun :]
return U torch diag (sigma), V,
20

---

Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Algorithm 3 PyTorch code for Matrix Sharing & Trimming of MoB-SVD
class MoBSYDCoopressioon:
def init (self, truncate_k M top k expeets==))
Initialize Activation-Weigiten SVD with Matrix Sharing and Trimming
Args:
truncate_F (int): : Number singular values keep
k_peete (int): Number of top experts select U-matrix trimming
self. truncate k truncate_k
self. self. top_t__pppeeeeeeeeeeeeeeeeeeeeee = top_k_experts
def init (self, truncate truncate_ k=None, top k experts=2):
Initialize Activation-Weightes SVD with Matrix Sharing and Trimming
Args:
truncate_ (int): : Number singular
keep
top_k experts (int): Number of top experts select U-matrix trimming
self. tww.ccaeeeeeeee. k truncate k self talf.to_.k.e k_ experts = top_k_eet e_experts
def compress moe (self, oxpert_weights activations, coating_frequenciee,
Compress MOE using V-matrix sharing and U-matrix trimming
Args:
expert weight_ (list): List of expert weight matrices activations (list): List of activation matrices for each expert routing frequencies (torch. Tensor) Expert selection frequencies
num_experta = len
compressed_expeett []]
Decompose experts
decomposed []] for (aaaeemmm_eeepttaaaaee
range (num experts)
Sigma, V, self. www.ccdeeeseeeeee.. _eppeeteeeeeeteeeeeeeeeeeeeeeeee
activations (i])
decomposed.appent (U, Sigma, V, S))
Select shared V-matrix based on highest routing frequency
max freq idx torch. argmax (routing frequencies)
v shared decomposed freq idx] [2]
Sort experts by routing frequency U-matrix trimming
sorted indices torch. argsort poouting_frequeeeeeeeeeeeeeeeeee
descending=True))
Perform U-matrix trimming and construct compressed experts for 1 in range (num experts)
# Find top-k U-matrices from more frequently used experts
more frequent = [] for in sorted indices if routing {re routing_ frequencies 1] top indices more frequent (:self. top k experts]
if len (top_t indices) self. top_k experts: not enough frequent experts, Lindccendccss top k_indices____ [i]
U-matrix
Combine selected U-matrices and corresponding Sigma matrices U combined torch. zeros_ [= [0])
Sigma_ combined Bigma_co torch. zeros like (decomposed [1] [1])
for idx, expert idx in enumerate (top indices [ teel..topppp exxxerts])
U_ combinne Sigma_ combined Sigma__combined decomposed[expers idx] [1]
(xxpert pixpppett_xxx______p [0]
Reconstruct compressed expert
W compressed torch. mm (torch. mm combined, Sigma Sigma__cominee combined),
torch. mm (V (V_shaeed, torch. inverse (decomposed [i] [3))))
compressed experts........ experts.append append ( (W_compressed]
return compressed_expent experts

---

Structured Mixture-of-Expers LLMs Compression via Singular Value Decomposition
Algorithm 4 SVD-MOE: Expert Decomposition, V-Matrix Sharing, and U-Matrix Trimming
Input: model: Language model, data: Calibration data, device: Device, layers: Selected layers, config: Expert configuration
Output: model_compressed: Compressed model
(Step I: Calibration Processing)
for layer& in layers do
for expert in e do
ft TeTok/G(()).)) X (Update activation matrix)
S; f S, AA";
E, yr +- SVD(expprli))
+ Rank(_));
[Update weight outlier)
W. - (reannezpert;.W))
a, # #(IA)>aW.),
experl;..r. + experl,..r. ferag
[Sep 2: Decompose Experts)
for layer in layers do for expert in f do if experrl,.rr. < then
if S; is not positive definite then L Adjust S:
W. + expert,. W Sj; U,E, VT SVD(W.): ));
Etrane + -Truncate (
expert; t- Eeruneei expert,.V + +V²;
(Seep3 3: V-Matrix Sharing)
(Select shared V-matrix)
V. + arg maxy, f(V);;
(Select top-2 U-matrices)
{Ua.1, V.2] + TopK({U)), = 2):
(Step U-Matrix Trimming and Selection) (Update expert function]
Fa(x)) (U,1E, V22
Algorithm 5 MakePositiveDDefnite: Adjust Matrix to be Positive Definite
Function MakePositiveDefiin
(M, tolerance, max attempts):
Input: M Input matrix; tolerance Small value for adjustment; max_ max_attempts Maximum number of attempts
Output: Mpa Positive definite matrix
Step 1: Symmetrize the Matrix Mym <- M+MT // Ensure the matrix is symmetric 2
Step 2: Check Eigenvalues Compute eigenvalues À of Maym
any X; < then
min((x)) + tolerance Shift negative eigenvalues positive
end
Step Reconstruct Positive Definite Matrix Mp V diag(A) V where V the eigenvectors of Maym
Step 4: Ensure Matrix is Symmetric Mpa M
2
return Mpd
22


Reviews:
[{"review_id": "nDH8IxVk87", "reviewer": "Reviewer_Cmem", "summary": "This paper introduces MoE-SVD, a decomposition-based compression approach specifically designed for Mixture of Experts (MoE) Large Language Models (LLMs). Leveraging Singular Value Decomposition (SVD), the method reduces parameter redundancy and memory requirements without requiring additional training. The authors propose selective decomposition using sensitivity metrics, employing a shared V-matrix across experts and trimming U-matrices through top-k selection. Experiments conducted on various MoE models such as Mixtral, Phi-3.5, DeepSeek, and Qwen2 demonstrate a 60% compression ratio and 1.5× faster inference speed with minimal performance degradation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, **we would really appreciate it if the reviewer considers raising the score.** Please see our responses below one by one：\n\n-----\n\n\n\n**Q1: Explanation of Figure 3**\n\n **A1: We clarify that we already provided detailed explanations in Section 3.2 [Lines 220-239] Decomposable Expert Layer Analysis.** \nFigure 3 shows our decomposition strategy results across different compression ratios, visualising which expert layers are selected for decomposition at each ratio is the best strategy based on formulation 4, revealing how compression targets different layers as the ratio increases. We will revise the figure caption and add clarifications in Section 3.3 in the revision.\n\n\n\n\n\n**Q2: About `a_i` in Equation 4**\n\n **A2:** **As detailed in Section 3.2 [Line 207-211], Appendix D.1  [Line 850-859] and pyhon pseudo-code Algorithm 1  [Line 980-985],** the  $a_i$ signifies the activation outliers for the $i$-th expert, defined as the outlier ratio within the expert's weight matrix. Outliers are identified as values exceeding a predefined threshold, which is set as a multiple of the mean value of the matrix. Mathematically, the outlier ratio is calculated as:\n$$\na_i = \\frac{\\sum_{a \\in A_i} \\mathbb{I}(|a| > \\tau \\cdot \\operatorname{Mean}(|A_i|))}{|A_i|}\n$$\n\n\nwhere $|A_i|$ denotes the total number of activations for the $i$-th expert, $\\operatorname{Mean}(|A_i|)$ is the mean absolute value of these activations, and $\\tau$ is a user-defined threshold. This metric highlights the presence of outlier activations indicative of the expert's contribution to the model's capacity. \n\n**Algorithm: Pyhon pseudo-code for $a_i$.**  \n\n```pseudocode\n    # Compute activation outliers (a_i)\n    a_i = torch.zeros(num_experts, device=device)\n    for i, activation in enumerate(activations):\n        mean_abs_act = torch.mean(torch.abs(activation))\n        outliers = torch.sum(torch.abs(activation) > tau * mean_abs_act)\n        a_i[i] = outliers / activation.numel()\n    \n    # Compute final sensitivity metric S_L\n    S_L = torch.sum(f_i * r_i * a_i)\n    \n\n```\n\n\n\n\n\n**Q3: About Performance Improvements**\n\n **A3:**\n\n **(1)** As shown in Table 2. At 20% compression ratio, MoE-SVD achieves an average accuracy of 0.58 across reasoning tasks compared to 0.51 for ASVD, 0.44 for SVD-LLM and 0.33 for MC-SMoE. This represents a 23% reduction in performance degradation relative to the original model (0.63). At higher compression ratios (50-60%), where other methods experience catastrophic collapse (e.g., ASVD and SVD-LLM reach perplexities >10,000 on WikiText-2), MoE-SVD maintains reasonable performance with a perplexity of 13.52 on WikiText-2.\n\n**(2)** The real-world utility is validated by significant efficiency gains: 1.5-1.8× inference speedup on Mixtral-8×7B at 60% compression making deployment feasible on resource-constrained devices.\n\n**(3)** Ablation studies in Tables 4-6 confirm that each component of our approach contributes meaningfully to its success, with selective decomposition, V-matrix sharing, and U-matrix trimming all playing crucial roles in balancing efficiency and performance.\n\n\n\n**Q4: Generalizability of the Shared V-Matrix**\n\n **A4:**\n\n\n\nThe shared V-matrix is not universal but dynamically determined for each model architecture. To demonstrate this, we conducted comprehensive layer-wise CKA similarity analysis across different MoE architectures:\n\n**Layer-wise Analysis Within Models**: this table shows the layer-wise CKA similarity patterns for U and V matrices within three diverse MoE architectures:\n\n| Model           | Layer Position | U-matrix Similarity | V-matrix Similarity |\n| --------------- | -------------- | ------------------- | ------------------- |\n| Phi-3.5-MoE     | Early (1-10)   | 0.29                | 0.78                |\n|                 | Middle (11-20) | 0.36                | 0.86                |\n|                 | Final (21-32)  | 0.31                | 0.74                |\n| DeepSeekMoE-16B | Early (1-8)    | 0.21                | 0.75                |\n|                 | Middle (9-18)  | 0.26                | 0.82                |\n|                 | Final (19-28)  | 0.25                | 0.71                |\n| Qwen2-57B-A14B  | Early (1-8)    | 0.18                | 0.77                |\n|                 | Middle (9-18)  | 0.23                | 0.83                |\n|                 | Final (19-28)  | 0.16                | 0.73                |\n\nThese findings provide strong empirical evidence that our approach of dynamically selecting V-matrices based on layer-specific router statistics is well-justified. \n\n-----\n\n**Finally,** **we hope our response could address the concerns, and we thank the reviewer again for the helpful comments.** **We genuinely hope that our explanations and efforts can improve the overall evaluation of our work.**", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Figure 3 lacks clear explanations regarding the depicted increases and baselines. Clarifying the baseline used and explicitly describing the nature of the improvements observed would significantly enhance reader comprehension.", "theoretical_claims": "Yes. The ai in equation 4 is still unclear to me even after checking the supplementaries.", "experimental_designs_or_analyses": "The improvements shown in evaluation results are not particularly significant. While the proposed approach successfully reduces memory and parameters, the extent of performance gain or maintenance could be more convincingly demonstrated.", "relation_to_broader_scientific_literature": "This paper provides an interesting and practical solution for compressing MoE architectures in LLMs.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nQuantitative Parameter Reduction Analysis: The paper presents a thorough quantitative analysis in Section 3.3, particularly highlighting the efficiency of parameter reduction via V-matrix sharing and U-matrix trimming.\n\nComprehensive Evaluation: The experimental results are extensive, covering essential aspects such as generalizability, scalability, and multiple ablation studies. This robust evaluation contributes significantly to understanding the efficacy of the proposed method.\n\nClarity and Readability: The paper is well-organized, clearly written, and accessible, which helps readers easily grasp the concepts and methodology.\n\nWeaknesses:\n\nAmbiguity Regarding the Shared Matrix V: It remains unclear whether the shared V-matrix is universal across different models and tasks or sensitive to specific models and tasks. Addressing the generalizability or specificity of this shared matrix explicitly would clarify the practical implications of the method.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses", "questions_for_authors": "See Other Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDCuIWwdZR", "reviewer": "Reviewer_V7cs", "summary": "The paper presents a new compression method (MoE-SVD) for Mixture-of-Experts. The framework introduces a selective decomposition strategy and employs low-rank matrix sharing and trimming. Comprehensive experiments on models like Mixtral, Phi-3.5, DeepSeek, and Qwen2 demonstrate that MoE-SVD achieves faster inference, outperforming other compression methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and constructive feedback.  We have tried our best to address all concerns in the last few days. Please see our responses below one by one：\n\n----\n\n**Q1: About Theoretical Justification**\n\n\n **A1:**\n\n\n**(1)** **Theoretical Foundation of Decomposition Sensitivity**: We have formalized the relationship between expert sensitivity and model performance by analyzing how SVD decomposition affects information flow in MoE architectures. In Section 3.2, we present a detailed theoretical analysis that demonstrates why initial and final layers are particularly sensitive to decomposition due to their role in translating between embedding space and MoE-specific representations. This explains the empirical observations in Figure 1 (left), where decomposing these layers leads to disproportionate performance degradation.\n\n**(2)** **Information-Theoretic Interpretation of Sensitivity Metric**: We have derived our sensitivity metric ( $S_L = Σ f_i · p_i · a_i$) from an information-theoretic perspective please see **A2** in responsing to Reviewer Cmem.\n\nWe show that under certain assumptions, our metric approximates the expected information loss from decomposition:\n\n$E[I(W_i; Y|X)] - E[I(W̃_i; Y|X)] ∝ f_i · p_i · a_i$\n\nwhere $I(W_i; Y|X)$ represents the mutual information between expert weights $W_i$ and model output $Y$ given input $X$.\n\n\n\n**(3)** **Matrix Redundancy Theory**: In Section 3.3, we have added theoretical analysis explaining why V-matrices exhibit higher redundancy than U-matrices in MoE architectures. This analysis builds on prior work in Similarity of Neural Network Representations Revisited (Kornblith et al., 2019), showing that the shared output space constraints in MoE models naturally lead to similar output transformations (V-matrices) while maintaining diverse input transformations (U-matrices) for specialization.\n\n**(4)** **Error Bounds for Matrix Sharing**: We have derived error bounds for our V-matrix sharing approach based on matrix approximation theory. For an expert weight $W_i = U_iΣ_iV_i^T$ with selection rank-r approximation $W_i^{r} = U_i^{r}Σ_i^{r}(V_i^{r})^T$, sharing the V-matrix introduces additional error bounded by: $||W_i - U_iΣ_iV_s^T||_F^2 ≤ ||W_i - W_i^r||_F^2 + ||Σ_i||_F^2 · ||V_i^T - V_s^T||_F^2$ . $V_s$ is the shared V-matrix selected based on router sampling frequency as defined in Equation 5 in the paper.This bound becomes tighter as the similarity between V-matrices increases, aligning with our empirical observations of high CKA similarity (average >0.8) between expert V-matrices.\n\n**(5)** **Compression-Performance Tradeoff Analysis**: We have added a theoretical framework that characterizes the relationship between compression ratio, parameter efficiency, and model performance. This analysis explains why selective decomposition based on sensitivity metrics achieves a better Pareto frontier than uniform compression approaches. We derive performance bounds that predict deterioration patterns at different compression ratios, validating our experimental findings in Table 2.\n\n**Q2: About minimal performance loss**\n\n **A3:**\n\n**(1)** The term \"minimal\" is comparative rather than absolute. We will revise our abstract: \" For 40-60% compression,  MoE-SVD achieves .... while maintaining relative performance superior to other compression methods.\" in the revision.\n\n**(1)** Our claim of \"minimal performance loss\" is relative to other compression methods at the same compression ratio. As shown in Table 2, at 20% compression, MoE-SVD maintains 92% of the original model's average accuracy (0.58 vs. 0.63), while competing methods maintain less than 85%.\n\n\n\n**Q3: Novelty Relative to Prior Work**\n\n **A3:**\n\n\n\n**(1)** In Table 1 and Appendix D, we have already compared and analyzed of MoE-SVD to provide  insights into why our approach succeeds where standard SVD methods (ASVD, SVD-LLM) fail on MoE architectures. \n\n**(2)** Our V-matrix sharing and U-matrix trimming strategies represent a novel approach to exploiting the unique redundancy patterns in MoE models. **We emphasize that these observations and designs not present in existing MoE and SVD methods.**\n\n**(3)** Compared to expert pruning methods (MoE-Compression, MoE-I²), our approach maintains the sparse activation mechanism while making each expert more efficient. This avoids the significant performance drops associated with expert elimination (e.g., 23% accuracy drop when pruning 25% of experts in Mixtral-8×7B).\n\n**(4)** MoE-SVD requires no retraining and is hardware-independent, making it more practical for real-world deployment than methods requiring extensive fine-tuning or specialized hardware.\n\n----\n\n\n\n**Q4: Typos**\n\n**A4:** Thanks for the suggestion. We will fix \"TThese\" and double check and revise all typos in the revision.\n\n------\n\n**Finally, we genuinely hope that our explanations and efforts can improve the overall evaluation of our work.**  We thank the reviewer again for the helpful comments.", "claims_and_evidence": "The central claims are supported by experiments across multiple models and tasks. Table 2 shows MoE-SVD’s superior perplexity and accuracy over baselines, and Figure 4 validates speedup. The ablation studies (Tables 4–6) further substantiate design choices.\n\nHowever, the term \"minimal performance loss\" is ambiguous for higher compression ratios (e.g., 60% compression reduces average accuracy from 0.63 to 0.37 on Mixtral).", "methods_and_evaluation_criteria": "The evaluation criteria, including perplexity on language modeling datasets and accuracy on reasoning tasks, are appropriate and align with standard benchmarks in the field.", "theoretical_claims": "The paper does not present theoretical claims.", "experimental_designs_or_analyses": "The experimental design is sound and thorough. The authors evaluate MoE-SVD on multiple models and datasets, including Mixtral, Phi-3.5, and DeepSeek, and provide detailed ablation studies to isolate the contributions of individual components.", "relation_to_broader_scientific_literature": "This work already contains detailed comparisons of the literature on SVD, pruning, and MoE compression.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents a novel and effective approach to compressing MoE LLMs, addressing key challenges like decomposition collapse and matrix redundancy.\n\n2. The experimental results are comprehensive and demonstrate clear improvements over existing methods.\n\n3. The framework is practical, requiring no additional training and being hardware-independent.\n\n**Weaknesses**\n\n1. The theoretical underpinnings of the method are not explored in depth, which could limit understanding of its generalizability.\n\n2. Clarify \"minimal performance loss\" in the abstract relative to baselines vs. absolute metrics.\n\n3. While MoE and SVD are widely adopted in existing approaches, the novelty of this paper remains limited and requires further enhancement.", "other_comments_or_suggestions": "Typos: \"TThese\" in Figure 1 caption;", "questions_for_authors": "Please refer to the above weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5jd3eO40N", "reviewer": "Reviewer_F3P9", "summary": "This paper introduces MoE-SVD, a new compression framework specifically designed for MoE LLMs. Specifically, they first decompose experts into low-rank matrices via SVD. In particular, they selectively decompose the expert layers based on sensitivity metrics.\n\n\nThanks for the authors' detailed responses. All my concerns have been satisfactorily addressed, and I lean to vote for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for constructive comments, and  recognition. Please see our below responses:\n\n-----\n\n**Q1: Performance Degradation at Higher Compression Ratios**\n\n **A1:**\n\n(1) MoE-SVD outperforms alternatives at all compression levels. At 60% compression: MoE-SVD=13.52 perplexity; ASVD/SVD-LLM>10,000 perplexity on WikiText-2.\n\n(2) Lightweight LoRA fine-tuning (MoE-SVD†) mitigates degradation at 1% of full training cost. At 50% compression: improves accuracy from 0.43 to 0.49, recovering 9.5% original performance.\n\n(3) Following the general phenomenon of performance loss in almost high-ratio compression,  MoE-SVD significantly improves trade-off curve vs. existing approaches.\n\n(4) Task-specific analysis shows reasoning tasks like ARC-e maintain performance at 60% compression, while MathQA shows higher sensitivity.\n\n\n\n**Q2: Theoretical Justification for the Sensitivity Metric**\n\n **A2:**\n\n**(1)** Our sensitivity metric integrates three complementary components with theoretical foundations:\n\n(a) Sampling frequency ($f_i$): expert utilization rate determined by router.\n\n(b) Principal rank ($p_i$): effective dimensionality of expert's weight matrix from matrix approximation theory.\n\n(b) Activation outliers ($a_i$): functional distinctiveness of experts.\n\n\n\n**(2)** **Information-Theoretic Interpretation of Sensitivity Metric**: We have derived our sensitivity metric ( $S_L = Σ f_i · p_i · a_i$) from an information-theoretic perspective please see **A2** in responsing to Reviewer Cmem.\n\nWe show that under certain assumptions, our metric approximates the expected information loss from decomposition:\n\n$E[I(W_i; Y|X)] - E[I(W̃_i; Y|X)] ∝ f_i · p_i · a_i$\n\nwhere $I(W_i; Y|X)$ represents the mutual information between expert weights $W_i$ and model output $Y$ given input $X$.\n\n**(3)** Table 5 validates that combining these components ($f_i · p_i · a_i$) achieves optimal performance (8.67 perplexity) vs. individual components (9.27-12.65 perplexity).\n\n\n\n**Q3: Fixed `k=2` in U-Matrix Trimming and more principled methodology for K selection**\n\n **A3:**\n\n**(1)**  Table 6 shows perplexity trade-offs on Mixtral-8×7B: 4.86 (no trimming) → 5.98 → 7.11 → 10.25 as k increases. k=2 balances performance and speed at high compression.\n\n**(2)** This value aligns with the standard top-k routing mechanism in MoE architectures, which typically activates 2 experts per token. By retaining 2 U-matrices, we maintain consistency with the model's inherent routing structure.\n\n**(3)** We further propose **Adaptive k Selection Framework** to improve U-matrix trimming:\n\n$$k_l^* = \\arg\\min_k {\\mathcal{L}(k) + \\lambda \\cdot \\mathcal{Q}(k)}$$\n\nwhere $\\mathcal{L}(k)$ represents the estimated performance loss from trimming to k matrices, $\\mathcal{Q}(k)$ denotes the parameter count, and $\\lambda$ controls the tradeoff. The performance loss is approximated using the information coverage of retained matrices:\n\n$$\\mathcal{L}(k) \\approx 1 - \\frac{\\sum_{i=1}^k f_i \\cdot \\sigma_i}{\\sum_{i=1}^N f_i \\cdot \\sigma_i}$$\n\nwhere $f_i$ is the expert sampling frequency and $\\sigma_i$ is the sum of singular values, jointly capturing the expert's contribution to model performance. Using Automatic k Determination, k is computed automatically during compression based on calibration data statistics, without manual tuning. For each layer, the algorithm calculates the marginal utility of increasing k and stops when additional U-matrices provide diminishing returns relative to their parameter cost. Our experiments with Mixtral-8×7B demonstrate the effectiveness of this approach:\n\n| Compression   | Method           | WikiText-2 PPL | Runtime (Tokens/sec) |\n| ------------- | ---------------- | -------------- | -------------------- |\n| 0% (Original) | -                | 3.98           | 87.7                 |\n| 40%           | Fixed k=2        | 6.74           | 109.8                |\n| 40%           | Adaptive k       | 6.53           | 107.5                |\n| 40%           | U-matrix merging | 6.83           | 112.3                |\n| 60%           | Fixed k=2        | 13.52          | 156.1                |\n| 60%           | Adaptive k       | 12.91          | 143.5                |\n| 60%           | U-matrix merging | 13.30          | 158.4                |\n\n\n\n**Q4: Alternative V-Matrix Sharing Strategies**\n\n **A4:**\n\nYes. For DeepSeek-MoE and Qwen2 (with inherented organized expert groups),  we actually employ clustering strategy following architecture setting. We will emphasize this point and add more analysis in the revision. \n\n**Q5: About size and layout** \n\n **A5:**\n\nWe will carefully revise all size and layout of some tables and figures in the revision. Thanks.\n\n---\n\n**Finally,** we hope our response could address the concerns, and we thank the reviewer again for the helpful comments.", "claims_and_evidence": "The claims made in this paper are generally supported by experimental evidence.", "methods_and_evaluation_criteria": "The evaluation across language modeling (perplexity on WikiText-2, PTB, C4) and reasoning tasks (accuracy on seven common sense benchmarks) provides a comprehensive picture of model performance across different capabilities.", "theoretical_claims": "The paper doesn't contain formal proofs but does provide theoretical justifications for its approaches.", "experimental_designs_or_analyses": "The experimental designs appear sound and comprehensive. The authors evaluate their method across multiple model architectures and compression ratios. The ablation studies also analyze the contribution of each component.", "relation_to_broader_scientific_literature": "The paper positions its contributions well within the broader literature on LLM compression and MoE optimization.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n1.The paper identifies and addresses the failure of standard SVD methods when applied to MoE architectures. Their  sensitivity metric provides an automated and principled way to identify which expert layers to decompose.\n\n2.The matrix sharing and trimming approach effectively balances parameter reduction with performance preservation. \n\nWeaknesses:\n\n1. The absolute performance degradation at higher compression ratios (e.g., 60%) is still significant. \n\n2. The theoretical justification for the combined sensitivity metric could be strengthened with more analysis of how the three components interact.\n\n3. The methodology for selecting the number of U-matrices to trim could be more principled rather than using a fixed value of k=2", "other_comments_or_suggestions": "Size and layout of some tables and figures need minor refinement.", "questions_for_authors": "1.The paper proposes sharing a single V-matrix across all experts. Have you explored more flexible sharing schemes, such as clustering similar experts and sharing V-matrices within clusters rather than globally? \n\n2.For the U-matrix trimming, you use a fixed value of k=2. What is the justification for this particular value?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lF8ybvPdNU", "reviewer": "Reviewer_cMac", "summary": "This paper decomposes expert layers into low-rank matrices to reduce parameter counts and memory demands in MoE LLMs. The key innovations include a selective decomposition strategy based on sensitivity metrics and a low-rank matrix sharing and trimming scheme. The authors claim MoE-SVD achieves significant compression and faster inference on models like Mixtral, Phi-3.5, DeepSeek, and Qwen2.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback and recognition. We have tried our best to address all concerns in the last few days. Please see our below responses to your concerns and questions one by one.\n\n---\n\n**Q1: More Examples of Matrix Similarity Across MoE Models**\n\n **A1:**\n\n**(1)** We've conducted additional analysis on **Phi-3.5-MoE and DeepSeekMoE** (within inherent expert group) that shows similar patterns of V-matrix redundancy across architectures. **These models exhibit 0.92 and 0. 81 average CKA similarity among V-matrices individuals**, confirming that this property generalizes across different MoE architectures.\n\n**(2)**  Table 3 provides empirical validation that our V-matrix sharing approach works effectively across multiple architectures, confirming that matrix similarity is a general property of MoE models rather than specific to Mixtral.\n\n\n\n**(3)** The theoretical basis for V-matrix redundancy relates to how MoE models are trained. Since all experts process similar input distributions but specialize in different aspects, their output projections (captured by V-matrices) share substantial structure while their internal representations (captured by U-matrices) diverge more significantly.\n\n\n\n**Q2: Refining U-Matrix Trimming Strategy**\n\n **A2:**\n\n**(1)**  Table 6 shows perplexity trade-offs on Mixtral-8×7B: 4.86 (no trimming) → 5.98 → 7.11 → 10.25 as k increases. k=2 balances performance and speed at high compression.\n\n**(2)** This value aligns with the standard top-k routing mechanism in MoE architectures, which typically activates 2 experts per token. By retaining 2 U-matrices, we maintain consistency with the model's inherent routing structure.\n\n**(3)** We further propose **Adaptive k Selection Framework** to improve U-matrix trimming:\n\n$$k_l^* = \\arg\\min_k {\\mathcal{L}(k) + \\lambda \\cdot \\mathcal{Q}(k)}$$\n\nwhere $\\mathcal{L}(k)$ represents the estimated performance loss from trimming to k matrices, $\\mathcal{Q}(k)$ denotes the parameter count, and $\\lambda$ controls the tradeoff. The performance loss is approximated using the information coverage of retained matrices:\n\n$$\\mathcal{L}(k) \\approx 1 - \\frac{\\sum_{i=1}^k f_i \\cdot \\sigma_i}{\\sum_{i=1}^N f_i \\cdot \\sigma_i}$$\n\nwhere $f_i$ is the expert sampling frequency and $\\sigma_i$ is the sum of singular values, jointly capturing the expert's contribution to model performance.\n\nUsing Automatic k Determination, k is computed automatically during compression based on calibration data statistics, without manual tuning. For each layer, the algorithm calculates the marginal utility of increasing k and stops when additional U-matrices provide diminishing returns relative to their parameter cost. Our experiments with Mixtral-8×7B demonstrate the effectiveness of this approach:\n\n| Compression   | Method           | WikiText-2 PPL | Runtime (Tokens/sec) |\n| ------------- | ---------------- | -------------- | -------------------- |\n| 0% (Original) | -                | 3.98           | 87.7                 |\n| 40%           | Fixed k=2        | 6.74           | 109.8                |\n| 40%           | Adaptive k       | 6.53           | 107.5                |\n| 40%           | U-matrix merging | 6.83           | 112.3                |\n| 60%           | Fixed k=2        | 13.52          | 156.1                |\n| 60%           | Adaptive k       | 12.91          | 143.5                |\n| 60%           | U-matrix merging | 13.30          | 158.4                |\n\n- Layer-specific k Distribution: Analysis of the automatically determined k values reveals an intuitive pattern: early and late layers (which our sensitivity metric identifies as more critical) receive higher k values (typically 2-3), while middle layers receive lower values (typically 1-2). This aligns with our understanding of information flow in transformer architectures and confirms that the algorithm captures meaningful layer-wise differences.\n\n\n\nThese results show that  adaptive k selection  consistently outperforms fixed k=2, with Adaptive k providing the best performance by preserving more information from the original expert space, particularly at higher compression ratios but U-matrix merging can find the best tradeoff between Runtime and performance.\n\n\n\n**Q3: Minor Typos and Readability of Table 1**\n\n **A3:**\n\n**(1)** We'll correct the typo \"TThese\" in Figure 1's caption and conduct a thorough proofreading of the entire manuscript to ensure clarity and consistency in revision.\n\n**(2)** Table 1 will be redesigned for improved readability. We'll use a clearer layout with better spacing and more consistent formatting to highlight the key differences between methods.\n\n**(3)** We'll ensure that method descriptions are concise yet precise, using a consistent terminology throughout the table and the rest of the paper.\n\n---\n\n**Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments.**", "claims_and_evidence": "The claims seem generally supported by the experimental results, and ablation studies further validate the contributions of the strategy.", "methods_and_evaluation_criteria": "This paper presents the experiments and comparisons on multiple language task benchmarks following the previous LLM Compression studies such as SVD-LLM and Wanda etc. The evaluation uses standard datasets and metrics, and it would be nice to see some evaluations on more complex tasks.", "theoretical_claims": "MoE-SVD mainly illustrates some of the properties of MoE in SVD decomposition via empirical observations, which are quite interesting to me. However, it lacks theoretical analysis to back up the empirical results.", "experimental_designs_or_analyses": "The experimental designs appear sound. The paper compares MoE-SVD with several existing compression methods and conducts ablation studies to evaluate the impact of different components.", "relation_to_broader_scientific_literature": "The paper adequately discusses related work, including general LLM compressors and MoE-specific compression methods. The introduction part is especially very interesting and carefully introduces the potential challenges in previous work.", "essential_references_not_discussed": "To my knowledge, most critical references have been discussed within the paper. I suggest the authors cite the published version of previous work rather than the arxiv version.", "other_strengths_and_weaknesses": "Overall, the paper presents an interesting empirical finding regarding the sensitivity and similarity of MoE LLMs decomposition, introduces a method that is easy to implement and reproduce while achieving good inference acceleration, and provides comprehensive experiments on multiple MoE models and benchmarks.\nHowever, I have several concerns:\n- The similarity of decomposition matrices is an important aspect. It would be great if the authors could provide more solid examples across multiple MoE LLMs or include a theoretical discussion to strengthen their findings.\n- The trimming of the U matrix appears to have a substantial effect on performance. I encourage the authors to explore more nuanced approaches.\n- There are minor typos (e.g., the caption in Figure 1). Additionally, the method comparison in Table 1 is overly complex, making it difficult for readers to follow. A clearer presentation would enhance readability.", "other_comments_or_suggestions": "I recommend the authors double-check the line space across the paper.", "questions_for_authors": "See the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to sparsify MoE architecture for faster inference without losing much in terms of performance. The method is very similar to well-known merging techniques where a compressed representation of experts is stored by doing the best rank-k decomposition, which is nothing but performing SVD and choosing the top k singular vectors. The only novelty is in using a shared V matrix for all the experts, after performing SVD on all the experts, to further compress the model. There is some indirect justification for this -- All experts are similar, and V matrix of each expert is similar to its model weights, in terms of CKA metric defined in previous work. Even then, this is a pure heuristic and lacks theoretical grounding. Reviewers V7cs, Cmem, and cMac have pointed out this as a concern and the authors have justified this to some extent. The empirical results show improvements over baselines. I suggest the authors to carefully address all the concerns raised by the reviewers, and ground the motivations of the method proposed, in the final version of their paper.
Decision: Accept (poster)