Delta Decompression  for MoE-based LLMs Compression
Authors: Hao Gu
Wei Li
Lujun Li
Zhu Qiyuan
Mark G. Lee
Shengjie Sun
Wei Xue
Yike Guo
Venue: ICML 2025 poster (2025)
Keywords: Mixture of Experts
Efficient Large Language Models
Delta Decompression
Structured Compression
Primary area: applications->language_speech_and_dialog
TL;DR: In this paper, we present a novel approach to compressing Mixture of Experts LLMs based on delta decompression.

Abstract: Mixture-of-Experts (MoE) architectures in large language models (LLMs) achieve exceptional performance, but face prohibitive storage and memory requirements. To address these challenges, we present $D^2$-MoE, a new delta decompression compressor for reducing the parameters of MoE LLMs. Based on observations of expert diversity, we decompose their weights into a shared base weight and unique delta weights.  Specifically, our method first merges each expert's weight into the base weight using the Fisher information matrix to capture shared components.  Then, we compress delta weights through Singular Value Decomposition (SVD) by exploiting their low-rank properties.
Finally, we introduce a semi-dynamical structured pruning strategy for the base weights, combining static and dynamic redundancy analysis to achieve further parameter reduction while maintaining input adaptivity. In this way, our $D^2$-MoE successfully compacts MoE LLMs to high compression ratios without additional training. Extensive experiments highlight the superiority of our approach, with over 13\% performance gains than other compressors on Mixtral|Phi-3.5|DeepSeek|Qwen2 MoE  LLMs at 40$\sim$60\% compression rates. Codes are available in https://github.com/lliai/D2MoE.

Delta Decompression for MoE-based LLMs Compression
Hao Gu
Wei Li
Lujun Li
Qiyuan Zhu
Mark Lee?
Shengjie Sun Wei Xue Yike Guo
Abstract
Table 1. Comparison of method with other MoE compressors.
Mfxture--o-Expertt (MoE) architectures in large language models (LLMs) achieve exceptional per- formance, but face prohibitive storage and mem- ory requirements. To address these challenges, we present D2-MoE, new delta decompression compressor for reducing the parameters of MoE LLMs. Based on observations of expert diversity, we decompose their weights into shared base weight and unique delta weights. Specifically, our method first merges each expert's weight into the base weight using the Fisher information ma- trix to capture shared components. Then, we compress delta weights through Singular Value Decomposition (SVD) by exploiting their low- rank properties. Finally, we introduce a semi- dynamical structured pruning strategy for the base weights, combining static and dynamic redun- dancy analysis to achieve further parameter re- duction while maintaining input adaptivity. In this way, our D successfully compacts MoE LLMs to high compression ratios without additionaa training. Extensive experiments high- light the superiority of our approach, with over 13% performance gains than other compressors on MoE LLMs 40~60% compression rates. Codes are available in https//github.com/llaa//DM4
Diversity means retaining individual per-expert information. Max-
imum ratio denotes the maximum parameter compression ratio.
Method
Stratogy
Structured
Train free
Diversity
Max Ranio
NABE (2024b)
Prunc
X
50%
MoE-Compeess (2024)
Prune
X
X
McE-Pruner (2021)
Prunc
X
x
X
50%
MoE- /2 (2024)
Prunc
X
55%
MC RM-E (2023b)
Mergs
X
75%
HC-SMoE (2024)
Merga
x
EEP (2024e)
Merge
x
75%
M.. (Ours)
Dela
/
80%
et al., 2024) and MiniMax-01 (MiniMax et al., 2025), which demonstrate unprecedented capabilities in language under- standing and generation tasks. Despite their compelling advantages, MoE LLMs face critical challenges practical deployment scenarios (Tang et al., 2024; Zhong et al., 2024; Hwang al., 2024). Their substantial parameter foot- print, coupled with considerable memory overhead from storing multiple expert weights (Song et al., 2023), creates significant barriers to resource-constraine environments.
To address these challenges, MoE compression methods have recently gained significant attention. As illustrated in Table 1, current approaches broadly categorized into expert pruning and expert merging methods. (1) Expert pruning approaches, represented by MoE-Prunee (Xie et al., 2024), NABE (Lu et al., 2024a), and MoE-I? et al., 2024), im- plement inter-expert pruning and intra-cxpert weight sparsi- fication. While these approaches achieve significant param- eter reduction, they often result in substantial performance degradation due to the irreversible loss of expert knowl- edge. The direct removal of expert weights compromises the model's specialized capabilities, frequently necessitating additional fine-tuning to partially recover performance. (2) Expert merging methods, on the other hand, aim to consol- idate multiple experts into fewer, more compact representa- tions. Methods like EEP (Liu al., 2024a), MC-SMoE (Li et al., 2023b), and HC-SMoE (Chen al., 2024) develop various weighting schemes for weighted summation of dif- ferent experts' weights. While these approaches preserve more information than direct pruning, it introduces new challenges. The merging process assumes significant over- lap in expert functionalities, but in practice, experts often possess distinct, complementary specializations. This leads
MixtralPPii3351DeeppecckOO
1. Introduction
Recent advances in Large Language Models (LLMs) in- creasingly favor Mixture of Experts (MoE) (Cai al., 2024) architectures for their ability to scale model capacity through specialized expert networks while maintaining computa- tional efficiency via sparse activation. The success of MoE evident in recent L.Mss like DeepSeek-V V3 (DeepSeek-AA
'Equal contribution 'Hong Kong University of Science and Technology "University of Birmingham "AlSpeech Co., Ltd.. Cor- respondence Mark Lee Lee@ bham.ac Yike Guo Syikeguo@ust.hk.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Delta Decompression for MoE-based LLMs Compression
Mixtral
Phi
DeepSeek
Mistral Energy Retention
PN--U Energy Retention
Deepseek Energy Retention
Expert
12
Lager Indes
40
Dxpert Indes
Figure 2. Single values energy retention of experts original weights, merged base weights and delta weights (difference in original weights and merged base weights) from Mixtral-8x7B, Phi-3.5-MoF, ReopSeckMoF-166-BBae
Figure 1. Centered Kemel Alignment (CKA) similarity of experts weights of Mixtral- Bx7r, Phi-3,5-MoE, DeepSeekMoE- (-BBBse..
to fundamental dilemma: experts with similar weights can be effectively merged, but those with dissimilar yet impor- tant weights resist efficient compression, resulting in either suboptimal compression ratios or performance degradation. These challenges present the question: How can we design new frameworks beyond pruning and merging meth- ods in effectively balancing compression and preserving expert diversity?
that balances efficiency, diversity, and performance.
Building on these insights, we develop D2-MoE, novel compression framework to address the growing challenges of parameter redundancy, memory overhead, and storage inefficiency in MoE LLMs while preserving model perfor- mance and scalability. Rather than directly removing or merging experts, our approach strategically decomposes ex- pert weights into shared base weight, which captures the commonalities across all experts, and a delta weight, which encodes the expert-specific variations. This decomposition not only reduces redundancy but also facilitates efficient compression of delta weights by exploiting their inherent low-rank structure. To ensure that the shared base weight accurately represents the most critical information across experts, D incorporates a Fisher- weightee averaging mechanism. This approach computes the shared base weight by weighting each expert's contribution based on its Fisher importance, which quantifies the sensitivity of the model's 's parameters to the input data. By prioritizing the contribu- tions of the most important experts, weighte aver- aging balances the trade-off between redundancy reduction and representational fidelity. To further compress the delta weights, D employs truncation- -wware SVD method that integrates activation statistics into the decomposition process. This method adjusts the singular value truncation threshold based on the input activation patterns, ensuring that essential information is preserved while compressing delta weights. Finally, D2-MoE, proposes semi-dynamical structured pruning on the shared base weight, combining static and dynamic pruning phases to eliminate redundant parameters while adapting to the input distribution in real- time. With these new schemes, our DD-MoE enjoys the benefits of being structured and acceleratable, requiring no extra training, preserving expert diversity and performance, and realizing high compression ratios (see Table 1).
"Diversity is not about how we differ. Diversity is about embracing one another's uniqueness."
Ola Joseph
As the quote goes, recent fine-tuning methods (Ping et al., 2024) quantize delta weights between fine-tuned and origi- nal models to effectively capture both similarities and varia- tions. Inspired by these successes, we investigate whether it is possible to recycle the difference (delta) weights that are always discarded during expert merging to maintain per- formance without introducing excessive computational or memory overhead. Specifically, we brainstorm the idea to efficiently reallocate these abandoned delta weights (differ- ences between merged expert weights and original weights) to preserve the diversity and specialization of experts. To explore this, we conduct two key experiments to analyze the properties of expert weights in MoE LLMs: (1) We evaluate expert similarity using centered kernel alignment (CKA) metrics. As shown in Figure 1, the similarity between dif- ferent expert weights consistently falls within the 0.3 to 0.5 range. This indicates moderate overlap in their feature spaces, suggesting that while some aspects of their weights can be merged, preserving expert diversity remains crucial. (2) We examine distributions of single values energy reten- tion for different expert weight decompositions (detailed in Appendix B.2). As illustrated in Figure 2, the larger singular values of energy retentions in the delta weights show that most of the matrix's 's information is concentrated in small number of singular vectors, indicating a strong low-rank structure. This shows that these delta weights can be efficiently approximated using low-rank decomposi- tion methods without excessive degradation of information. These findings underscore that reutilizing delta weights to expert merging a promising way for MoE compression
Our extensive experimental evaluation highlights the ex- ceptional performance of D across multiple of the-art MoE language models and a wide range of bench- marks. For models like Mixtral-8x77 -xx7B and DeepSeekMoFF 16B-Base, D achieves the lowest perplexity on lan- guage modeling datasets and the highest average accuracy
2

---

Delta Decompression for MoE-based LLMs Compression
on reasoning benchmarks, even high compression ra- tios (e.g.0.52 average accuracy at 60% compression for Mixtral-8x7B, compared to 0.36 for NAEE). On large-scale models such as Phi-3.5-MoE and Qwen2-57B-A14B, D MoE maintains strong performance, delivering accuracy close to the original model while significantly outperform- ing methods like MoE-I². The consistent superiority of D² across diverse MoE LLMs and tasks demonstrates its general applicability and effectiveness in preserving pert specialization and task performance while achieving substantial efficiency gains, setting new standard for MoE. compression.
other approach, DeltaZip (Yao & Klimovic, 2023) develops multi-tenant serving system by compressing delta weights. In contrast to these quantization and system-level works, we not only first introduce delta compression into MoE compression, but also propase new techniques like MoF- specitic SVD and semi-dynamic pruning, achieving the optimal performaace--effciinnyy trade-off.
3. Methodology
3.1. Delta Compression in MoE LLMs
MoE Formulation. MoE architectures enhance the capacity and efficiency of LLMs by employing expert-based Feed- Forward Network (FFN) layers. The output of the MoE- FFN layer for an input æ is computed as:
2. Related Work
Mixture of Experts Compression methods (see Table 1) reduce parameter redundancy and minimize storage in MoE models. For example, MoFL-Pruner (Xie et al., 2024) achieves compression by pruning weights based on their activations and router importance. However, these unstruc- tured methods typically provide only limited inference ac- celeration. For structured pruning. NAEE (Lu et al., 2024a) skips non-redundant experts and trims unimportant weight connections, while MoF-I12 (Yang et al., 2024) combines inter-expert pruning with intra-expert low-rank decompo- sition. Yet, these methods involve a serious loss of expert knowledge, requiring additional fine-tuning. Our approach differs from these methods by avoiding the direct removal of experts and no need for retraining. Expert merging methods like EEP (Liu et al., 2024a) introduce a two-stage pipeline where experts are first pruned and then merged into con- solidated representations. Similarly, MC-SMoE (Li et al., 2023b) groups experts based on routing policies and merges each group into a single expert. However, merging experts inherently reduces the diversity of the model, potentially harming its ability to generalize across diverse input distribu- tions. Methods like HC-SMoE (Chen et al., 2024) mitigate retraining requirements but are still limited by the trade-off between compression and preserving the model's capac- ity. In contrast, our framework strategically isolates shared knowledge into a base weight while retaining expert-specific variations as delta weights. In addition, our semi-d semi-dynamic pruning and other techniques also do not exist in the previ- ous methods.
N
Σ G(z).
where N is the total number of experts, G(z) E R repre- sents the gating weights, and (x) is the output of the i-th expert. The sparsity is achieved through top-k selection mechanism:
G(z) Softmax (x Wgl)
(2)
where TopK[-] selects the i experts with highest gating weights, and Softmax normalizes their weights. This results in a sparse activation of experts for efficiency. Each expert Fs is a standard FFN layer, typically consisting of two or three fully connected layers. These experts constitute the majority of the weights in MoE models (e.g. 9.%% for Mixtral- 8x7B), making them the most focus of compressors (e.g.- MC-MoE and our D2-MoE).
Experts Delta Decomposition. MoE models are highly parameterized due to the presence of multiple experts, lead- ing to significant redundancy among expert weights. Di- rectly compressing these weights often results in perfor- mance degradation, as shared structures across experts are not fully exploited. To address this, we introduce a delta compression strategy that decomposes the weights of each expert into two components: shared base weight that cap- tures commonalities across all experts and delta weight that encodes expert-specific variations. This decomposition reduces redundancy, facilitates efficient compression, and minimizes performance loss. Let W. E Rmmxn represent the weight matrix of the i-th expert, where m and n denote the imput and output dimensions of the FFN layer, respectively. We express W; the sum of shared base weight W,, and expert-specine delta weight AW;
Delta compression in LLMs has emerged as a critical tech- nique to reduce the storage and computational costs of de- ploying multiple fine-tuned models by compressing the dif- ferences (delta weights) between a base model and its fine- tuned variants. Recent advancements, GPT-Zip (Isik et al., 2023) and BitDelta Liu et al. (2024b) successfully quan- tize the delta weights into ultra-low bit. Delta-CoMe (Ping et al., 2024) employs mixed-precisien quantization to the varying singular vectors of decomposed delta weights. An-
W W₁ + AW,
(3)
Here, Wo E R=X= is shared across all experts, and AW, E Rmxn represents the unique characteristies of the
3

---

Delta Decompression for MoE-based LLMs Compression
Weighted Fisher Merge
Truncation aware SVD
Semii dynamical Structured Pruning
Deployment Inference
Step thete treain Paase thep Drenn flour
Queprtilsinn M
Wi
W
IrS Bund
A
Fg
Fi
AW..
W
| locaesado
W
dass My My
Figure 3. Overall Process of MoE. We first merge original expert weights into a shared base weight, weighted according to their Fisher importance. Then, we derive delta weights and compress them using Singular Value Decomposition (SVD). Finally, we apply two-step pruning strategy: static column-wise pruning followed by dynamic column-wise pruning to further optimize the base weight.
i-th expert. By separating the shared and expert-specific components, we ensure that W3 captures the common struc- ture, reducing redundancy in the delta weights AW,
Here, F; acts as a weight that amplifies the influence of more important experts in the merging process. By normalizing the weights using the sum of Fisher importance values, we ensure that the merged base weight remains appropriately scaled and the delta weights AW; are more compact and exhibit stronger low-rank properties. This facilitates the application of low-rank compression techniques, such as Singular Value Decomposition (SVD), in later stages of our framework (see Section 3.3). The improved compressibil- ity of delta weights reduces both storage requirements and memory overhead during inference.
3.2. Experts Fisher Merging
To effectively derive base weights that represent the shared knowledge across subset of experts IC S {1,..., N} (of size K IK]) while retaining essential diversity, our goal is to compute merged base weight W2 that minimizes re- dundancy and preserves the critical information required for downstream tasks. Traditional methods, such as simple averaging, compute the merged weight as the element-wise arithmetic mean of the weights of all experts, performed as Wi = = Eiek We. Although simple averaging is computa- tionally efficient, it fails to consider the varying importance of different experts. This can lead to under-represeentaioo of critical weights in the base weight W, and increase the difficulty of compressing delta weights AW; in later stages. To address this, we incorporate the Fisher information ma- trix (Matena & Raffel, 2022), which captures the importance of the parameters of each expert in the merging process. Our merging function uses wisher-weighted averaging to com- pute the base weight Wi- The importance of each expert is quantified using the Fisher information matrix, which measures the sensitivity of the model's parameters to the log-lik ligliikli of the data, Specifically, the Fisher informa- tion for the i-th expert is given by:
3.3. Truncation-aware Singular Value Decomposition
To compress the delta weights AW, we apply the truncation-ave awwrr SVD approach in SVD-LLM (Wang et al., 2024) that enhances traditional decomposition methods by incorporating activation statistics. For each delta weight AW, we first compute its setivation-weightee representa- tion, Wscale as:
Werate AW,S,
(6)
where S; E IR is derived from the activation Gram ma- trix X,X,", with X; representing the i th expert's activation matrix. Specifically, S, is computed using Cholesky decom- position of the Gram matrix. Using Wseate. we perform SVD to decompose it into three components:
Weaaee UEVT.
(7)
F Ez Eyyno (u)=) Vo. log [)
(4)
where U E m and V E R are orthogonal matrices, and Σ E B is diagonal matrix containing the singular values. We then truncate the smallest singular values in y to retain only the top-k components, then the compressed delta U matrix and V matrix are as follows:
where D; represents the data distribution handled by expert i, po (z)lz) is the predicted probability of label y given input .r and Vo, log log (ylæz) is the gradient of the likeliioddoo with respect to the parameters Og of the i-th expert. Intu- itively, F; measures the average magnitude of the gradient norm, with higher values indicating that the expert's param- eters are more critical for the model's performance. Using the Fisher importance F,, we compute the Fisher-weighted base weight We as:
A = U (Trnn(E))
(8)
AV. <Tru
This truncation-aware SVD approach mitigates reconstruc- tion loss caused by activation outliers and ensures a mapping between singular values and compression loss to preserve the essential characteristics of the original weight distribu- tion, enabling a more effective compression process.
Wo = EIEK F.W;
(5)
Σiex Fi
4

---

Delta Decompression for MoE-based LLMs Compression
3.4. Semi-dynamical Structured Pruning
gating to activate only the top-k delta weights for each in- put. For example, the gating function selects the top-k most relevant experts' delta weights based on G(z), and their con- tributions are aggregated along with the shared base weight. The forward computation can be expressed as:
The base weight matrix Wi in our framework represents combination of multiple expert weights, making them full-rank and highly expressive. However, their high dimen- sionality introduces significant redundancy, which increases storage and computational costs during inference. Tradi- tional low-rank decomposition or static pruning methods often fail to effectively compress these base weights without incurring substantial performance degradation, owing to the unique structure of the base weights that stores information from all experts. Through empirical analysis, we observe key phenomenon: while subset of the columns in the base weights matrix consistently exhibits negligible contribu- tions across different inputs (static redundancy). the relative importance of the remaining columns varies significantly depending on the input batch (dynamic redundancy). This insight motivates to develop new two-phase (first static then dynamic) pruning paradigm that separately handles these two types of redundancies: In the static pruning phase, we identify and prune columns of W& that consis- tently contribute the least across all inputs. To achieve this, we compute a column-wist pruning metric that combines the magnitude of the weights and their interaction with the input activations. Specifically, the pruning metric for the j-th column of Wi is computed as:
K
y = W +
G(z); AUAVV Mlselected token]- (10)
1=1
where AU; and AV; are the decomposed delta weights of selected experts. This structure ensures efficient sparse computation while leveraging the specialized knowledge of the selected experts.
Parameter Compression Analysis. For experts with m the size of individual parameters, we assign p% the com- pression ratio for delta weights, and a% the compression ratio for the base weight after pruning. For static parameter storage, the original model requires n m parameters for the experts. After delta decomposition, the storage requirement increases slightly to (n + 1)m due to the addition of the shared base weight Wo. After static compression, storage parameters can be expressed as:(n- -%t-p% + s% m2 For acti- vation parameter reduction, the original activation storage requirement is k m, as top-k experts are active at a time. After compression, the activation parameter requirement becomes: (k p% + +%) m.
Cj [W[:-]ll> ||
(9)
where W2 E R x with m number of rows and n num- ber of columns (channels), X Raxb represent the in- put activations for a batch of size b. [[]];,j]lla is the L2 norm of the j-th column of Wo and | 2lll is the L2 norm of the activations corresponding to the j-th column. We then sort all columns by their pruning metric C in ascending order and prune the lowest-scoring columns to achieve half of the target sparsity level. In the dynamic pruning phase, we handle input-dependent redundancies by dynamically updating the pruning metrics for the remain- ing columns based on the current input batch. For given batch of inputs X, we recompute the column-wise pruning metric: C, dynamic = [[a[ts A1l |X] :11l2, but only for the columns retained after static pruning. We then prune the lowest-scoring columns to achieve the remaining half of the target sparsity. This dynamic pruning ensures that the model adapts to the specific input distribution of each batch, optimizing the number of active parameters during inference.
4. Experiments
In this section, we conduct a comprehensive series of experi- ments to evaluate the effectiveness of our proposed D method. We first compare our approach with state-of-the-art compression methods across various MoE models at dif- ferent compression ratios. To provide deeper insights into our method's performance, we also conduct detailed abla- tion studies on D2-MoE. All experiments performed on NVIDIA A100 GPUs.
4.1. Experimental Setups
To showcase the versatility of our D method, we assess its effectiveness common MoE models: Mixtral-8x7B, SeepSeek-moe--t 16b-base, Phi- 3.5-MoE and Qwen2-57B-A14B Mixtral-8x7B employs 8 experts, and Phi-3.5-MoE features 16 experts, each with 3.8 billion parameters. In comparison, DeepSeek-moe-16h and Qwen2-57B-A14B adopt an even more fine-grained expert architecture, leveraging 64 experts. We conduct experiments MoE models with fewer experts, such as Mixtral-8x77 and Phi-3.5-MoF, as well as those with a greater number of experts, such as DeepSeckMoE- 16B- Base and Owen2-57B-A14B. to demonstrate the versatility of D We evaluate our method across 10 datasets, encompassing language modeling datasets (WikiText-
Models and Datasets.
3.5. Overall Algorithm Procedure
The overall algorithm flow is summarized in Figure 3, which outlines the main steps of our framework, includ- ing Fisher weighted merging of base weights, delta weight compression, and vemi-dynamical structured pruning for base weights. the forward pass, the process uses sparse
5

---

Delta Decompression for MoE-based LLMs Compression
Table 2. Performance of D for Mixtral-8xe 8x7B Dee..eeeSeeKMoE 16B Base, Mhi- modeling datasets (measured by perplexity (L)) and common sense reasoning datasets (measured by accuracy (t)).
MoE and Qwen2-57hh 1112-577-A144 on 3 language
Ratio
Method
WikiText-2.
PTR.
C4)
Openh
A
WinoG,
HelloS.
ARC_e
PIQA
MatiQA
Averaget
Mixtral-8x7B
0%
Original
3,98
12.99
6.78
0.36
0.84
0.76
0.65
0.57
0.82
0.43
0.63
NAEE (2024b)
4.77
16.09
8,89
0.32
0.76
0.72
0.58
0.47
0.79
0.40
0.58
20%
MoE-I" (2024)
4,86
26.50
11.07
0.32
0.79
0.74
0.55
0.48
0.78
0.37
D7-MoE (Ours)
4.65
16.32
8.59
0.33
0.80
0.75
0.61
0.51
0.81
0.39
0.60
NAFR (2024h)
6,44
22.15
13.86
0.25
0.63
0.64
0.46
0.36
0.72
0.35
0.48
MoE-I" (2024)
6.74
60.45
22.44
0.26
0.66
0.43
0.38
0.69
0.31
40%
D"MMoE (Ours)
5,28
20.54
10.10
0.32
0.78
0.73
0.57
0.47
0.78
0.34
0.57
NABE (2024b)
11.43
47.28
31.16
0.17
0.42
0.55
0.33
0.23
0.62
0.26
0.36
MoE-Il (2024)
13.52
182.99
74.62
0.18
0.44
0.55
0.32
0.22
0.58
0.23
0.36
60%
D*-MoE (Ours)
6.46
23.63
12.76
0.28
0.72
0.71
0.51
0.38
0.73
0.31
0.52
DeegSeck-MoE-166--
0%
Original
6,38
9.47
9.82
0.32
0.76
0.70
0.58
0.44
0.79
0.31
0.56
NAGE (2024h)
9.44
15.02
15.34
0.32
0.71
0.66
0.55
0.40
0.77
0.29
0.53
20%
MoF-I" (2024)
7.69
11.59
13.72
0.26
0.71
0.68
0.49
0.38
0.73
0:29
0.50
DP (Ours)
6.84
11.10
11.88
0.30
0.74
0.69
0.55
0.41
0.76
0.31
0.54
NABE (2024b)
8.55
14:47
17.98
0.23
0.67
0.67
0.41
0.32
0.69
0.26
0.46
MoE-I' (2024)
9.73
15.75
19.75
0.23
0.64
0.66
0.41
0.31
0.68
0.26
0.45
17-MoR (Ours)
7,93
14.07
15.18
0.26
0.69
0.65
0.45
0.36
0.72
0.28
0.49
NABE (2024b)
23,20
49.89
0.49
0.58
0.33
0.24
0.23
Mo (2024)
15,83
32.2
38.60
0.17
0.48
0.58
0.32
0.23
0.61
0.22
0.37
60%
D*-MoR (Ours)
11.67
27.73
27.63
0.54
0.61
0.35
0.29
0.63
0.24
Phi-3.5.MoR
0%
Original
3.48
8:43
8.22
0.40
0.77
0,76
0.68
0,56
0.79
0.38
0.62
NABE (2024b)
8.18
20.07
16.11
0.73
0.61
0.48
0.37
40%
MoE-I' (2024)
7,46
20.95
20.95
0.29
0.59
0.67
0,27
0.40
0.70
0.25
0.45
MRM (Ours)
6.07
13.79
14.01
0.34
0,72
0.73
0.65
0.53
0.78
0.38
0.60
Qwenn2-7B-A144
0%
Original
5.12
9.18
8.36
0.33
0.75
0.74
0.63
0.46
0.81
0.39
0.59
NABE (2024b)
6.81
11.34
11.57
0.31
0.73
0.75
0.55
0.46
0.76
0.36
0.55
40%
MoE-I' (2024)
24.90
77.05
22.50
0.26
0.70
0.46
0.71
0.41
0.75
0.30
0.51
17-MMoR (Ours)
8.19
11.23
12.70
0.33
0.75
0.75
0.61
0.45
0.79
0.36
0.58
2 (Merity et al., 2017), PTB (Mareus et al., 1993), and C4 (Raffel al., 2020)), along with 7 common sense reasoning datasets (OpenbookQA (Mihaylov et al., 2018), WinoGrande (Sakaguchi et al., 2020), HellaS (Zellers et al., 2019), PIQA (Bisk et al., 2020), MathQA (Amini et al., 2019), ARC-e, and ARC-c (Clark et al., 2018)) in a zero-shot setting using the LM-Evaluation- Harness frame- work (Gao et al., 2023).
of the original performance 0.63), outperforming NAEE (0.58) and MoE-I2 (0.57). Even at 60% compression, our method maintains competitive score of 0.52, significantly surpassing both baselines (0.36). The advantages extend to models with more experts. For DeepSeek-Mon 16B- Base, our method achieves average scores of 0.54, 0.49, and 0.41 at 20%, 40%, and 60% compression respectively, showing significant improvements over baselines, particu- larly in perplexity metrics. Similar superior performance observed on Phi-3.5-MoE and Qwen2-57B-A A14B, demon- strating the effectiveness of D2 across different model scales. More detail results are shown in Appendix A.1.
Implementation Details. For fair comparisons, we use 512 random samples from WikiText-2 as calibration data to conduct all experiments. We focus on compressing the model without retraining the full model parameters. See Appendix A for more details.
Comparing to Different Compressors. We compare D MoE against three categories of compression methods on Mixtral-8x at 20% compression ratio: pruning- methods (SparseGPT, NAEB). SVD-based methods (ASVD, MoE-I?), and hybrid methods (LoSparse, MC-SMoE, MoE- Compress). As shown in Table 3, 2 achieves the best overall performance with an average score of 0.60 (95.2% of original 0.63), outperforming all baselines across multiple metrics. Our method shows competitive perplexity scores
4.2. Compression Performance and Comparisons
Main Results in Multiple MoE LLMs. Our experimen- tal results shown in Table 2 demonstrate the superior per- formance of D*-MoE across different MoE models and compression ratios. On Mixtral-8x7B, at 20% compres- sion, D achieves an average score of 0.60 (95.2%
6

---

Delta Decompression for MoE-based LLMs Compression
Table 3. Performance of Mixtral-8xx 8x7B compressed by D2-MoE under 20% compression ratios.
Methods
WikiText-2
PTRI
C4)
Openh.
ARC,
WinoG.
HellaS.
ARC.
PIQA
MaiiQA
Averaget
Original
3.98
12.99
6.78
0.36
0.84
0.76
0.65
0.57
0.82
0.43
0.63
NAEE (2024b)
16.09
0.32
0.76
0.58
0,47
0.58
SpursGGPT1224) (2023)
4.69
21.11
9.19
0.30
0.77
0.74
0.56
0.45
0.77
0.35
0.56
MoE-I" (2024)
4.86
26.50
11.07
0.32
0.79
0.74
0.55
0,48
0.78
0.37
0.57
ASVD (2023b)
9.44
47.29
20.30
0.25
0.71
0.66
0:48
0.40
0.73
0.35
0.51
LaSparse (20234)
953.51
805.16
1273.12
0.20
0.27
0.49
0.28
0.26
0.53
0.20
0.32
MC MC-S (2024g)
1341.36
1316.52
1478.13
0.26
0.28
0.51
0.29
0.15
0.54
0.19
0.33
MoE. Compress (2024)
6.12
14:67
11.61
0.30
0.73
0.70
0.54
0.46
0.73
0.33
0.54
D'-MoE
16.32
0.33
0,80
0.75
0.61
0.51
0.60
Table 4. Throughput (Tokens/sec), Memory of Mixtral-8x7B model under 60%, 70%, 80% compress ratio of different methods. And the perplexity on WikiText-2 of 60% 70%, 80% is 6.35, 8.15, 12.95 respectively which gain well performance under high com-
Table 6. Perplexity of compressors for Base and Delta Weights.
Part
Method
WikiText-21
PTB.
C41
Truncution-aware SVD
5.63
27.40
12.65
W
Stutie Pruner Semi-dynamic Pruner
5.31
20.43
10.75
pression ratio.
5.28
20.54
10.10
Methods
BSZ-64, Ratic=60%
BSZ-64, Ratie=70%
BSZ-64, Ratio=80%
Pruning
5.74
20.83
11.48
Model Size
Memory
18 18,680
34.8G
D4D
26.1G
9.33B
17.3G
AW; Vanilla SVD
6.22
22.54
10.72
Activation- -aaatonon-waaaaooooo SVD
5.91
22.63
11.31
TFLOPs
Tokensssec
TELOPs
Tokens/sec
TELOPs
Tokens/eec
Truncation aware SVD
5.28
20.54
10.10
NAER
481
271.89
385
272.66
240
278.53
M
838
227.60
745
252.55
647
204,04
LaSpure 1)7-MoB
1240
191.45
198.04
481
237.73
186
300.33
290
313.29
performing NAEE by 12.5% and MoE-12 by 6.5%. In con- trast, LoSparse uses 4.6x more TFLOPs (1330) but achieves only 198.04 tokens/sec. Meanwhile, our method maintains reasonable perplexity on WikiText-2 (6.35, 8.15, and 12.95 at 60%, 70%, and 80% compression), demonstrating an optimal balance between efficiency and performance. Refer to Appendix A.3 to see detail results analysis.
Table 5. Perplexity of different merge methods for Base Weights.
Method
Wik
PTBJ
C4
Original
3.98
12.99
6.78
Mean average
7.66
46.85
24.39
TIES (2024)
12.45
87.31
29.10
RegMean (2022)
187.19
1206.05
612.70
Frequency (2023c)
6.42
35.12
13.79
4.3. Ablation study
Fisher (Ours)
5.28
20.54
10.10
Various Merger for Base Weights. Table delves into the performance of different merge methods for base weights, demonstrate the effectiveness of our Fisher merge (Matena & Raffel, 2022) method compared to other base weight merging techniques, including mean average, expert fre- quency average (Li et al., 2023c), RegMean (Jin et al., 2022), and TIES (Yadav et al., 2024). The analysis reveals that while expert frequency merging which uses expert activation frequency for weighted averaging on base weights demon- strates promising results, our Fisher merge method selec- tively extracts important weights from different experts and integrates them into the base weights which achieves the lowest perplexity scores on WikiText-2 (5.28), PTB (20.54), and C4 (10.10) on Mixtral-8x7B under 40% compression ratios setting.
on WikiText-2 (4.65), PTB (16.32), and C4 (8.59), surpass- ing both pruning and SVD-based methods. For downstream tasks, M achieves superior performance on reason- ing tasks like ARC-e (0.80) and WinoG (0.75). While hy- brid methods show significant degradation, especially in perplexity metrics, our method maintains consistent per- formance across all evaluations, demonstrating an optimal balance between compression and model capabilities. See Appendix A.2 to get more detail results analysis.
Inference Speed Acceleration and Memory Reduc- tioneTable 4 demonstrates the inference efficiency of various methods on Mixtral-8x77 (batch size=64) under high com- pression settings. At 60% compression, 2 achieves 277.72 tokens/sec with only 481 TFLOPs, surpassing NAEE in throughput while matching its computational efficiency. MoE-I2 and LoSparse require significantly more TFLOPs (838 and 11 1150) but deliver lower throughput. The advan- tages become more evident at 80% compression, where Mo- achieves 313.29 tokens/sec with 290 TFLOPs, out-
Varying Compressors for Base and Delta Weights. Ta- ble 6 unveils the different methods to compress both base and delta weights. For base weights, our dynamic pruning method achieves superior performance, with the lowest per- plexity scores on the WikiText-2 (5.28). PTB (20.60), and C4 (10. (0.112) da)atsess outperforming both truncation-ane aware
7

---

Delta Decompression for MoE-based LLMs Compression
AWeights Trimming Performance Mixtral-8x7B, 40% Baseline Compression)
Table 8. Perplexity of 40% compressed Mixtral-8x7B via calibra- tion data with varying number from WikiText-2 and C4
Co Compression Ratio
00%
Wiifextt-2
PTB
75
C4
Method
32
64
128
256
512
Palio (%6
70
WikiText 21
5.81
5.56
5.48
5.37
5.28
42%
C41
10.92
10.79
10.65
10.52
10.10
Perpleiig
65
50
Table 9. Perplexity of varying calibration data on Mixtral-8x7B 40% compression ratios.
20
Calibration
WikiText-2]
PTB/
C41
80
Wikitext-2
5.28
20.54
10.10
AW Trimming numbers
C4
5.37
21.03
11.52
Figure 4. Expanding D2-MoE via Delta Weights Trimming.
ually, but our method maintains competitive performance across different compression ratios (43%-81%) With I trimming delta weight (43% compression), we achieve a WikiText-2 perplexity of 6.43, comparable non-trim mod- els. Even with 7 trimming experts (75% compression), the perplexity remains reasonable at 14.71, demonstrating effec- tive balance between compression and performance. More detailed results can be found in Appendix B.1.
Table Perplexity of various individual ratios for Base and Delta Weights on Mixtral-8x7B 40% compression ratios.
W
AW,
2
PTB
C44
10%
52.66%
5.63
27.40
12.65
20%
51.41%
5.31
20.43
10.75
30%
50.16%
5.28
20.54
10.10
40%
48.90%
5.74
20.83
11,48
Impact of Calibration Data. Table 9 demonstrates that the choice of calibration data, whether WikiText-2 or C4, has minimal influence on overall task performance, highlighting the robustness of our method across diverse datasets. Table 9 explores the effects of varying the number of calibration samples. Results indicate that increasing the number of data samples generally leads to a decrease in perplexity, suggesting improved performance with more samples.
50%
47.67%
6.22
22.54
10.72
60%
46.42%
5.28
20.54
10.10
SVD with a scale matrix using in SVD-LLM and static pruning (Wanda-sp). For delta weights, we compare prun- ing, Vanilla SVD without scale matrix, activation-aware SVD (Yuan et al., 2023a). Our truncation-aware SVD method with scale matrix achieves the best performance. semi-dynamie pruning for base weights and truncation-aweet aware SVD for delta weights in our M--M
5. Conclusion
Finally,
In this work, we present De-MoE, a unified framework for compressing MoE LLMs by addressing the inherent redun- dancy in their weight structures. Our approach systemati- cally integrates delta decomposition, Fisher- weighted merg- ing. truncation-awea -warre singular value decomposition (SVD), and semi-dynamical structured pruning achieve efficient parameter reduction while maintaining the performance of MoE models. By decomposing expert weights into shared base weight and expert-specific delta weights, we effectively isolate common structures and reduce redundancy. Our em- pirical analysis demonstrates that D achieves signifi- cant parameter compression while preserving the predictive performance of MoE models on benchmark tasks. Future work may explore integrating 2 with advanced train- ing techniques, such as knowledge distillation and parameter quantization. We hope that the proposed framework con- tributes to the broader field of efficient large-scale modeling, offering a practical pathway for deploying high- cigh-ccp MoB models real-world applications.
Sensitivity of Compression Ratio Hyperparameters. Ta- ble 7 demonstrate the sensitivity of compression ratio be- tween base weights and delta weights. Under the setting of 40% compression of Mixtral-8x7B model, we observe that less compression of the base weights generally leads to better performance, as it preserves more critical information among all experts and maintains model accuracy. However, there is a trade-off between the compression ratio and in- ference time speedup, as higher compression ratios of base weights typically result in faster inference but may degrade performance. After careful evaluation, we choose bal- anced ratio that optimizes both performance and inference efficiency, and more Hyperparameters are in Appendix B.5.
Expanding M via Delta Weights Trimming. Fig- ure 4 shows how delta weights trimming affects D2-MoE's performance, As trimming increases, perplexity rises grad-
8

---

Delta Decompression for MoE-based LLMs Compression
Limitations Our D2-MoE involves decomposition and pruning steps with some complexity (see more analysis in Appendix A.4). We aim to simplify it in future work.
Dong, P., Li, L.., Wei, Z., Niu, X., Tian, Z., and Pan, H. Emq: Evolving training-free proxies for automated mixed precision quantization. arXiv preprint arXiv:2307.10544 2023b.
Acknowledgements
Dong, P., Li, L., Tang, Z., Liu, X., Pan, X., Wang, Q., and Chu, X. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In ICML, 2024.
This work funded in part by the HKUST Start-up Fund 1), Theme-based Research Scheme grant (T45- 205/21-N), the InnoHK funding for Hong Kong Generative AI Research and Development Center, Hong Kong SAR, and the research funding under HKUST-DXM Al for Fi- nance Joint Laboratory (DXM25EG01).
Dong, P., Li, L., Tang, Z., Liu, X.. Wei, Z., Wang, Q., and Chu, X. Parzc: Parametric zero-cost proxies for efficient nas. In AAAI, 2025a.
Dong, P., Li, L.,, Zhong, Y., Du, D., Fan, R., Chen, Y., Tang, Z., Wang, Q. Xue, W., Guo, Y., et al. Stbllm: Breaking the 1-bit barrier with structured binary 1lms. ICLR, 2025b.
Impact Statement
The primary focus of this work is develop and evaluate technical approaches for improving the storage and reason- ing efficiency of MoE LLMs. By addressing the inherent redundancy in MoE architectures, our D*-MoE framework contributes to the ongoing effort to design green and easy-to- use LLMs. All ovaluations and experiments are performed on publicly available benchmarks, ensuring transparency and reproducibility. We believe that our framework will not be controversial in ethical impacts and expected societal implications.
Frantar, E. and Alistarh, D. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.0077
2023.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
L., Tang, E., Thite, A.. Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation,
References
12 2023. URL https: /enodo atg/records/ 10256836.
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorsk, R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation aperation-based formalisms. In NAACL-HLT (1), pp. 2357-2367. Association for Com- putational Linguistics, 2019.
He, S., Dong, D., Ding, L., and Li, A. Demystifying the com- pression of mixtureoof-experts through a unified frame- work. arXiv preprint arXiv:2406.02500. 2024.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. AAAI, pp. 7432-744 7439. AAAI Press, 2020.
Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T. and Yang. M. Pre- Pre-gated moe: An seporithh-systene co- design for fast and scalable mixture-of-expprt inference. In 2024 ACM/IEEE 51 Annual International Symposium on Computer Architecture (ISCA), pp. 1018-100110 1031. IEEE, 2024.
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, A survey mixture of experts. arXiv preprint arXiv:2407.06204, 2024.
Chen, I., Liu, H.-S., Sun, W.-F, Chao, C.-H., Hsu, Y.-C., Lec, C.-Y., et al. Retraining-fie free merging of sparse mixture-of-expper via hierarchical clustering. arXiv preprint arXiv:2410.08589 2024.
Isik, B., Kumbong, H., Ning, W., Yao, X., Koyejo, S., and Zhang, C. Gpt-zip: Deep compression of finetuned large language models. In Workshop on Efficient Systems for Foundation Models@ ICM1,2023, 2023.
Clark, P., Cowhey, I., Etzioni, O.,, Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, abs/ 0alb//8803.054/ 05553, 2018.
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights language models. arXiv preprint arXiv:2212.098449 2022..
DeepSeck-AL et al. Deepseck-v3 technical report. arXiv preprint Arxi::2412. /9437, 2024.
Li, L. relf-regulated feature learning via teacher-free feature distillation. In ECCV, 2022.
Dong, P., Li, L., and Wei, Z. Diswot: Student architecture search for distillation without training. In CVPR, 2023a.
Li, L. and Jin, Z. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeulPS, 2022.

---

Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., Jiao, E., Li, G., Zhang, G., Sun, H., Dong, H., Zhu, J., Zhuang, J.. Song, J., Zhu, J., Han, J., Li, J., Xie, J., Xu, J., Yan, J.. Zhang, K., Xiao, K., Kang, K., Han. L., Wang, L., Yu, L., Feng, L.., Zheng, L.,, Chai, L.., Xing, L., Ju, M., Chi, M., Zhang, M., Huang, P., Niu, P., Li, P., Zhao, P., Yang, Q., Xu, Q., Wang, Q., Wang. Q. Li, Q., Leng, R., Shi. S., Yu, S., Li. S., Zhu, S., Huang, T., Liang, T., Sun, W., Sun, Delta Decompression for MoE-based LLMs Compression
Li, L., Dong, P., Wei, Z.,, and Yang, Y. Automated knowl- edge distillation via monte carlo tree search. In ICCV, 2023a.
Liu, J., Xiao, G., Li, K., Lee, J. D., Han, S., Dao, T., and
Cai, T. Bitdelta: Your fine-tune may only be worth one bit. arXiv preprint arXiv:2402.10199, 2024b.
Li, L., Bao, Y., Dong, P., Yang, Li, A., Luo, W., Liu, Q.. Xue, W., and Guo, Y. Detkds: Knowledge distillation search for object detectors. In ICML,, 2024a.
Lu, X., Liu, Q. Xu, Y., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models, 2024a.
Li, L., Dong, P., Li, A., Wei, Z., and Yang, Y. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. NeulPS, 2024b.
Lu, X., Liu, Q., Xu, Y., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large lan- guage models. arXiv preprint UrXiv:2402.14800, 2024b.
Li, L.. Peijie, Tang, Z., Liu, X., Wang, Q. Luo, W., Xue, W., Liu, Q., Chu, X., and Guo, Y. Discovering sparsity al- location for layer-wise pruning of large language models. In NeulPS, 2024c.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of english: The penn treebank. Comput. Linguistics, 19(2): 1993.
Li, L., Sun, H., Li, S., Dong, P., Luo, W., Xue, W., Liu, Q. and Guo, Y. Auto-gas: Automated proxy discovery for training-free free generative architecture search. ECCV, 2024d.
Matena, S. and Raffel, C. A. Merging models with fisher- weighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716 2022.
Li, L., Wei, Z., Dong, P., Luo, W., Xue, W., Liu, Q., and Guo, Y. Attazero: efficient attention discovery for vision transformers. In ECCV, 2024e.
Merity, S., Xiong, C., Bradbury. J., and Socher, R. Pointer sentinel mixture models. ICLR (Poster). OpenRe- view.net, 2017.
Li, P., Zhang, Z., Yadav, P., Sung, Y.-.., Cheng, Y., Bansal, M.. and Chen, T. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint ar arXiv22310001332 2023b.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? A new dataset for open book question answering. In EMNLP, pp. 2381-2391. Association for Computational Linguistics, 2018
Li, P., Zhang, Z., Yadav, P., Sung, Y.L.., Cheng, Y., Bansal, M., and Chen, T. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint ar 2xiv:23100001334 2023c.
MiniMax, Li, A., Gong, B., Yang. B., Shan, B., Liu, C.,
Li, P., Zhang, Z., Yadav, P., Sung, Y.-L., Cheng, Y., Bansal, M., and Chen, T. Merge, then compress: Demystify ef- ficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Represen- tations, 2024f. URL https: httpp::///ppereee wpwp//peee..eeeeeeeeee net/ forum? w.d.eedGFFPyWWW...
W., Cheng, W., Li, W., Song, X.. Su, X., Han, X., Zhang, X., Hou, X., Min, X., Zou, X., Shen, X., Gong, Y., Zhu, Y., Zhou, Y., Zhong, Y., Hu, Y., Fan, Y., Yu, Y., Yang,
Li, W., Li, L.,, Lee, M., and Sun, S. Als: Adaptive layer spar- sity for large language models via activation correlation assessment. In NeulPS, 2024g.
Y., Li, Y., Huang. Y., Li, Y., Huang, Y., Xu, Y., Mao, Y.,
Li, Z., Li, Z., Tao, Z., Ying. Z., Cong, Z., Qin, Z., Fan, Z., Yu, 7.,, Jiang, 7.,, and Wu, 7. Minimax- Scaling foundation models with lightning attention, 2025. URL https: /ttp//aaxvv..//////////.../..... www.wiiwwg......e..........ooore awaaabasaee 22002/51.110 08313.
Li, Y., Yu, Y., Zhang, Q.. Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large lan- guage models based on low-rank and sparse approxima- tion. In International Conference on Machine Learning, PMIR, 2023d.
Ping, B., Wang, S., Wang, H., Han, X., Xu, Y., Yan, Y.,
pp.
Chen, Y., Chang, B., Liu, Z., and Sun, M. Delta-come: Training-fre free delta-compression with mixed-precision for large language models. In Thirty-ighht Conference on Neural Information Processing Systems, 2024.
Liu, E., Zhu, J., Lin, Z., Ning, X., Blaschko, M. B., Yan, S., Dai, G., Yang, H., and Wang, Y. Efficient expert pruning for sparse mixture-of-experts language models: Enhanc- ing performance and reducing inference costs. arXiv preprint arXiv:2407.0m 00945, 2024a.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the
10

---

Delta Decompression for MoE-based LLMs Compression
limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:::: 2020.
Zhong, S., Liang, L.,, Wang, Y., Wang, R., Huang, R., and Adaptive sensitivity- expert gat- ing and management for efficient moe inference. arXiv preprint arXiv:2408. 10284, 2024.
Li, M. Adapmoe:
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, pp. 8732-8740. AAAI Press, 2020.
Song, Y., Mi, Z., Xie, H., and Chen, H. Powerinfer: Fast large language model serving with consumer- grade gpu, 2023.
Tang, P., Liu, J., Hou, X., Pu, Y., Wang, J., Heng, P.-.,, Li, C., and Guo, M. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint arXiv:2411.01433 2024.
uyuk, C., Lasby, M., Yassin, M., Evei, U., and Ioannou, Y. Learning parameter sharing with tensor decompositions and sparsity. arXiv preprint (eX.::2411.098//11 2024.
Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd- Ilm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024.
Xie, Y. Zhang, Z., Zhou, D., Xie, C., Song, Z., Liu, X., Wang, Y., Lin, X., and Xu, A. Moe-pruner: Pruning mixture-of of-eeesss large language model using the hints from its router. arXiv preprint arXiv:2410. /20/3, 2024.
Yadav, P., Tam, D., Choshen, L.,, Raffel, C. A., and Bansal, M. fiesingrging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Yang, C., Sui, Y., Xiao, J., Huang, L., Gong, Y., Duan, Y., Jia, W., Yin. M., Cheng, Y., and Yuan, Moe-i2: Compressing mixture of experts models through inter- expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.01011 2024.
Yao, X. and Klimovic, A. Deltazip: Multi-tenant language model serving via delta compression. arXiv preprint arXiv:2312.05215. 2023.
Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0582// 2023a.
Yuan, Z., Shang, Y., Song, Y., Wu, Q. Yan, Y., and Sun, G. ASVD: activation-awar singular value decompo- sition for compressing large language models. CoRR, abs/2312.05821, 2023b.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In ACL (1), pp. 4791-4800. Association for Computational Linguisties, 2019.

---

Delta Decompression for MoE-based LLMs Compression
Appendix
This appendix provides additional details and analyses to complement the experiments and methodology described the main paper. We first present an extended discussion of experimental results, including comparisons with other methods, analyses of compression efficiency, and the impact of the calibration dataset. Next, we provide detailed information about our experimental setups, including the evaluated Mixture taare-off-ppperttt (MoE) models, datasets, hyperparameters, and experimental configurations. Finally, we include algorithmic tables and pseudo-code for key components of the D framework to ensure clarity and reproducibility of our approach.
A. More Discussion and Experimental Results
A.1. Detail Analysis of Main Results
Our experimental results demonstrate the superior performance of D2-MoE across different MoE models and compression ratios. On Mixtral-8x7B, our method consistently outperforms existing approaches across all compression ratios (20%, 40%, and 60%). At 20% compression, D2-M achieves an average score of 0.60, maintaining 95.2% of the original model's performance (0.63), while NAFE and MoF-I only achieve 0.58 and 0.57 respectively. Notably, even at aggressive 60% compression, D2-MoE maintains a competitive average score of 0.52, significantly surpassing NAEE (0.36) and MoE-12 (0.36). The advantages of M2 are further validated on models with more experts. For DeepSeek BeepSeek-MoEE-T 110-168-BBas our method maintains stable performance across different compression ratios, achieving average scores of 0.54, 0.49, and 0.41 at 20%, 40%, and 60% compression respectively. This represents a significant improvement over baseline methods, particularly in perplexity metries (WikiText-2J,, PTB and C4.) where our method shows orders of magnitude better results compared to MoE-12. Similar patterss observed in Phi-3.5-MoE and Qwen2-57B-A144 maintains higher performance scores while achieving target compression ratios. Most remarkably, our method exhibits exceptional stability in maintaining model performance across different evaluation tasks. For instance, on downstream tasks such as ARC-e, WinoG, and PIQA, D2-MoE consistently preserves close to 90% of the original model's performance at 20% compression across all tested models. This demonstrates that our compression method not only achieves high compression ratios but also preserves the model's general language understanding and reasoning capabilities.
where D consistently
A.2. Comparison with Other Methods
Our experimental results demonstrate the superior effectiveness of D2-MoE in compressing the Mixtral-877B model at a 20% compression ratio, outperforming various state-of-the-are compression methods across multiple metrics, as shown in Table 3. Specifically, we compare our method against three categories of compression approaches: (1) pruning-based methods (SparseGPT, NAEE), (2) SVD-baser methods (ASVD, MoE-I?), and (3) hybrid methods that combine multiple compression techniques (LoSparse, MC-SMoE, MoE- MoE-Compress), - achieves the best overall performance with an average score of 0.60, maintaining 95.2% of the original model's capabilities (0.63). In perplexity evaluations, our method achieves competitive scores on WikiText-2 (4.65), PTB (16.32), and C4 (8.59), matching or outperforming pure pruning methods like NAEE and GPT Notably, our approach significantly outperforms SVD-based methods such as ASVD and MoE-1?, which achieve average scores of 0,51 and 0.57 respectively. For downstream tasks, -2 demonstrates remarkable performance, particularly in reasoning tasks such as ARC-e (0.80) and WinoG (0.75), surpassing all baseline methods. The hybrid methods (LoSparse and MC-SMoE) show significant degradation in performance, especially in perplexity metrics, while our method maintains stable performance across all evaluation dimensions. Other model compression methods follow different technical approaches- targeting either dense models (Dong et al., 2024; Li et al., 2024c;g; uyuk et al., 2024), smaller architectures (Li et al., 2024d:e), or addressing other aspects of compression such as quantization (Dong et al., 2025h; 2023b), AutoMI. (Dong et al., 2025a; 2023a), or distillation (Li et al., 2024a;b; 2023a: Li & Jin, 2022; Li, 2022).
This comprehensive comparison validates that D2-MoB effectively preserves model capabilities while achieving the desired compression ratio, striking an optimal balance between model efficiency and performance.
A.3. Analysis of Runtime SpeedUp and Memory Usage
Table 4 demonstrates significant hardware inference acceleration across high compression ratios. We evaluate the inference efficiency of various methods on Mixtral-8x77 with batch size of 64 under high compression settings (60%, 70%, and 80%). Our MoE achieves superior throughput while maintaining the lowest TFLOPs among all compared methods. At 60% compression (18.68B parameters, 34.8G memory), D2-MoE achieves 277.72 tokens/sec throughput while requiring
12

---

Delta Decompression for MoE-based LLMs Compression
only 481 TFLOPs, matching NAEE's computational efficiency but with 2.1% higher throughput. In contrast, MoE-12 and LoSparse require substantially higher computational resources (838 and 1150 TFL.Pss respectively) while delivering lower throughput. The efficiency advantages of D²-MoE become more pronounced at higher compression ratios. At 80% compression (9.33B parameters, 17.3G memory), our method achieves 313.29 tokens/sec, outperforming NAEE by 12.5% and MoF-I by 6.5% in throughput while maintaining the lowest TFLOPs (290). Notably, LoSparse, despite using 4.6x more TFLOPs (1330), achieves only 198.04 tokens/sec, demonstrating the superior efficiency of our approach. While achieving these significant speedups, D maintains reasonable model performance. The perplexity scores on WikiText-2 at 60%, 70%, and 80% compression ratios are 6.35, 8.15, and 12.95 respectively. showing a gradual and controlled degradation even extreme compression levels. These results highlight that D not only achieves better compression quality but also delivers practical benefits in terms of inference speed and computational efficiency, making it particularly attractive for real-world deployments where both model size and inference speed are critical considerations.
A.4. Computational Cost Discussion of D2-MoE
We present detailed computational cost analysis for each stage of our compressing procedure. The D merging approach encompasses two principal stages: 1.Metric calculate and M.Merging expert weights. To begin with, the Fisher metric is calculated by feeding a calibration dataset through the model and computing gradients with respect to each weight parameter. Secondly, Expert weights merging is performed through weighte averaging, where the Fisher metric of each parameter serves as its importance weight the merging process. This Process spends m when conducting on Mixtral-8x7B Then, we compute the scale matrix for SVD and decompose the delta weights using Singular Value Decomposition (SVD). In terms of computational cost, collecting the scale matrix through calibration takes 19 minutes, while performing SVD decomposition on delta weights requires 25 minutes on Mixtral models. These two steps constitute the main computational overhead of our method. Time of other models are shown in Table 10.
Table 10. Compressing time and memory used of different models
Stage
Metric
Mixtral-8x77 DeepSeekMME--16 16B-B----6-BBss
Phi3.5-MoE
Qwen2-57B-AI4B
Time Cost
1l mins
8 mins
13 mins
23 mins
Merging
Memory Cost
131.01 GB
34.6 GB
118.27 GB
127.21 GB
Time Cost
44 mins
15 mins
35 mins
59 mins
SVD
Memory Cost
109.45 GB
52.9 GB
117.41 GB
126.41 GB
A.5. Additional Results on Compression Ratios between Base Weights and Delta Weights
In Table 7, we provide additional results for Mixtral-8x7B across a range of compression ratios. These results highlight the flexibility of D2-MoE in balancing compression and performance. For example, at the 40% compression ratio, D2-MoE achieves a WikiText-2 perplexity of 5.28, significantly lower than competing methods. We ultimately selected a 10% pruning perpextyt
ratio for the Base weights and a corresponding SVD decomposition ratio for the Delta weights to preserve performance, as the Base weights contain more shared knowledge across all experts. Additionally, a 60% pruning ratio for the Base weights chosen when prioritizing throughput efficiency.
A.6. Pushing the Limit of Compressing Delta Weight
Based on the CKA (Centered Kernel Alignment) similarity analysis demonstrated in Figure 5 of delta weights among V matrices and U matrices in the Mixtral-8x7B model, we observe that all V matrices exhibit extremely high CKA similarity scores of approximately 0.9. This significant finding motivates compress the delta weights. Furthermore, the U matrices demonstrate moderate CKA similarity scores around 0.3, indicating considerable redundancy among them. This observation suggests that we can merge the U matrices in a manner similar to base weight compression(e.gg fi.hhee merge), thereby achieving additional parameter reduction in U matrices. Refer to Table 11 to see the results.
share single V matrix among experts to further
A.7. Adaptive Compression Ratio for Delta Weight
Given the observed layer-wise wise sensitivity shown in Figure6 in the importance of delta weights, we propose an adaptive compression strategy that allocates more parameters to sensitive layers while maintaining the same total parameter budget. As illustrated in Figure 6, our analysis reveals significant variations in layer sensitivity, with layer 2 exhibiting the highest
13

---

Delta Decompression for MoE-based LLMs Compression
Methods
WikiText-21
PTBT
C44
Tokens/sec
ShareV 80% ratio
12.95
33.30
17.14
337.82
MergeU 80% ratio
13.21
40.07
19.50
290.55
ShareV+Merger 85% ratio
18.93
61.18
24.94
328.04
Table 11. Perplexity and Throughput of V and MergeU methods on extremly high compression ratio
Expert Delta Wee Simiarity (D0A)
Excert Datta gir Similrtty (DX4)
Frantt
Figure 5. CKA of Delta V weights and Delta U weights of Mixtral-8x7B
sensitivity to compression while layer showing the least sensitivity. Based on these findings, we implement an adaptive parameter allocation strategy where layer 2 receives the largest parameter budget, and layer is assigned the smallest share of parameters, optimizing the distribution of compression ratios according layer-wise sensitivity. This approach optimizes the distribution of parameters across layers based their relative importance, potentially leading better performance compared to uniform compression across all layers.
B. More Detailed Experimental Details
B.1. Details Delta Weights Trimming
Figure provides insights into the impact of delta weights trimming of D2-MoE's performance. We utilize expert frequency the criterion for trimming decisions, every time the lowest router sampling frequency expert's delta weight trimed. The results show general trend of raising perplexity the number of trimmed delta weights increases, indicating trade-off between model size reduction and performance retention. However, method achieves significant compression ratios, ranging from 43% to 81% %,, while maintaining relatively low perplexity scores compared to baseline methods. For instance, with trimming delta weight, we achieve a 43% compression ratio with a WikiText-2 perplexity of 6.43, which is competitive with non-trim compressed model. Even at higher compression ratios, such as 75% with 7 trimming experts, our method maintains a reasonable perplexity 14.71 on WikiText-2, demonstrating its robustness. This highlights the advantage of our approach in balancing model efficiency and performance, making suitable for resource resource-constrained environments without substantial degradation in language modeling quality.
Notial Seniiiily Aunno
Figure 6. layer wise sensitivity of Mixtral-8x77
14

---

Delta Decompression for MoE-based LLMs Compression
B.2. Motivation of CKA Similarity between experts and Single Value energy Retention
The CKA Similarity between experts reflects the redundancy among experts, which motivates us to merge their base weights to capture shared knowledge, as formulated in Equation 11. Energy-Retentie quantifies the amount of information preserved after singular value truncation in the SVD decomposition process. As formulated in Equation 12, where k represents the number of largest singular values retained, this metric helps us evaluate the effectiveness of our compression while maintaining essential information.
t((KHL)
CKA(WI, W:))
(1))
Σx A E A, where d; = og
Energy-Retentio
(12)
B.3. Evaluated MoE Models and Datasets
We evaluate D2-MoE on a range of common MoE large language models (LLMs), including:
Mixtral-8x7B: A state-of-the-art MoE model developed by Mistral AI, featuring 8 experts with each expert containing 7 billion parameters represent large experts MoE model in our experiment. It employs a top-2 routing mechanism where two experts are activated for each token, demonstrating strong performance across various tasks while maintaining computational efficiency.
Dee-SeekMoE-166--B An advanced MoE architecture with 16 experts and shared expert, where each expert contains I billion parameters represent small and fine-grained experts MoE model in our experiment. The model incorporates shared expert mechanism to capture common knowledge across tasks, while specialized experts focus on domain-spectie features, Its routing strategy dynamically selects the most relevant experts for each input token.
Phi-3.5-MoE: Developed by Microsoft, this model features 16 experts, each containing 3.5 billion parameters. implements a unique expert configuration that balances model capacity and computational efficiency. The model utilizes a sophisticated routing mechanism to effectively distribute computational load across experts.
Qwen2-57B-A14B: A large-scale MoE model developed by Alibaba, comprising 64 experts and a shared expert, with total of 57 billion parameters (14B active). The model employs an advanced sparse gating mechanism that activates only a small subset of experts for each token, achieving remarkable parameter efficiency despite its scale. The shared expert serves as a knowledge hub while specialized experts capture Nomain--peciif features.
These models represent different design philosophies in MoE architecture, varying in their number of experts, parameter distribution, and routing strategies, providing a comprehensive testbed for evaluating our compression method.
We test these models on 10 datasets, including language modeling datasets (WikiText-2, PTB, and C4) and 7 common-sente reasoning datasets (OpenbookQA, WinoGrande, HellaSwag, PIQA, MathQA, ARC-easy, and ARC-challenge).
B.4. Calibration Dataset
We use WikiText-2 as the primary calibration dataset, selecting 512 random samples for all experiments. To assess the effect of calibration data type, we also use samples from C4. As shown in Table 9, WikiTeext22 consistently outperforms C4 in terms of compression quality. The calibration data size plays an important role in our method's performance. While increasing calibration samples generally improves results, we observe that using 2048 samples leads to a slight performance degradation. Therefore, considering the trade-off between performance and computational efficiency, we choose 512 samples as our optimal calibration set size.
B.5. Hyperparameters and Experimental Configurations
Table 12 lists the key hyperparameters used our experiments, including the target compression ratios, SVD truncation thresholds, and sparsity levels for static and dynamic pruning.
15

---

Delta Decompression for MoE-based LLMs Compression
Table 12. Hyperparameter Settings for D2 MoE Experiments.
Hyperparameter
20% ratio
40% ratio
60 % ratio
70% ratio
80% ratio
Pruning ratio for Performance
10% Bose weights
Pruning ratio for Throughput
60% Buse weights preserve 26.62%
SVD Truncation Threshold for Performance
preserve 68.05%
preserve 47.34%
preserve 16.26%
preserve 5.93%
SVD Truncation Threshold Throughput
preserve 74.30%
preserve 53.58%
preserve 32.86%
preserve 22.54%
preserve 12.18%
Static Pruning Sparsity for Performance
of Base weights weights 30% Base weights
Dynamic Pruning Sparsity Performunce
Statie Pruning Sparsity Throughput
Dynamic Pruning Sparsity for Throughput
30% weights
Calibration Dataset Size
samples
Batch Size
128
C. Algorithm Tables and Pseudo-Code
We provide pseudo-code for key components of D2 to ensure reproducibility.
C.1. Pytorch like Implement of D2MoE layer
C.2. Pytorch like Implement SVD and Merge Process
Algorithm Base Weight Merge and Delta SVD Decomposition for D2MoE
wen.i.nnnonnnoe..oeeeo Xodula):
input original experts weighta, tiaher_ aca_e matrix def woriginsl_epppeeeeeeeeeeee
fisher_info, scale_matrix)
merge Base weight with fisher informaticn
weighted_aun sum (wwssfrrinnfofffffooel self.Base_M. veighted_sum
original_ experta [idx] weight for idx in rango (salf, f, nun_experts)) for idx range (sel..
for idx in range caloulate Delta weight
Delta_N oziginal_experts [idx] weightt self.Base. F decomposite Gelta weight with Irancation-Awars SVD v SVD (Dalta_X, paale_ratiix]1 ex..rttettttti De,lta self. [idx] Delta_V U, V
(M, ssale):
Truncation-Awav scale K_acale N scale
inv scale torch, linalg.inee inv U, sigma, y torch. avd
trun_sigma torch. diiign [: num num_trunc)) avd_u Ul: num_trune] torch. Itrue_alqmal
absorb seale into V avd_y torch.aqrt (truc_sigma) V[num_trune inv_scale return svd_u, svd_y
16

---

Delta Decompression for MoE-based LLMs Compression
Algorithm 1 D2MoE layer Implement
claas MoE Block inn (odule:
def
init
self, Rase_M
Baae Weight among exports
iutt.nin_exxertt
self sel1.D2_experta an.NoduleLiat (D2_ (D2_MoB_expertt dynamis pruning function
trouter function of MoB Block
self delf.dynamic_prmeer two_phase__"umn (aos x_aalibratior, target_sparsity)
def forward(x1:
routing_weights. F. F.saftmax (self.gate (x))
select expert maak for each token
expart_mask = torch. (routtinnnng
dim 1)
dynamic pruning indice during inference
graned_indices = ael..ddramie_praa
(x)
Base hidden
pre valoolate for experts with sparse inference
@asa_hidden_ctateen belf.Base_Mie praned_ind:cer
tinal_ridden__ Lina__tidde_stttees torch, zer zeroa_likee
expert_idk range www.e.ttmeeepeeeeeeeeeooo carrent_expert se sp___xxpets_xxperttt
selested
current expert
curront_stata = [[oxper=_mask]
correspend hidden states curreat expert
current current_Rase_hate www._aaadddd_sttteeeeeooeo btddddnnntaassssssssseesesssoeea hase_hidden_stateen (export_nosk)
calculate horrent_hidden_staee ourrent_expect Btapeeen__saateeerrett_BBaa.
token
expert
multiply current hidden
with router weights
-- routing_weights
add current hidden state final hiddan
sirai__iddee__ttee add tttpeatttttetttttteeeeeeeeeeeeee worederrnnnnncceeeeeettteeee return final_ hidden_atates
class D2 MoE expert (nn. :odule):
def init (1:
rank representation whighta
Waights
belf.Delta_] selt.Delta_5
def forward (x, current current_Base Base_hidden hidden_stateeeeeeee states):
delta_hidden_state hidder_ssateee states cal calculate from Base weights
current hateent_aseeehhaeeeeeee hidder
Dell.Delta_] (self Del..Deeta_iiii V(x)) return cursent_ Base htse_hiidee_sstttee
17

---

Delta Decompression for MoE-based LLMs Compression
C.3. Pytorch like Implement of two stage Semi-Dynamic Pruning Process
Algorithm Semi-Dynamic Pruning on Base Weight for D2MoE
two pww._phasspspnnnggoooe (R_b, X, target_sparsitty):
W_b: base weight matrix Im, nJ
input activations b]
target_aparsizy; desized sparsizy ratio le.g.r 0.5 means 508 pruned)
Phase
Pruning
static_metrics []
for in [n)
static pruning metric
c_j = norm_ (R 511 1213 :))
static_nerries..ppmmm
c_31)
sort the pruning metric sratic_netrica.ce (key x: x[1]]
static_k intin tazget_aparsity 0.5)
prarie (edx idx, static_metries [sstatic_k))
Phase Dynamic Pruning dynamic_prune (X_betch) sasaiccmmttrcca []
for 1 in range (A1:
j not statie_
C_1 = norm norm_1210_0n [M]b 1)) norm [orm_t22X__btchh :))
dynamic_metrrce..pppee ((), c_])1
dynemic a muw.tricc..cc. (key=lambda x: x(1]
dynamic_k int (n target_aparsity 0.51
dynam
(1dx idx, in dynamlo__etrics
return pruned dynamic_pruned
return dynamic prane tunction return dynemic_prune
C.4. Weighted Merge for Base Weights
Algorithm 4 Delta Decomposition for MoE Models
0: Input: Expert weights [W Metric
0: Output: Base weight W,, delta weights {AW;)
0: W, ELEN Metric,-W4 W; [Frequency mean, wishht- mean, and other merging methods...]
EnEN Metric,
0: for each expert i {1...., N do
0: Compute AW; W; W.
0: return Wo (AW;), =0
C.5. Truncation-Aware SVD Algorithm
Algorithm 5 Truncation-AAare SVD Compression
0: Input: Delta weight AW; activation matrix X
0: Output: Compressed delta weight AU, AV;
O: Compute activation Gram matrix S = XXT
O: Perform SVD: AW, S U E VT
0: Truncate Σ: Fuune TopK [X)
0: Reconstruct: AU;
trunee
AV, = V Yunnc
8-1
0: return AU, AV: =0
18


Reviews:
[{"review_id": "15d2IstP6n", "reviewer": "Reviewer_73q5", "summary": "This paper introduces D²-MoE for MoE Language Models. The author decomposes expert weights into a shared base weight and expert-specific delta weights, then compresses each component separately.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Dear Reviewer 73q5**\n\nThank you for your thoughtful review and constructive feedback. We appreciate your recognition of the novelty of D²-MoE and its strong empirical performance. Below, we address your concerns in detail.\n\n**Q1: The relationship between compression ratio and performance degradation is not analyzed in detail.**\n\n**A1:**\n\n(1) Table 2 demonstrates D²-MoE's robustness at various compression ratios. For example, on Mixtral-8×7B, performance declines gradually with increased compression (from 20% to 60%) without model collapse. Specifically, at **40% compression**, D²-MoE achieves an average accuracy of **0.57**, significantly outperforming NAEE (**0.48**) and MoE-I² (**0.49**). Even at **60% compression**, D²-MoE maintains **0.52** accuracy, whereas NAEE drops sharply to **0.36**.\n\n(2) Figure 4 shows how delta weight trimming affects D²-MoE's performance. As trimming increases, perplexity gradually rises, yet our method maintains stable performance even at extreme compression ratios of **81%**. Specifically, at **43%** compression (trimming 1 delta weight), D²-MoE achieves a WikiText-2 perplexity of **6.43**. Remarkably, even under extreme 81% compression, the model does not collapse, demonstrating the robustness and effective trade-off between compression ratio and performance.\n\n------\n\n**Q2: The interaction of D²-MoE with other compression techniques, particularly quantization, is not explored.**\n\n**A2:**\n\n(1)  ***Our current approach combines SVD decomposition and pruning, orthogonal to quantization.*** We focus on decomposing experts into Fisher-weighted bases and SVD-compressed delta weights without formally integrating quantization yet. However, our design allows straightforward quantization of low-rank delta factors ***(Section “Delta compression in MoE LLMs”).***\n\n(2) ***Existing quantization methods can be easily integrated into the D²-MoE framework:*** We plan to integrate quantization techniques to further reduce memory footprint, as demonstrated in the following table. Approaches such as GPTQ have shown effective quantization of delta and base-merged weights. Additionally, we apply the mixed-precision quantization method from [**MC-MoE**](https://arxiv.org/abs/2410.06270) to our D²-MoE.\n\n***Table D²-MoE+quantization***\n\n| Method                                       | WikiText-2↓ | PTB↓  | C4↓   | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA  | MathQA | Average↑ |\n| :------------------------------------------- | :---------- | :---- | :---- | :----- | ----- | :----- | :------ | :---- | :---- | :----- | :------- |\n| Mixtral-8x7B D²-MoE(25%) + GPTQ-4bit         | 5.34        | 22.03 | 9.56  | 0.288  | 0.761 | 0.741  | 0.556   | 0.460 | 0.771 | 0.345  | 0.56     |\n| Mixtral-8x7B GPTQ-3bit                       | 5.93        | 31.15 | 10.71 | 0.282  | 0.735 | 0.674  | 0.534   | 0.422 | 0.772 | 0.302  | 0.53     |\n| Mixtral-8x7B D²-MoE(40%) + MC-MoE-4bit       | 5.42        | 22.71 | 9.85  | 0.286  | 0.742 | 0.730  | 0.541   | 0.423 | 0.766 | 0.331  | 0.54     |\n| DeepSeekMoE-16B-Base D²-MoE(25%) + GPTQ-4bit | 7.62        | 12.61 | 12.94 | 0.264  | 0.707 | 0.655  | 0.511   | 0.373 | 0.769 | 0.275  | 0.51     |\n| DeepSeekMoE-16B-Base GPTQ-3bit               | 8.33        | 13.79 | 16.01 | 0.252  | 0.677 | 0.653  | 0.445   | 0.358 | 0.711 | 0.269  | 0.48     |\n\n\n\n------\n\n**Q3: The introduction should better separate the motivation from the proposed solution.**\n\n**A3:**\n\n(1) ***Problems with existing approaches:*** We will revise the introduction to juxtapose the storage/memory challenges of MoE (motivation) with our delta decomposition approach (solution), referencing new Table 1 (Section “Related Work”) to highlight how D²-MoE differs from pure pruning or merging. \n\n(2) ***D²-MoE's motivation:*** We emphasize that the moderate overlap (CKA 0.3–0.5) between experts to extract a shared base weight while preserving expert-specific diversity in compressed delta form. \n\n------\n\n**Q4: Figures (e.g., CKA similarity, singular value energy retention) need more detailed captions explaining their implications.**\n\n**A4:**\n\n(1) We will **expand the captions** of Figures 3 and 4 to explicitly describe what each metric represents.\n\n(2) For **Figure 3 (CKA similarity)**, we will clarify that **lower similarity values indicate that merging all experts directly would lead to performance loss**, justifying the need for delta decomposition.\n\n(3) For **Figure 4 (singular value energy retention)**, we will highlight that **delta weights exhibit strong low-rank properties**, making them well-suited for **SVD-based compression**.\n\n------\n\n**Q5: Minor typos need correction.**\n\n**A5:**\n\n(1) We will correct all typos, including those in **page 1 (\"successful compact\" → \"successfully compacts\")** and **page 3 (\"experts delta decomposition\" → \"expert delta decomposition\")**.\n\n(2) We will conduct a **thorough proofreading pass** to ensure consistency and clarity throughout the paper.", "claims_and_evidence": "The primary claim that D²-MoE outperforms existing compression methods is backed by comparative evaluations across multiple MoE architectures. The ablation studies further validate design choices.", "methods_and_evaluation_criteria": "The authors evaluate on both language modeling (perplexity on WikiText-2, PTB, and C4) and reasoning tasks (accuracy on seven reasoning benchmarks), providing a holistic assessment of model capabilities after compression.", "theoretical_claims": "The paper does not make formal theoretical claims requiring proofs, but rather presents empirically-grounded algorithmic innovations. The mathematical formulations are clearly presented.", "experimental_designs_or_analyses": "The experimental design is comprehensive. I verified the methodology for evaluating MoE compression across different models, compression ratios, and benchmark tasks.", "relation_to_broader_scientific_literature": "This work builds upon expertise from both MoE compression methods (like NAEE, MoE-I², and MoE-Pruner) and general LLM compression techniques.", "essential_references_not_discussed": "A few additional references would strengthen the context:\n\n\"ZipLM: Inference-Aware Structured Pruning of Language Models\" (Kurtic et al., 2023).\n\n\"Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning\" (Xia et al., 2023).", "other_strengths_and_weaknesses": "Strengths:\n\n1.The delta decomposition approach is a new contribution to MoE compression. The method achieves substantial compression ratios.\n\n2. The experiments on multiple models and benchmarks provide strong evidence for the method's effectiveness across diverse settings.\n\n3.  D²-MoE doesn't require expensive fine-tuning after compression, making it more practical for large models.\n\nWeaknesses:\n\n1.Limited more systematic analysis of the relationship between compression ratio and performance degradation.\n\n2.The paper doesn't examine how D²-MoE might interact with other compression techniques (like quantization) in a comprehensive compression pipeline.", "other_comments_or_suggestions": "The paper is well-written and organized, but could benefit from a few improvements:\n\n1.\tThe introduction could more clearly separate the motivation (problems with existing approaches) from the proposed solution (D²-MoE components).\\\n\n2.\tThe figures showing CKA similarity and singular value energy retention could benefit from more detailed captions explaining the implications of these results.\n\n3.\tMinor typos: \"experts delta decomposition\" → \"expert delta decomposition\" (page 3), \"Our D²-MoE successful compact\" → \"Our D²-MoE successfully compacts\" (page 1).", "questions_for_authors": "How does D²-MoE interact with quantization techniques? Since quantization is a common complementary approach to model compression, understanding whether these methods can be effectively combined would provide valuable insights for practitioners.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HAzNqOUte2", "reviewer": "Reviewer_LEQk", "summary": "This paper introduces D2-MoE, which decomposes expert weights into a shared base weight and unique delta weights. The delta weights are then compressed using SVD, and the base weight is further compressed using a semi-dynamical structured pruning strategy. The authors claim D2-MoE achieves better compression ratios and performance compared to other methods on models like Mixtral, Phi-3.5, DeepSeek, and Qwen2.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Dear Reviewer LEQk**\n\n**Q1: Careful discussion on relation with LoRA**\n\n**A1:**\n\n(1) Our framework structurally builds a multi-LoRA setup for MoE compression, consisting of a single base branch and multiple delta low-rank branches, enabling us to leverage existing LoRA research for further fine-tuning and ***efficient multi-LoRA inference***.\n\n(2) Unlike standard LoRA methods, our approach is a post-training compression framework that ***does not require additional training.***\n\n(3) Compared with similar methods such as [**LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation(ICML 2023)**] and [**Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (MC-sMoE, ICLR 2024 Spotlight)**], ***which also apply pruning and SVD but involve further training and both come from top-tier machine learning conferences***, our method significantly outperforms these approaches in performance and ***requires no training***, as illustrated in the table.\n\n***Table D²-MoE vs LoSparse(ICML23), MC-sMoE(ICLR24 Spotlight)***\n\n| Method                     | WikiText-2↓ | PTB↓    | C4↓     | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA  | MathQA | Average↑ |\n| :------------------------- | :---------- | :------ | :------ | :----- | :---- | :----- | :------ | :---- | :---- | :----- | :------- |\n| Mixtral-8x7B D²-MoE(20%)   | 4.65        | 16.32   | 8.59    | 0.33   | 0.80  | 0.75   | 0.61    | 0.51  | 0.81  | 0.39   | 0.60     |\n| Mixtral-8x7B MC-sMoE(20%)  | 5.00        | 15.36   | 9.68    | 0.336  | 0.794 | 0.766  | 0.603   | 0.503 | 0.794 | 0.380  | 0.59     |\n| Mixtral-8x7B MC-sMoE(40%)  | 4881.31     | 3276.94 | 4467.16 | 0.12   | 0.276 | 0.524  | 0.267   | 0.195 | 0.539 | 0.206  | 0.30     |\n| Mixtral-8x7B LoSparse(20%) | 953.51      | 805.16  | 1273.12 | 0.20   | 0.27  | 0.49   | 0.28    | 0.26  | 0.53  | 0.20   | 0.31     |\n\n(4) Additionally, further fine-tuning and knowledge distill to effectively recover performance, as shown in the table\n\n***Table D²-MoE+further training***\n\n| Method                          | WikiText-2↓ | PTB↓  | C4↓   | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA  | MathQA | Average↑ |\n| :------------------------------ | :---------- | :---- | :---- | :----- | :---- | :----- | :------ | :---- | :---- | :----- | :------- |\n| Mixtral-8x7B D²-MoE(40%) + KD   | 4.69        | 21.61 | 10.74 | 0.318  | 0.792 | 0.726  | 0.603   | 0.506 | 0.795 | 0.354  | 0.58     |\n| Mixtral-8x7B D²-MoE(40%) + LoRA | 4.54        | 16.04 | 9.17  | 0.31   | 0.771 | 0.739  | 0.604   | 0.463 | 0.789 | 0.348  | 0.57     |\n\n------\n\n**Q2: Discussion on MoE quantization methods.**\n\n**A2:** \n\n(1) Quantization and D²-MoE are ***orthogonal approaches:*** Quantization aims to reduce the model's precision, primarily speeding up inference and reducing memory usage ***rather than parameter count***, while our D²-MoE targets parameter reduction explicitly through delta decomposition, it does acknowledge quantization as a complementary approach in the related work section, mentioning techniques like BitDelta that successfully quantize delta weights.\n\n(2) ***Existing quantization methods can be easily integrated into the D²-MoE framework.*** \n\n***Table D²-MoE+quantization***\n\n| Method                                 | WikiText-2↓ | PTB↓  | C4↓  | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA  | MathQA | Average↑ |\n| :------------------------------------- | :---------- | :---- | :--- | :----- | :---- | :----- | :------ | :---- | :---- | :----- | :------- |\n| Mixtral-8x7B D²-MoE(25%) + GPTQ-4bit   | 5.34        | 22.03 | 9.56 | 0.288  | 0.761 | 0.741  | 0.556   | 0.460 | 0.771 | 0.345  | 0.56     |\n| Mixtral-8x7B D²-MoE(40%) + MC-MoE-4bit | 5.42        | 22.71 | 9.85 | 0.286  | 0.742 | 0.730  | 0.541   | 0.423 | 0.766 | 0.331  | 0.55     |\n\n------\n\n**Q3: The scalability of D²-MoE to larger models** \n\n**A3:** \n\n(1) Table 2 show results on DeepSeekMoE-16B-Base at up to 60% compression. While not the exact DeepSeek V3 variant, these 16B-parameter experiments reveal consistent advantages, indicating D²-MoE scales to large MoE. \n\n(2) Due to limited experimental resources, we plan to integrate advanced DeepSeek V3 configurations in future work. To demonstrate the scalability of D²-MoE to larger models, ***we conduct experiments on Mixtral-8x22B.*** Experimental results are summarized in the table below:\n\n***Table D²-MoE on larger scalability***\n\n| Method                    | WikiText-2↓ | PTB↓  | C4↓  | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA | MathQA | Average↑ |\n| :------------------------ | :---------- | :---- | :--- | :----- | :---- | :----- | :------ | :---- | :--- | :----- | :------- |\n| Mixtral-8x22B             | 2.95        | 10.1  | 6.14 | 0.37   | 0.86  | 0.80   | 0.67    | 0.59  | 0.83 | 0.50   | 0.66     |\n| Mixtral-8x22B D²-MoE(20%) | 3.99        | 14.61 | 8.65 | 0.36   | 0.83  | 0.78   | 0.63    | 0.55  | 0.80 | 0.44   | 0.63     |", "claims_and_evidence": "The claims are partially supported by the empirical results. The improvement against baseline methods is not very significant as Table 2 shows.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense.", "theoretical_claims": "N/A. There are no formal theoretical claims or proofs in the paper.", "experimental_designs_or_analyses": "I highly recommend the authors select some state-of-the-art baseline methods from top-tier machine learning conferences (i.e., ICML, NeurIPS, ICLR) for comparison, which makes their conclusion more convincing. Additionally, I suggest the authors use the conference version of each reference rather than the arxiv version.", "relation_to_broader_scientific_literature": "I am wondering how the proposed method relates to the idea of LoRA which also utilizes a delta weight.", "essential_references_not_discussed": "I suggest the authors carefully discuss the novelty of the proposed method against LoRA [1]. Additionally, the paper could discuss more recent work on the quantization method [2] for MoEs, as quantization is a common technique for further compressing LLMs. \n\n[1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" ICLR, 2022.\n\n[2] MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More. ICLR 2025.", "other_strengths_and_weaknesses": "Overall, the proposed method is interesting and well motivates the MoE compression problem.\n\nHowever, I have several concerns:\n\n- The scalability of the proposed methods to even larger MoE models needs to be addressed. It would be great if the authors could do experiments on larger MoE models such as Deepseek V3.\n\n- I would like to hear more about the novelty of this paper against LoRA which seems to leverage the delta weight idea as well.\n\n- The selected baseline methods are relatively weak (from ACL or findings of ACL). I suggest the authors compare their method with some SOTA baselines from top-tier ML conferences.\n\n- The paper should discuss more recent work on quantization methods for MoEs, as quantization is a common technique for further compressing large language models.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N4fWxvf3bm", "reviewer": "Reviewer_V5e9", "summary": "This paper  decompose expert weights into a shared base weight and expert-specific delta weights, allowing for effective compression while preserving expert diversity. The delta weights are then compressed using SVD and the base weights undergo semi-dynamical structured pruning. The paper provides extensive empirical validation on Mixtral etc, demonstrating superior accuracy compared to prior methods.\n\n## update after rebuttal\n\nThanks to the authors for their efforts to provide feedback. After the rebuttal, all of my concerns have been adequately addressed. \n\nThus, I tend to accept this submission.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Reviewer V5e9**\n\nThank you for your detailed review and for recognizing the novelty of D²-MoE and its strong empirical performance. Below, we address your concerns in depth.\n\n------\n\n**Q1: The criteria for setting the SVD truncation threshold should be explained.**\n\n**A1:**\n\n(1) We set an overall compression ratio (e.g., 40%), and to enhance performance, we prune 10% of the base weights. Under this setting, ***we can calculate the truncation threshold in D²-MoE accordingly.*** Detail can be seen in Table 7.\n\n(2) We integrate activation statistics by computing a scaling matrix from the activation Gram matrix (see Section3.3). This ensures that truncating fewer singular values preserves more essential information. And the scale matrix can be represented as $\\text{Cholesky}(X \\cdot X^T)$. And Table 6 demonstrates that truncation-aware SVD significantly outperforms vanilla and activation-aware SVD, reducing WikiText-2 perplexity to 5.28 compared to 6.22 for standard SVD.\n\n(3) Why we represent the scale matrix as $\\text{Cholesky}(X \\cdot X^T)$ can be proven as follows: When the scaling matrix $S$ is the Cholesky decomposition of $X \\cdot X^T$, we have $S \\cdot S^T = X \\cdot X^T$. Under this condition, the compression loss $L_i$ caused by truncating singular values equals precisely to the singular value $sigma_i$ itself. Consequently, truncating the smallest singular values results in minimal compression loss, theoretically justifying our use of the scale matrix defined as $\\text{Cholesky}(X \\cdot X^T)$.\n\n------\n\n**Q2: The theoretical justification for Fisher-weighted merging should be expanded—why is it better than simpler merging strategies?**\n\n**A2:**\n\n(1) We draw inspiration from model merging methods, where typically a base model is merged with various task-specific vectors. However, standard model merging techniques are ineffective for MoE expert merging since there is ***no explicit base expert weight.*** In contrast, ***Fisher merging directly operates on expert weights***, emphasizing parameters with higher gradient norms relative to the likelihood. This effectively identifies and retains the most critical expert weights.\n\n (2) Our experiments in Table 5 confirm that  Fisher-weighted merging consistently outperforms mean averaging and frequency-based merging. Fisher merging achieves a 5.28 perplexity on WikiText-2, compared to 7.66 for mean averaging and 6.42 for frequency-based merging.\n\n------\n\n**Q3: Does D²-MoE generalize to non-MoE architectures? Could this delta compression approach be applied to dense transformers?**\n\n**A3:**\n\n(1) We design D²-MoE specifically for multi-expert redundancy, but the principle of extracting a shared base plus compressed deltas is not limited to specialized experts. \n\n(2) We are now carrying out similar approach apply to large dense transformers ***by factoring out common subspaces from multiple layers and storing residual differences.*** \n\n(3) We leave systematically exploring dense variants for future work, as the our D²-MoE  currently focuses on MoE layers.", "claims_and_evidence": "Empirical results support the claim on performance, showing that D²-MoE consistently outperforms baselines.", "methods_and_evaluation_criteria": "Evaluation covers multiple MoE models and diverse tasks. The use of WikiText-2, PTB, and C4 for language modeling perplexity evaluation, as well as ARC-e, PIQA, HellaSwag, and others tasks.", "theoretical_claims": "The paper does not present significant new theoretical results but relies on well-established techniques like Fisher information weighting, SVD, and structured pruning. The Fisher-weighted merging is reasonable but would benefit from a more detailed theoretical justification of why it outperforms simpler merging strategies. The truncation-aware SVD is well-motivated, but a more formal discussion on how the truncation threshold is determined based on activation patterns would be useful.", "experimental_designs_or_analyses": "The experiments are comprehensive, covering various model sizes and compression ratios. The comparison across multiple baselines is a strong aspect of the paper, and the ablation on different merging methods (Table 5) is insightful.", "relation_to_broader_scientific_literature": "The paper is well-aligned with recent advances in Mixture-of-Experts compression, building on methods like MoE-I², NAEE, and MoE-Compress.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths:**   \n\n+ The core idea of delta weight decomposition is novel and well-motivated.\n\n+ Comprehensive experiments across multiple MoE models and datasets demonstrate clear performance gains. \n\n+ High reproducibility with detailed pseudo-code, supplementary experiments, and claimed code availability.\n\n**Weaknesses:** \n\n+ Limited details on SVD truncation: how is the threshold determined dynamically?\n\n+ Limited theoretical justification for Fisher merging: why is it better than other merge methods?", "other_comments_or_suggestions": "None", "questions_for_authors": "Does the method generalize to non-MoE architectures? Could the same delta compression be used for compressing dense transformer models, or is it fundamentally MoE-specific?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DKQIZ2Vp2B", "reviewer": "Reviewer_4qJS", "summary": "The paper presents D²-MoE, a new compression framework designed to tackle issues of parameter redundancy, memory usage, and storage inefficiency in MoE LLMs. D²-MoE enhances efficiency by breaking down expert weights into a shared base weight, which captures common features, and a delta weight that represents expert-specific differences. It  employs truncation-aware Singular Value Decomposition to compress delta weights effectively and incorporates a semi-dynamic pruning strategy to remove redundancies while adapting to input distribution. Experimental results show that D²-MoE outperforms existing methods like NAEE and MoE-I², maintaining high accuracy and low perplexity even at high compression rates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Dear Reviewer 4qJS**\n\nThank you for your insightful comments and for acknowledging the strengths of D²-MoE in terms of **efficiency, accuracy, and practical applicability**. Below, we address your concerns in detail.\n\n------\n\n**Q1: Motivation and Theoretical Justification of the D²-MoE Framework**\n\n**A1:**\n\nAs illustrated by the CKA similarity analysis in Figure 1, MoE models ***exhibit substantial similarities among different experts.*** Motivated by this observation, we propose merging experts into a shared representation. During this merging process, we utilize Fisher-weighted merging to emphasize critical expert weights effectively. Furthermore, ***to retain task-specific details and enhance model performance***, we introduce a delta branch by applying Singular Value Decomposition (SVD) to the delta weights.\n\n------\n\n**Q2: The dynamic reallocation of delta weights might introduce additional implementation complexity—can this be analyzed further?**\n\n**A2:**\n\n(1) While delta weight reallocation **adds computational overhead**, Table 4 shows that **D²-MoE achieves up to 1.5× inference speedup** even with dynamic structured pruning.\n\n(2) Figure 2 demonstrates that **delta weights exhibit strong low-rank properties**, allowing for **efficient compression without excessive reallocation overhead**.\n\n(3) We plan to further refine caching or partial updates to reduce overhead, but for now the two-phase pruning strategy offers a favorable trade-off between complexity and improved compression.\n\n------\n\n**Q3: The theoretical foundation of truncation-aware SVD should be elaborated on.**\n\n**A3:**\n\n(1) As shown in Table 6, ***naively truncating singular values leads to performance degradation and model collapse***. In contrast, truncation-aware SVD significantly outperforms both vanilla and activation-aware SVD, achieving a lower WikiText-2 perplexity (**5.28**) compared to standard SVD (**6.22**).\n\n(2) Therefore, we leverage the activation Gram matrix to compute a scale matrix represented as $\\text{Cholesky}(X \\cdot X^T)$. By introducing this scale matrix, truncating fewer singular values allows us to preserve more essential information.\n\n(3) Why we represent the scale matrix as $\\text{Cholesky}(X \\cdot X^T)$ can be proven as follows: When the scaling matrix $S$ is the Cholesky decomposition of $X \\cdot X^T$, we have $S \\cdot S^T = X \\cdot X^T$. Under this condition, the compression loss $L_i$ caused by truncating singular values equals precisely to the singular value $sigma_i$ itself. Consequently, truncating the smallest singular values results in minimal compression loss, theoretically justifying our use of the scale matrix defined as $\\text{Cholesky}(X \\cdot X^T)$.\n\n------\n\n**Q4: The discussion on combining D²-MoE with other compression techniques (e.g., quantization, knowledge distillation) should be expanded.**\n\n**A4:**\n\n(1) We integrate quantization to further reduce memory footprints in the following Table, following approaches like GPTQ that already show how delta weights can be quantized effectively. Additionally, we apply the mixed-precision quantization method from [**MC-MoE**](https://arxiv.org/abs/2410.06270) to our D²-MoE.\n\n***Table D²-MoE+quantization***\n\n| Method                                 | WikiText-2↓ | PTB↓  | C4↓   | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA  | MathQA | Average↑ |\n| :------------------------------------- | :---------- | :---- | :---- | :----- | :---- | :----- | :------ | :---- | :---- | :----- | :------- |\n| Mixtral-8x7B D²-MoE(25%) + GPTQ-4bit   | 5.34        | 22.03 | 9.56  | 0.288  | 0.761 | 0.741  | 0.556   | 0.460 | 0.771 | 0.345  | 0.56     |\n| Mixtral-8x7B GPTQ-3bit                 | 5.93        | 31.15 | 10.71 | 0.282  | 0.735 | 0.674  | 0.534   | 0.422 | 0.772 | 0.302  | 0.53     |\n| Mixtral-8x7B D²-MoE(40%) + MC-MoE-4bit | 5.42        | 22.71 | 9.85  | 0.286  | 0.742 | 0.730  | 0.541   | 0.423 | 0.766 | 0.331  | 0.55     |\n\n\n\n(2) Knowledge distillation can further enhance D²-MoE by **transferring knowledge from the full model to its compressed counterpart**, effectively preserving generalization capabilities. As demonstrated in the following table, applying advanced distillation methods indeed improves our approach's performance.\n\n***Table D²-MoE+KD***\n\n| Method                        | WikiText-2↓ | PTB↓  | C4↓   | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA  | MathQA | Average↑ |\n| :---------------------------- | :---------- | :---- | :---- | :----- | :---- | :----- | :------ | :---- | :---- | :----- | :------- |\n| Mixtral-8x7B D²-MoE(20%) + KD | 4.31        | 15.40 | 9.78  | 0.328  | 0.805 | 0.738  | 0.631   | 0.526 | 0.807 | 0.391  | 0.60     |\n| Mixtral-8x7B D²-MoE(40%) + KD | 4.69        | 21.61 | 10.74 | 0.318  | 0.792 | 0.726  | 0.603   | 0.506 | 0.795 | 0.354  | 0.58     |\n| Mixtral-8x7B D²-MoE(60%) + KD | 5.35        | 33.06 | 12.21 | 0.292  | 0.753 | 0.701  | 0.565   | 0.434 | 0.768 | 0.318  | 0.55     |", "claims_and_evidence": "The claims made in paper are generally well-supported by experimental evidence, particularly those concerning the performance of the D²-MoE framework in achieving higher compression ratios and maintaining or improving accuracy compared to existing methods. The clear experiments demonstrating improvements in perplexity and other metrics provide credible evidence.", "methods_and_evaluation_criteria": "The proposed methods for compressing MoE LLMs, including delta compression and semi-dynamical structured pruning, make sense for the problem at hand. They address well-known issues of redundancy and high memory usage in large-scale models. The evaluation criteria utilizing standard benchmark datasets (like WikiText-2, PTB, and C4) are robust, as they allow for meaningful performance comparisons against state-of-the-art methods.", "theoretical_claims": "The theoretical foundation requires further elaboration. Authors could delve deeper into the mathematical principles behind truncation-aware SVD.", "experimental_designs_or_analyses": "The experiments in the paper are relatively well organized including different MoEs and different sparsities, as well as comparisons with different compression methods. Ablations and comprehensive and offer useful insights.", "relation_to_broader_scientific_literature": "The paper builds on existing literature about MoE architectures and LLM compression techniques.", "essential_references_not_discussed": "There might be relevant works on other sparsity of Transformers [1,2],  which are recommended for citation and discussion.\n\n[1] Wang et al. Q-Sparse: All Large Language Models can be Fully Sparsely-Activated. CoRR abs/2407.10969 (2024).\n\n[2] Li et al. The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers. ICLR 2023.", "other_strengths_and_weaknesses": "Pros:\n\n1.\tD²-MoE maintains that the delta parameter is novel and insightful in MoE merging.\n\n2.\tThe performance results are compelling, demonstrating high efficiency and preservation of accuracy across different model scales.\n\n3.\tThe writing and organization are relatively clear, making complex concepts understandable.\n\n\nCons:\n\n1.\tThe theoretical framework lacks sufficient depth, particularly in justifying certain methodologies implemented within D²-MoE.\n\n2.\tThe dynamic nature of reallocating delta weights might introduce additional complexity in implementation.", "other_comments_or_suggestions": "This paper would benefit from deeper analysis of limitations.", "questions_for_authors": "1.\tAuthors could delve deeper into the mathematical principles behind truncation-aware SVD? \n\n2.\tAuthors could discuss the potential for combining D^2-MoE with other advanced compression techniques, such as quantization or knowledge distillation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method to decompose expert weights in MoE models into a shared base weight and expert-specific delta weights using SVD. By doing this, the paper has shown that it could achieve higher quality than other MoE layer compression methods.

All four reviewers had a consensus that the proposed method was able to effectively compress MoE layers. Most of the reviewers also appreciated its novelty and comprehensive evaluations. As a result, all four reviewers agreed on accepting the paper (3 accepts and 1 weak accept).

There were a few clarification questions, but the authors provided answers to resolve those.

Overall, this paper will be a good contribution to the conference and community.
Decision: Accept (poster)