Mixture Compressor for Mixture-of-Experts LLMs Gains More
Authors: Wei Huang
Yue Liao
Jianhui Liu
Ruifei He
Haoru Tan
Shiming Zhang
Hongsheng Li
Si Liu
XIAOJUAN QI
Venue: ICLR 2025 Poster (2025)
Keywords: Mixture-of-Expert
LLM
Quantization
Pruning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A Mixture compression method, which achieves extreme compression for MoE-LLMs with less accuracy loss, ensuring a better trade-off between performance and efficiency.

Abstract: Mixture-of-Experts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consumption and loading latency; and 2) the current activated experts are redundant, as many tokens may only require a single expert. Motivated by these issues, we investigate the MoE-LLMs and make two key observations: a) different experts exhibit varying behaviors on activation reconstruction error, routing scores, and activated frequencies, highlighting their differing importance, and b) not all tokens are equally important-- only a small subset is critical. Building on these insights, we propose MC, a training-free Mixture-Compressor for MoE-LLMs, which leverages the significance of both experts and tokens to achieve an extreme compression. First, to mitigate storage and loading overheads, we introduce Pre-Loading Mixed-Precision Quantization (PMQ), which formulates the adaptive bit-width allocation as a Linear Programming (LP) problem, where the objective function balances multi-factors reflecting the importance of each expert. Additionally, we develop Online Dynamic Pruning (ODP),  which identifies important tokens to retain and dynamically select activated experts for other tokens during inference to optimize efficiency while maintaining performance. Our MC integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss, ensuring an optimal trade-off between performance and efficiency Extensive experiments confirm the effectiveness of our approach. For instance, at 2.54 bits, MC compresses 76.6% of the model, with only a 3.8% average accuracy loss. During dynamic inference, we further reduce activated parameters by 15%, with a performance drop of less than 0.6%. Remarkably, MC even surpasses floating-point 13b dense LLMs with significantly smaller parameter sizes, suggesting that mixture compression in MoE-LLMs has the potential to outperform both comparable and larger dense LLMs.  Our code is
available at https://github.com/Aaronhuang-778/MC-MoE

Published as a conference paper at ICLR 2025
MIXTURE COMPRESSOR FOR MIXTURE-OF-EXPERTT LLMs GAINS MORE
Wei Huang" 1 Yue Liao 24, Jianhui Liu', Ruifei He', Haoru Tan'
Shiming Zhangi 1 Hongsheng Li 4, Li Xiaojuan Qit
The University of Hong Kong "hhe Chinese University of Hong Kong 'Beihang University CCnntee for Perceptual and Interactive Intelligence, Hong Kong
ABSTRACT
Mixture-of- of-eexerts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consumption and loading latency; and 2) the current activated experts are redundant, many tokens may only require a single expert. Motivated by these issues, we investigate the MoE-LLMs and make two key observations: a) different experts exhibit varying behaviors on activation reconstruction error, routing scores, and activated frequen- cies, highlighting their differing importance, and b) not all tokens are equally important- only small subset is critical. Building on these insights, we propose MC, a training-free Mixture- Compressor for MoE-LLMs, which leverages the significance of both experts and tokens to achieve an extreme compression. First, to mitigate storage and loading overheads, we introduce Pre-Loading Mixxe-Precission Quantization (PMQ), which formulates the adaptive bit-width allocation as a Linear Programming (LP) problem, where the objective function balances multi-factors reflecting the importance of each expert. Additionally, we develop Online Dynamic Pruning (ODP), which identifies important tokens to retain and dynamically select activated experts for other tokens during inference to optimize efficiency while maintaining performance. Our MC integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss, ensuring an optimal trade-off between performance and efficiency. Extensive experiments confirm the effectiveness of approach. For instance, 2.54 bits, MC compresses 76.6% of the model, with only a 3.8% average accuracy loss in eight commonsense benchmarks. During dynamic inference, we further reduce activated parameters by 15%, with a performance drop of less than 0.6%. Remarkably, MC even surpasses floating-pot 13b dense LLMs with significantly smaller parameter sizes, suggesting that mixture compression in MoE-LLMs has the potential to outperform both comparable and larger dense LLMs. Our code is available at
INTRODUCTION
Mixture-of- Experts large language models (MoE-LLMs) (Muennighoff et al., 2024; Jiang et al., 2024; Dai et al., 2024) provide efficient model-scaling mechanism by utilizing sparse architecture, in which only a subset of experts is activated by router. This selective activation boosts computational efficiency and scalability by assigning experts dynamically based on the specific needs of each input. Despite reducing the number of active experts to improve inference efficiency, MoE models still face significant deployment challenges. All experts must be loaded into memory simultaneously, and typically at least two experts are activated during inference, resulting in considerable memory and computational overhead. Even an NVIDIA A100-80GB GPU cannot accommodate typical MoE models like Mixtral 8x 7b (Jiang et al., 2024) (Fig. 1(b)). The proposed challenges hinder the deployment of L.LM with limited hardware resources which further promotes study on MoF-LLLM compression for better deploying model-scaling paradigm.
"Equal Contribution. Corresponding Author

---

Published as a conference paper at ICLR 2025
(4)
6x22bb
Alnuure Conpresorr
Per Perfoomunce (MMLII S-ahot)
00-
Parameters Size (GB)
AID
Joal Paranater
X3.6
11
Authoatd Phoamooos
50
42
Mastral6x7::22 frrm
Mixtral
Mistraiax:7 MCT
Mistrarex228 MC-M
MAMA LLMAAS-T Dicyfeeie Lis Micrrl 1o:
Mistral 8k 2088
Activated Parameters Size (billions)
Large Language Mudels
Figure 1: (a) MMLU (5-shott) accuracy across different open-source LLMs with various activated parameters (dot-lines denote the quantized models, solid-lines are 16-bit models). To align quantized models' parameter size with 16-bit models, we define 16bits as one parameter (e.g. 8x2-bit elements represent one parameter). (b) Comparison of total parameter size and inference activated parameter size on few open-source LLMs and compressed Mixtral 8b
The primary goal of compressing MoE-LLMs is to reduce the size of expert parameters, as they dominate the memory usage (Li et al., 2024). For instance, in models like Mixtral 8 x 7b, the number of expert parameters is 33 times greater than that of the attention modules. On the other hand, recent studies (Chi et al., 2022; Lu al., 2024) have shown that due to the training strategies of MoE, not all experts are equally important, which indicates that both the static experts during the pre-loading phase and the dynamic experts during online inference need to be compressed. Previous expert compression methods have typically focused quantizing expert weights during the pre-loading stage (Li ct al., 2024) or pruning experts during the inference stage (Lu et al., 2024; Koishekenov et al., 2022; Kim et al., 2021). Furthermore, vanilla uniform bit-width quantization and expert pruning based solely on routing scores struggle to maintain performance extremely high compression ratios. Therefore, in this work, are the first to explore extreme training-free mixture compression for MoE-LLMs, efficiently combining static expert quantization with dynamic expert pruning using a combination of expert importance metrics to achieve ultra-liggtwent
compressing single phase, such as
MsE-LLMs without significantly sacrificing performance.
To this end, we propose the MC, i.e., Mixture- Mixture-Compressso for MoE LLMs, exploring the combined benefits of expert quantization and pruning. MC consists of two phases: Pre-Louding Mixed-Preciss Quantization (PMQ) and Online Dynamic Pruning (ODP), as shown Fig. (a)) In the pre-loading phase, we focus on extreme compression of the stored experts through low-bit quantization. Our empirical study reveals imbalances in activation reconstruction error, routing weights, and frequencies of activated expert (Sec. 3 and Fig. 3), which inspires the allocation of different bit-widths to each expert. However, relying solely on the routing frequencies or scores is insufficient to accurately determine the optimal bit- width, as the two distributions may not be consistent but rather the opposite (Li et al., 2024). Therefore, we developed a weighted evaluation function that considers both the frequency and scores of expert activations, well the associated quantization loss different bit-widths. This function then minimized within a Linear Programming (LP) model to determine the optimal quantization configuration. Utilizing a training-free Post-training Quantization (PTQ) approach, GPTQ (Frantar et al., 2022), PMQ achieves high-performance compression at extremely low bit-widths (1 (1.5-bit-2.5-bitt, and our mixed-precision strategy is compatible with other advanced quantization techniques (Tseng et al., 2024; Chen et al., 2024; Shao et al., 2023; Egiazarian et al., 2024; Liao & Monz, 2024). As for inference phase, ODP dynamically prunes cow-coniieecce experts for each token based on the routing weights. Our pruning strategy follows two key principles: first, experts with significantly lower routing scores are categorized as "low confidence" and can be pruned (Lu et al., 2024). Second, to prevent attention degradation that solely relies routing weights, we protect important tokens by considering both attention scores and feature magnitudes. Experiments show that protecting only 2% of the important tokens effectively mitigates pruning loss while maintaining nearly the same compression ratio.
The proposed mixture compression of low bit-width experts improves performance compared to uniform quantized experts or other mixed -reccssir strategies, even surpassing float-point (FP) models with the same number of activated parameters. Moreover, when compressing Mixtral 7b to
2

---

Published as a conference paper at ICLR 2025
around 8b (2.54-bit), its activated parameters amounted only 2b, while even outperforming 16-bit LLaMA2-13B by around 8% on the MMLU (5-shot), as shown in Fig. (()). Mixture compression exploits the disparities between MoE experts, for the first time enabling surpassing of smaller FP models of equivalent size under extreme compression without training. This achievement underscores the significant compression potential and practical utility of sparse MoF-LLMMs.
RELATED WORKS
Mixture-of-Experr LLMs. LLMs have achieved significant advancements across various natural language domains (Chang et al., 2024; Zhao et al., 2023). Despite their success, these models rely heavily on dense parameters, which presents significant challenges for deployment(Zhou et al., 2024; Zhu et al., 2023). Sparse activated MoE models have been identified as an essential strategy to enhance the cost-performance balance in LLMs. In MoE models, each layer is comprised of several experts, with each token activating only a specific subset, thereby significantly improving efficiency compared to dense models, which activate all parameters for every input (Shazeer et al., 2017; Yun et al., 2024). Recent advancements in LLMs (Brown, 2020) have further popularized MoE-based architectures (Jiang al., 2024: Muennighoff et al., 2024). Industry-eeading models such as Mixtral 8 x 7b (Jiang et al., 2024) and Deepseek-R 1 (Guo et al., 2025) also incorporate this technology.
Quantization for LLMs. Post-Training Quantization (PTQ) is an efficient method that requires no additional training, making it well-suited for large-scale LLMs (Dettmers et al., 2022: Frantar et al., 2022; Xiao et al., 2023; Shao et al., 2023; Lin et al., 2024). Previous studies have investigated the diverse salience of weights and proposed mixed- precision methods to improve low-bitwidth mixed-precision
performance by allocating different bitwidths accordingly (Dong et al., 2020; Huang et al., 2024c;
Dettmers et al., 2023; Shang al., 2023; Huang al., 2024a). Recent research introduced expert- guided, block-wise mixed-precision benchmark for MoE-LLMs to address the disparities in expert weights (Li et al., 2024); however, developing more effective expert-wiss quantization strategies remains a challenge. Codebook-basee encoding approaches enable more precise quantization of LLMS and enhance post-quantization performance through fine-tuning (Egiazarian et al., 2024; Tseng et al., 2024). While Quantization Aware Training (QAT) requires significant resources (Chen et al., 2024; Liu et al., 2023b), QAT-based retraining strategies PTQ combined with additional fine- tuning (Liao & Monz, 2024; Guo et al., 2023; Huang et al., 2024b) are more effective in maintaining the performance of quantized lightweight LLMs.
Parameter pruning for LLMs. Parameter pruning is another effective method for neural network compression (Kwon et al., 2022; Hubara et al., 2021), and it has recently become crucial in reducing the size of LLM weights (Frantar & Alistarh, 2023; Sun et al., 2023). Traditional pruning approaches focus on two main techniques: structured and unstructured pruning, both of which selectively zero out certain parameters based on their importance (Zhou et al., 2024). In MoB-LLMs, less important experts can be pruned based on activation frequencies or the statistical characteristics of gating (Kim et al., 2021; Koishekenov et al., 2022; Liu et al., 2024). During the model preloading stage, pruning tends to incur greater loss than quantization at the same compression rate. However, dynamically adjusting the quantization bit-width during inference remains challenge, whereas pruning offers the flexibility to dynamically select activation parameters during inference (Zhu et al., 2023). Recent work by (Lu et al., 2024) has explored dynamically activating top-k experts based on gating weights in MoE, significantly improving inference efficiency.
3 METHOD
3.1 PRELIMINARIES
Mixture-of-Experts LLM. In decoder-only MoE-LLMs, conventional feed-forward networks (FFN) are replaced by MoE layer, each having N experts (Gale et al., 2023). The MoE- LLMs selectively activates the top-k experts for different tokens by a group of routing scores Wtop-k {Wo, W1, .... wk-1]. }. k N, generated by gating layer G(t). Fig. 2(a) illustrates the experts selection mechanism during the inference phase based on routing scores. Specifically, in the Mixtral 8 7b model, there are experts and each token is routed to the lop-2 experts (Jiang et al.,
3

---

Published as a conference paper at ICLR 2025
(a)
(b)
Top 2% Protection
8
E2
Acthation
Geting
Multi Head SelE-Attention
Malti- Self self-Attention
Pre-Loading Static Mixed-Precision Quantization
Online Dynamic Mixture Pruning
Figure The overview of our proposed MC pipeline with two stages compression for experts. (a) Framework of pre-loading static mixed-precision quantization (PMQ) of MoE-LLMs. PMQ determins the activated feature and loss sensitivity of experts and plans the optimal precision configuration under ultra-low -bit-width. (b) Schematic of online dynamic mixture pruning (ODP) of MoE-LLMs. ODP utilizes significant token protection mechanism with weigh-guide experts pruning, which only need to keep 2% token successfully safeguard the MoE performance.
2024). The output of each token in MoE layer is calculated
E
20 Et),,,
(I)
WLETop-2(G(tt)
where E; represents the feed-forward operator of the i-th expert and w; is the routing weights/scores calculated by the gating G(t). Therefore, according to the definition in Eq (1), the routing mechanism establishes the correspondence between tokens and experts.
Quantization Technique. Since the substantial memory overhead of MoE models mainly arises from the weights of its experts (over 96% weights of the model), quantization is employed for the experts. Specifically, floating point weights distributed in the interval [W min, W mar] are mapped to the integer range of [0, 2...,,2) where represents the target bit-width, and the quantization reconstruction for the weights W E pinxout can be defined as:
arg min
|[X - W
where 4 denotes the quantized weight and || ||2 is the mean square error (MSE) loss. The primary objective of this study is explore the optimal mixture compression strategy for MoE-LLMs. To this end, we employ the efficient PTQ scheme, GPTQ (Frantar et al., 2022), as our foundational tool. By utilizing bbased estimation (H 2XX and quantization
compensation, GPTQ effectively reduces the group-wise quantization error of weights, enabling the quantization of Mixtral 7b within 90 minutes. This work focuses on the design of optimal mixture compression strategies for MoE-LLMs and is therefore orthogonal to other quantization techniques, including PTQ methods (Shao et al., 2023; Lin al., 2024), rodebook-basee works (Egiazarian al., 2024; Tseng et al., 2024), and even the deployment of fine-tuning (Liao & Monz, 2024) or QAT (Chen et al., 2024; Liu et al., 2023b) can deployed for MC, additional evidences are shown in Appendix A.3.
3.2 PRE-LOADING MIXED- PRECIIPRE QUANTIZATION
As outlined in Sec. 3.1, the primary storage overhead of MoE-LLMs resides in the experts, necessitat- ing compression before loading onto devices. Mainstream LLM pruning suffers from performance degradation under extreme pruning conditions (250%) (Frantar & Alistarh, 2023; Sun al., 2023; Lu et al., 2024), whereas quantization has been demonstrated to achieve high levels of compression with lower performance drop (Huang al., 2024b; Zhou et al., 2024). Moreover, as shown in (Li al., 2024) and our Sec. 4. uniform bit-width quantization does not meet the extreme compression accuracy requirements for MoE-LLMs. Therefore, the diverse and uneven features of experts inspire us explore the optimal mixed-precision quantization approaches.
In this section, we introduce our Pre-Loading Mixed Precissin Quantization (PMQ) method, designed to effectively reduce the model size by applying pargetd--xpeers quantization. The core focus of PMQ is optimizing the bit-width allocation strategy for experts. To this end, we begin by conducting thorough analysis of experts" behavior on the calibration dataset and leverage this information
4

---

Published as a conference paper at ICLR 2025
Frohenius Map
Weight Score Map
Activation Frequency Map
CV
Mark
Figure 3: Distribution of expert drop F-norm (red), activated weights (green) and frequencies (blue) in the Mixtral 8 x 7b model, encompassing 32 MoE layers with 8 experts per layer. The top set of the heatmap calculated through C4 dataset, and the bottom set calculated through MATH dataset. MoE-LLMs selectively activate top-2 experts in each MoE layer, wherein a significant portion of experts remain less important or inactivated all the time.
design Integer Programming (IP) model that solves the optimal quantization configuration. For other components of the model, such as attention parameters, we apply the same bit-width.
3.2. EXPERTS SIGNIFICANCE ANALYSIS
The core principle of our expert quantization strategy is grounded in the significance of each expert, which enables the allocation of bit- bit-widths according to their relative importance within a block. We initially observed the performance of different experts in Mixtral x 7b in terms of expert- drop reconstruction loss (Frobenius norm) (He et al., 2017), and activation features on the dataset C4 (Raffel et al., 2020) and the specialized domain dataset Math (Hendrycks et al., 2021). As shown in Fig 3, the impact of experts on the model varies widely: 1) some experts, such as the one position [2, 4] (Fig left), have minimal influence on the output activation reconstruction loss, while others in layer 2 exhibit significantly higher losses, highlighting the imbalance among MoE-LLMs' experts; 2) the activation scores and frequencies reveal distinct patterns, where experts at positions [11, 3] and [12, 7] show extremely low activation frequencies and average scores, while the expert at position [1,3] has low scores but comparatively high activation frequencies; and 3) in task-specific contexts like mathematics, MoE activates fewer experts, resulting in a sparser distribution than general tasks. This variability in routing feature inspires the need to consider multiple factors in determining the optimal bit-width allocation for experts.
We mainly measure the significance of each expert through two factors: access frequency and activation weight. Given an sizeze calibration dataset C4 (general language understanding dataset), first perform inference refers to the rate at which the expert is activated. Thus, i-th expert's access frequency is di N. where n; is this expert's 's total activated number. A higher activation frequency indicates that the expert is triggered more often, suggesting its generality and applicability across a wide range of tokens. However, access frequency alone overlooks the potential significance of experts who are rarely activated. To account for this, we introduce the routing weights assigned each expert during inference. This metric for i-th expert can be denoted
the original 16-bit MoE-LLMs. For each expert, access frequency
metric, which sums the
as wi -i=:" 01 where oj is the expert's 's routing weight the j-th inference. This provides a finer-grale seer-graieed measure of an expert's contribution in MoE-LLMs, capturing its relative importance beyond mere frequencies. The final expert significance is computed of w.f where and are hyperparameters used to balance the two factors.
N
3.2.2 OPTIMAL EXPERTS BIT-WIDTH ALLOCATION WITH WEIGHTED IMPORTANCE FACTORS
After obtaining the expert significance, we proceed to explore how to leverage this significance for mixed-precisies quantization. The core idea is to assign different bit-widths to each expert based on its importance, preserving the contributions of more significant experts with higher bit-widths while applying more aggressive quantization to less significant experts. In addition to considering expert significance, we evaluate the reconstruction emror of output activations in each MoF. layer post-quantization, which allows us to quantify the impact of individually quantizing each expert. Specifically, for a given expert, we compute the Frobenius norm (F-norm) between the output of the model when this expert quantized and the output when no quantization is applied to any experts.
G = || (0) F (0le:
(3)
where F (0) is the model output with full parameters 6, and I ((ßee + Q(e,,))) represents the output when only expert e; is quantized bits. Q(-) denotes the quantization function.
5

---

Published as a conference paper at ICLR 2025
Block 15, Head 4
Block 15, Head 4
Block 15, Head
Drop one Export of T
Keep tap-2 Experts of Tokert
Weight-unly espert pruning Bluck
Attention-Sink Tokens
Attention-wware experts prusing Block
Figure 4: Typical attention map of block 15, head 4 in Mixtral x 7b under different dynamic pruning process. The middle with out pruning shows that attention, in a column-wise manner, highlighted several tokens with high scores, such as token 31 and token 67. However, after undergoing traditional weight-only pruning through block 14 layers, experts pruned at the position of token 67, resulting in a decay in the attention map. Through attention-aware pruning based on token importance, block 14 protected token 67, thereby avoiding attention decay in the subsequent layer.
Our goal is to ensure that the extremely-low average bit-width across all experts in a MoE block equals targeted value k with bit-width options restricted to {1,2,3}-bit. To achieve this, formulate the problem as an Integer Programming (IP) optimization, which only takes a second to finish the bit-width allocation computing. The IP model is defined as:
R
MINIMIZE
M (Ei. #zy)"
SUBJECT TO
j Zij = n ki
Tij =1 1, Vi,
(4)
j=
Σ TB 2 1,
Σ 242 2 1,
Iy E (0. 13, Vi.j.
Here, Bay is binary variable indicating whether the i-th expert is quantized to bits (riz = 1 if true, Ty = otherwise). To preserve the accuracy of key experts, we enforce a constraint that at least one expert must be quantized to 3 bits and at least expert to 2 bits. is weighting hyperparameter. After determining the optimal bit-width combination for each MoE Block expert, we apply the GPTQ quantization algorithm to quantize the experts accordingly. For the remaining weights in the attention or gating module, considering their extremely small parameter size, we quantize them to 4-bit, resulting in an introduced average bit-width of more than 0.05 bits.
3.3 ONLINE DYNAMIC PRUNING
Our PMQ strategy compresses the storage memory of experts during the pre-loading phase; however, the selection the top-k experts during inference still incurs high computational costs. As discussed in Lu et al. (2024), not all tokens require & experts for inference. To optimize efficiency while maintaining performance during inference, in this section, we introduce the Online Dynamic Prun- ing (ODP) technique, which identifies important tokens to retain and dynamically selects activated experts for other tokens.
3.3.1 ATTENTION DECAY UNDER WEIGHT-ONLY PRUNING
To effectively perform dynamic experts pruning, an intuitive and efficient method involves utilizing the top-k experts' routing scores during inference (Lu et al., 2024; Huang et al., 2023). This approach directly skips experts with lower routing weights among selected set for each token. For simplicity, when lo = 2 (as in Mixtral 8 7b), the pruning process follows:
W1
(wo wi 1, wo. w1 Top-2(((())
(5)
Wo
Wo and Wh denote the 2 experts, respectively, with μ erving as a hyperparameter threshold for each MoE layer. This threshold is set at the median value of derived from calibration data Wi

---

Published as a conference paper at ICLR 2025
(Lu al., 2024). According Eq. (5), when selected expert has notably low weight, it is feasible to be pruned for the current token, thus retaining only the primary expert for computation. Sec. 4.2 documents that employing this weight-basee dynamic pruming strategy reduces computational demands by 15%, but also incurs a performance decrease of approximately 10%. Further examination reveals that this decline is due to "attention decay" effect, which is evident in Fig. 4. Specifically, unpruned conditions show a pronounced vertical pattern in the attention map of block 1 15, head 4, at token 67 (Fig. 4, middle). However, the application of weight-only pruning in block 14 results in the elimination of one expert for token 67, leading to significant reduction in the attention map score at token 67 in block 15 (Fig. 4, left). This effect is herein defined as "attention decay."
3.3.2 SIGNIRICANCE AWARE TOKEN PROTECTION
Weight-only pruning considers only the routing weights of experts, but overlooks the intrinsic importance of tokens. However, the output capabilities of LLMs are often influenced by a few critical tokens (Zhang et al., 2024; Guo al., 2024; Nrusimha et al., 2024), and considering weights alone, illustrated in Fig. 4, can lead to the pruning of experts corresponding to salient tokens. To circumvent the issue of attention decay, we introduce a simple but effective method that safeguards the computational experts of the most critical tokens in dynamic inputs from being pruned. Inspired by (Guo et al., 2024), we introduce an evaluation metric for token importance:
1 = 11tt Ejsigt Aja L-j
(6)
where I denotes the importance of the i-th token, || || is the E, norm, and the total length of input
tokens is L. A represents the attention map in an LLM block, calculated from A = -sftmax K KTQ
of this layer. Considering the co-effects of token magnitude and attention socres, Eq. 6 flexibly combines these two factors to accurately define the importance of each token.
As demonstrated in Fig. 4, right, introducing important token protection into weight-only pruning effectively mitigates attention decay issues. Due to the high importance parameter I67 of token 67, all experts are preserved for computing token 67 in block 14, thereby preserving the expected distribution in the attention map of block 15 for token 67. Experiments in Sec. 4.2 indicate that selectively protecting merely 2% of important tokens can significantly reduce performance losses in MoE-LLLMs, while still maintaining a computational efficiency improvement of approximately 15%. We also provide the detailed computation overhead analysis in Appendix A.9.
4 EXPERIMENT
In this section, a series of experiments are conducted to evaluate the proposed MC. We present by describing the experimental parameter settings and results. In Sec. 4.1, we assess the parameter settings for the PMQ method and the performance of mixture quantization. We conduct a detailed evaluation of the performance loss and compression efficiency of ODP stage, shown in See. 4.2. Finally, we present the combined performance of MoE mixture compressor.
Experiment Setup. The mixed-precision fac- tors of experts are calibrated from C4 (Raffel al., 2020) dataset, with 128 sets of random sequences, each 2048 tokens long. After de- termining the bit-width configuration, the final quantization process follows the GPTQ (Frantar al., 2022) procedure. We select the open- source Mixtral 8 x 7b and Mixtral 8 x 22b as our target models, shown Tab. 1. Mixtral x 7 can be compressed on two NVIDIA A100-80GB GPUs, while Mixtral 8 x 22b is completed on four NVIDIA A100-80GB GPUs.
Table 1: Selected MoE-LLMs and model configu- rations. Size: the total parameter size, Act Size: ac- tivated parameter size per-token; B: decoder block number, H: hidden dimension, E: expert number.
Model
Size
Act Size
H
E
Mixtral 8 7b
49h
13b
32
4096
8
Mixtral 8 x 22b
141b
39b
56
6144
8
Other None-MoE layers are set to 4-bit. Due to the significant size of expert weights, the 4-bit quantization of other parameters has minimal impact on the average bit-width. In the performance experiments for the proposed MC, perplexity (PPL.)) was chosen as the metric to evaluate token prediction capabilities. primarily deploying the general text dataset 2ik. To comprchensively assess the language capabilities of the compressed LLMs, we evaluated the models' overall abilities in eight zero-shot benchmarks (1) tested by FleutherAl LM Harness (Gao al., 2013).
7

---

Published as a conference paper at ICLR 2025
Parasim brtww
Venghe Proben
Activoeed Wegtt
104
actioated ree undry
Wiifherd Barlecty pPu1
PNO
Probeniza
20-
FP20
107-
Uninamm 20r
35-
001
Biowith
Figure 5: Quantized PPL performance of Mix- tral 8 7b under different mixed -precisiinn strategies (with random allocation)
Figure 6: Quantized PPL performance of Mix- tral 8 7b under different mixed-precision strategies (without random allocation).
Table 2: Performance of quantized Mixtral 8 x 7b on eight zero-shot benchmarks. We deploy GPTQ as our baseline PTQ method for uniform quantization. "Uni" denotes the uniform quantization of 2-bit with GPTQ. Since the results some data sets in the block score predictor (BSP) (Li et al., 2024) method were not reported, repository and evaluated all the results under the same settings. In BSP, 25% MoE layers are 4-bit and the left are 2-bit to achieve 2it4-bit. "HellaS." is the short format of "HellaSwag" and "Wino." denotes "Winogrande". + gives the accuracy loss between quantized results and original 16-bit model.
resumed the relevant quantized model from the official code
Method
Bits PIQA ARC-e ARC-c BoolQ HellaS. Wino.
MathQA
MMLU Avg.5 1
16.00 85.20
84.01
57.17
85,35
81.48
75.93
39.29
67.88
71.29
Uni
3.00 82.10
78.58
55.80
82.94
79.28
74.19
39.26
60.58
69.092.299
Uni
2.00 61.98
47.20
25.71
62.39
41.91
53,22
22.79
30.36
42.67
BSP
2.54 68.23
54.97
28.38
68.16
55.61
62.19
24.07
27.74
29.07
(Li et al., 2024)
2.34 80.2T
76.38
31.20
81.1i
78.05
772.97
35.277
56.21
67.1887.161
Hessian
2.05
75.32
67.26
45.01
70.29
71.90
69.11
31.07
40.85
58.8512.4499
(Dong et al., 2020)
1.57
65.26
52.12
21.84
68.21
52.91
50.32
24.99
31.58
25.9125.498
2.54
80.52
77.10
51.28
82.54
79.03
73.95
39.18
56.37
67.50
3.8%4
2.42
80.36
75.76
50.17
80.00
78.13
73.09
34.97
53.22
65.71
5.6%
2.30
83.11
73.59
47.78
80.83
76.48
73.14
33.84
52.54
64.91 6.4%%
2.20
79.05
73.70
47.87
74,56
76.63
72.77
34.24
47.73
63.29
8.0%K
PMQ
2.05
79.16
73.06
48.38
80.58
74.95
71.27
31.79
46.80
63.25 8, 0%%
1.94
76.88
68.48
45.48
75.23
72.05
72.61
31.16
40.93
60.35
10.944
1.81
76.93
66.67
43.60
75.50
70.50
69.85
28.68
40.71
59.06
(2.2944
1.69
75.41
64.14
40.61
68.96
67.01
68.03
28.04
37.14
56.17
19.1
1.57
72.42
62.46
37.88
73.55
63.17
66.38
26.80
32.25
54.491a.8%%
4.1 EXPERIMENT ON PRE- LOADING MIXED- PREED--REEISION QUANTIZATION
Ablation of Bit- Width Allocating Metrics. Fig. 5 illustrates significant decline in model perfor- mance with random bit-width allocation. And employing only the routing scores of experts from calibration data, the curve in Fig. 6 though better than random allocation, the PPL curve is still high. However, activation frequencies, comparison to weight, shows better performance.
Furthermore, in conventional networks and dense LLMs, Hessian-based quantization loss is common use for bithw allocation (Dong et al., 2020; Huang et al., 2024c). We also utilized as compared metric for expert-wise bit-width allocation. Fig. 6 also contains three metric curves: Hessian, F-norm, and PMQ. F-norm and PMQ are more effective than Hessian for expert-wise bit-width allocation, exhibiting better performance under different bit-widths. When the average bit- hit-width exceeds 2-bit, the F-norm is similar to the PPL curve of PMQ; below 2-bit, the lead of PMQ gradually widens.
Comparison of Mixed-Precision Quantization. We present comprehensive comparison of the performance of PMQ within the ultra-low bit-width range. GPTQ was set as the baseline for uniform bit-width quantization, denoted "Uni" in Tab. 2. We also compare it with recent mixed-precision
8

---

Published as a conference paper at ICLR 2025
PPL and CR K5
PPL and
bepiort (0011
3.4-
2.4
1.4
camprnoon wodh
4.25
3.-
4.30-
222
0300
1836
8:0 E:t 1ai 3.ck 520
cm 1977
1033
01. Maiting
1992
1907
01.0
1179
Figure 7: Significant tokens protection of 2.05-bit Mixtral 8 x 7b. "CR" denotes the average com- putation compression ratio (blue); "PPL" denotes the perplexity(red); Star represents the weight- only pruning performance.
Figure 8: Less significant tokens drop of 2.05- bit Mixtral 8 x 7b. In Mixtral 8 x 7b, we mask all experts of the less significant tokens. "CR" denotes the average computation compression ratio (blue); "PPL. denotes the perplexity(red).
approach for MoE-LLMs known as the block score predictor (BSP) (Li et al., 2024). Following Eq. 4, we set the average bit-width of Mixtral 7b within the range of 1.57-bit to 2i54-bit. As shown in Tab. 2, the uncompressed 16-bit model achieves an average accuracy of 71.29%. With uniform precision quantization, the average loss for the 3-bit model is approximately 2.2%, while the loss for the 2-bit model increases significantly by 28 6%, highlighting the challenges in maintaining model accuracy with existing uniform precision quantization methods at ultra-low bit widths.
BSP achieves average accuracy of only 49.07% at 2.54-bit. However, our proposed PMQ achieves performance of 67.50% ceeding BSP by 18.4% and only falling of the 16-bit Mixtral x 7b by 3.8%. Notably, PMQ can maintain a accuracy of 54.49% at 1.57-bit, even outperforming BSP at 2.54-bit by 5.4%. The Hessian- Hessian-based method in Tab. 2 consistently underperform to PMQ across varying bit-width levels. Specifically, Hessian shows slight un- derperformance of 0.2% at 2.54-bit, while PMQ demonstrates more substantial advantages be- low 2-bit, leading by 9.4% at 1.5t-bit. Further- more, also evaluated the few-shot capability of PMQ in Tab. 3, where PMQ continues to demonstrate superior accuracy. More bit-widths performance results are shown in Appendix. A.1.
Table Comparison of different mixed-precision strategies on few-shot performance (MMLU five- shot 1) for Mixtral 8 7b. More results of different bit-widths are shown in Appendix. A.I.
Method
Bits
Accuracy % t
16.00
70.60
Uni BSP ct 2024)
2.00
34.05g6 6%14
2.54
31.57
2.54
558.
Hessian (Dong et al., 2020)
2.05
43.5127.19%
1.57
31. 31.96us 69K.
2.54
61.19,.4%
PMQ
2.05
49.8420.u 20.8886
1.57
33.4437.2%%
4.2 EXPERIMENT ON ONLINE DYNAMIC PRUNING
In pre-loading phase, PMQ enables the compression of MoE-LL to exceptionally low bit-width range. Furthermore, during the inference phase, we apply the ODP outlined in Sec. 3.3 to the quantized MoE model, further enhancing the efficiency real-time inference for lightweight models.
Ablation of Tokens Protection. As shown Fig. 7, when we select 2% crucial tokens to be protected, the PPL drops from 6.46 6.24, with activated experts' parameters decreasing only from 15.1 % to 14.8% Moreover, as we gradually increase the ratio, the performance remains relatively stable, while the compression ratio exhibits a nearly linear decline. Thus, we conclude that protecting just 2% of the important tokens can significantly enhance the compressed performance of MoE-LLMs with almost no impact on efficiency. Furthermore, we try to prune experts associated with the less important tokens, as illustrated in Fig. 8. When removing the experts to the 2% of tokens, the overall compression ratio reached 15.8%, while the PPL improved 6.35, yielding performance enhancements in both efficiency and accuracy compared to weight-only pruning. However, we observed that the performance curves of all experts' masking exhibited exponential growth, indicating that directly skipping experts results in significant accuracy loss. By employing protection mechanism for only 2% of the experts, we can maintain the accuracy of MoÆ-LLMs without compromising efficiency. We also provide the detailed ablation on pruning threshold in Appendix A.8.

---

Published as a conference paper at ICLR 2025
Table Ablation evaluation of PMQ and ODP for Mixtral 8 x 7/22b, and compared with dense LLaMA models. "Params" denotes the parameter size, and "Act Params" is averaged activated pa- rameters for one token. The parameter calculation of the compressed model includes the compressed weights and quantizer parameters (e.g., scaling factor and zero factor for dequantization). We carry out the average activated parameter size and speedup on C4 dataset. 16-bit Mixtral x 7b uses A100-80GB GPUs and Mixtral x 22b uses quantized models tested on A 100-80GB GPU.
LLMs
Bits
PMQ
ODP
Uni LM-Eval% 1
Params.(GB)
Act Params.(GB)
Speedup
LLaMA2-7b
16.00
61.52
13.48
13.48
LLaMA2-13 13b
16.00
65.19
26.03
26.03
16.00
71.29
96.80
26.31
1.00x
2.00
42.67
13.61
3.70
1.72x
2.54
67.50
16.24
4.53
1.63x
2.54
16.24
3.96
1.71x
Mixtral 8 7b
2.05
63.25
13.41
3.73
1.67
2.05
62.68
13.41
3.23
1.80
1.57
54.49
10.82
2.94
1.82x
1.57
53.77
10.82
2.55
1.89
16,00
76.33
281.24
76.49
x.00
2.00
50.44
38.08
10.35
1.95x
2.54
72.08
46.58
12.66
1.77
2.54
71.21
46.58
10.96
1.82x
Mixtral 8
22h
2.05
67.94
38.35
10.42
1.80
2.05
66.50
38.35
9.03
1.87
1.57
59.29
30.27
8.23
1.97
1.57
58.84
30.27
7.13
2.06x
Memory Saving and Inference Efficiency. Tab. 4 details the memory compression, speed tests, and average results (Gao al., 2013) (LM-Evva) of the proposed MC. The 16-bit Mixtral 8 x 7b model requires two A100-80G GPUs, while the Mixtral 8 x 22b model needs four. We utilize the HQQ (Badri& & Shaji, 2024) tool save quantized weights and handle dequantization. To saving the binary weight, Mixtral 8 x model can be compressed to a memory from 10.82 to 16.65 GB. During dynamic inference, ODP reduces activation parameters by about 15%, with average accuracy decreasing by less than 1%. At 2.05-bit, the average activation parameter per token is only 3.23 GB, resulting in 1.80 increase in inference speed and evaluation accuracy of 62.68%. Tab. 4 also compares the performance of the LLaMA series dense models. The MC compressed 2.54-bit Mixtral x 7b model outperforms the 26.03 GB 16-bit LLaMA2- 13b model, with total parameter size of 16.65 GB and activation parameters of 3.69 GB. We have also extended the compression experiments to the Mixtral 22b model. MC shows higher overall performance compared mainstream dense models, without any training of original model. (more real speedup results are shown in Appendix A.9).
design a bit-change transformation (see Appendix A.2). After applying PMQ. the
5 CONCLUSION
MoE represents promising framework of sparse models for natural language understanding through scaling up the model capacity. However, the memory demands and redundancy among experts pose significant challenges for their practical implementation. In this work, we propose MC, mixture compression strategy based on the imbalance of significance among experts. This method co-designs the Pre-Loading TixeedPrrecision Quantization (PMQ) and Online Dynamic Pruning (ODP) approach, allowing MoE models to be compressed to an ultra-low bit-width while maintaining exceptional memory and parameter efficiency, as well as knowledgeable performance. And our preccipn strategy is orthogonal to various quantization techniques. Comprehensive experiments validate the effectiveness of our mixture compression, revealing that highly compressed MoE-LLMs can outperform equal-size full- full-precision dense LLMs, thereby improving the feasibility of MoE compression. Future work will focus on adapting this strategy for multimodal applications and optimizing it for specifie hardware platforms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work has been supported in part by Hong Kong Research Grant Council - Farly Career Scheme (Grant No. 27209621), ) General Research Fund Scheme (Grant No. 17202422, 17212923), Theme- based Research (Grant No. T45-701/22-R), the Innovation and Technology Fund (Mainland-Hong Kong Joint Funding Scheme, MHP/055/21), and the Shenzhen-Hong Kong-Macau Technology This research is also supported in part by National Key R&D Program of China (2022ZD01 National Natural Science Foundation of China (NO. 62461 160308, U23B2010), "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01161).
Research Program
REFERENCES
Hicham Badri and Appu Shaji. Towards 1-bit machine learning models, March 2024. URL https: //mobiusml github. io/ /ibitbittte Sooi_bboog/
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Tom B Brown. Language models are shotshot learners. arXiv preprint arXiv:2005. 14165, 2020.
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang. Kaiji Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 20224.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Mengzhao Chen, Wengi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-awan training for large language models. arXiv preprint arXiv:2407. 1062, 2024.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600-3466 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 20.101111110 14166, 2021.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseckmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.0600 06066, 2024.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers at scale. NeurIPS, 35:30318 010338883333222 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized represen- tation for lossllsss Ilm weight compression. arXiv preprint arxiv:2306.03077, 2023.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware mnighte quantization of neural networks. NeurIPS, 33:18518- 18529, 2020.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.0611 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
11

---

Published as a conference paper at ICLR 2025
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq:
Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732.. 2022.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5:288 00888-30,,00 2023.
Gao, J Tow, B Abbasi, S Biderman, Black, A DiPofi, C Foster, L Golding, J Hsu, A Le Noac'h, h. al. A framework for few-shot language model evaluation. URL https://zenodo. org/reconds/102568866 7, 2013.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311 22ti:2331..1023,,, 2023.
Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. arXiv preprint arXiv:2406. 12335, 2024.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03872 2021.
Haiyang Huang. Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jcan Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference. arXiv preprint WwXiv:2303.06113 2023.
Wei Huang, Yangdong Liu, Haotong Qin. Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for 1lms. arXiv preprint arXiv:2402.04291 2024a.
Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study. arXiv preprint arXiv:2404. 14047, 2024b.
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-llm: Salience-driven mixed- -recissin quantization for large language models. arXiv preprint arXiv:2405. 149117 2024c.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. NeurIPS. 14:21099-21111 2021.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04m00 2024.
Renren Jin, Jiangeun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. arXiv preprint arXiv:2402.16775. 2024.
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari. Yuxiong He, and Hany Hassan Awadalla. Scalable and efficient moe training for multitask multilingual models. arXiv preprint 1rXiv:2109.10466 2021.
Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. Memory efficient nllb-200: Language-specific expert pruning of a massively multilingual machine translation model. arXiv preprint 2xXiv::2/2.098/// 2022.
12

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Schoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. NeurIPS, 35:24101-24116 2022.
Pingzhi Li, Xiaolong Jin, Yu Cheng, and Tianlong Chen. Examining post-training quantization for mixture-of-experts: A benchmark. arXiv preprint arXiv:2406.08/55, 2024.
Baohao Liao and Christof Monz Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arXiv:2402.051147 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse nfxture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407. 00945, 2024.
Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Do emergent abilitics exist in quantized large language models: An empirical study. arXiv preprint wwxxv:2307008077 2023a.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprinl arXiv:2305. 2023b.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. arXiv preprint w0xv:2240.148800, 2024.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint Sxkiv:2409.02066 2024.
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXiv:2404.03605, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (1(14)): 1-67, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arxiv preprint arXiv:2310.000.. 2023.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-experts layer. arXiv preprint arXiv: 1701. 006588, 2017.
Mingüe Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11955, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun. Volodymyr Kuleshov, and Christopher De Sa. Quipt:: Even better Ilm guantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04 04396. 2024.
13

---

Published as a conference paper at ICLR 2025
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: quantization for large language models. International
Accurate and efficient Conference Machine Learning, pp. 38087-38099, PMLR, 2023.
Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang. Toward inference-optimal mixture-of-expert large language models. arXiv preprint arXiv:2404.02852. 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. NeurIPS, 36, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 2.t.::233....223, 2023.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.1422
2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey model compression for large language models. arXiv preprint 2xXiv:2308.07633 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MORE QUANTIZED RESULTS OF PMQ
This section expands on the comparative results of the Hessian and PMQ mixed precision metrics across different bit-width settings. Tab. 5 serves as an extension of Tab. 2, specifically providing detailed comparison of the evaluation results across eight zero-shot datasets using the Hessian metric employed by HAWQ V2 (Dong et al., 2020) in the 1.57 to 2.54-bit range. Within the target bit-width interval, PMQ outperforms Hessian in all ranges, achieving better bit-width allocation results by 0.3% to 8.6%. Notably, at the ultra-low bit-width of 1.57-bit, PMQ achieves a comprehensive score of 54. 49%, while Hessian reaches only 45.91%.
Table Performance of quantized Mixtral 7b on eight zero-shot benchmarks. "HellaS." " the short format of "HellaSwag" and "Wino." denotes "Winogrande".
Method
Bits
PIQA
ARC-e
ARC-c
BoolQ
HellaS.
Wino.
MathQA
MMLU
Avg. %
16.00
85,20
84.01
57.17
85.35
81.48
75.93
39.29
67.88
71.29
2.54
80.21
76.38
51.20
81.11
78.05
72.97
35.27
56.21
67.18
2.42
78.81
73.97
47.58
81.04
77.72
72.77
33.01
52.16
64.23
2.30
79.21
72.41
46.70
79.15
76.38
71.25
31.97
50.60
63.47
2.20
78.46
72.98
46.66
77.29
75.31
70.22
31.84
45.29
62.25
Hessian
2.05
75.32
67.26
45.01
70.29
71.90
69.11
31.07
40.85
58.85
1.94
75.41
64.02
43.19
67.75
69.18
68.27
28.58
36.99
56.67
1.81
71.96
60.81
37.72
68.27
63.29
65.46
26.27
32.58
53.30
1.69
69.88
60.37
35.64
70.06
59.60
58.43
26.05
32.11
51.39
1.57
65.26
52.12
21.84
68.21
52.91
50.32
24.99
31.58
45.91
2.34
80.52
77.100
51.28
82.54
79.03
73.995
39.18
56.37
67.30
2.42
80.36
75.76
50.17
80.00
78.13
73.09
34.97
53.22
65.71
2.30
83.11
73.59
47.78
80.83
76.48
73.14
33.84
$2.54
64.91
2.20
79.05
73.70
47.87
74.56
76.63
72.77
34.24
47.73
63.29
PMQ
2,05
79.16
73.06
48.38
80.58
74.95
71.27
31.79
46.80
63.25
1.94
76.88
68.48
45.48
75.23
72.05
72.61
31.16
40.93
60.35
1.81
76.93
66.67
43.60
75.50
70.50
69.85
28.68
40.71
59.06
1.69
75.41
64.14
40.61
68.96
67.01
68.03
28.04
37.14
56.17
1.57
72.42
62.46
37.88
73.55
63.17
66.38
26.80
32.25
54.49
Additionally, in Tab. 6, we extend the comparison Hessian and PMQ's few-shot performance across different bit widths presented in Tab. 3. At 1.69-bit, PMQ achieves score of 38.35% on the MMLU (five-shot) benchmark, while maintaining a model size that 16% smaller than the 2-bit model under uniform quantization, with an accuracy improvement of 4.5%. More importantly, observe that PMQ at 2.54-bit compresses the model size by 84% compared to the 16-bit model, yet the few-shot performance is only 9.4% lower, highlighting the substantial advantages of mixed compression for MoE models. In comparison to Hessian at the same bit-width, PMQ demonstrates great overall improved accuracy. BSP, on the other hand, exhibits poor performance the few-shot evaluations, which is even lower than 2-bit uniform quantization. In Tab. 7, we also compare these precision allocating metrics on WikiText2 dataset; PMQ shows clearer advantage, particularly at 1.577bit, where it achieves PPL of 8.50, representing significant improvement over the 2-bit uniform quantization, while the Hessian at 1.57-bit achieves only 14.20.
A.2 ONE-BIT WEIGHT SAVING AND DEQUANTIZATION
This paper presents MC, which explores static compression strategies and dynamic pruning method for MoE-LLMs in the ultra-low bit-width range, with selected static bit-width of 1-bit, 2-bit, and b-bit. We observe that both 2t and 3-bit can be addressed using conventional linear quantizers, method commonly utilized in most studies (Frantar et al., 2022; Shao et al., 2023; Huang et al., 2024c; Lin al., 2024). In contrast, the quantization of 1-bit weights involves totally different calculations; first provide the binarization formula for the weights:
B sign W)
15

---

Published as a conference paper at ICLR 2025
Table 6:
Comparison of different mixed- precision strategies on few-shot performance (MMLU sive-shot for Mixtral 7b.
Table 7:
Comparison of different mixed- precision strategies on PPL performance (Wiki- Text2 PPL 1) for Mixtral 8 x 7b.
Method
Bits
Accuracy %
Method
Bits
PPL J
16,00
70,60
16.00
3.84
Uni
2.00
34.0528.4%1
Uni
2.00
16.38
BSP
2.54
31.57an.9%
BSP
2.54
13.61
2.54
38.22.TP 2%63
2.34
5.4T
2.42
$4.099..9%
2.42
5.81
2.30
51.37 11.
2.30
5.86
2.20
47.01 4..
2.20
6.58
Hessian
2.05
4 9.99
Hessian
1.05
6.65
1,94
38.62 23.8%%
1.97
7.88
1.81
33.87 28.
1.81
8.45
1.69
29.4%1
1.69
10.18
1.57
31.96 30.4944%
1.57
14.20
2.54
61.191.00
2.54
5.09
2.42
58.30, 4.2%
2.42
5.25
2.30
55.08 7.49J
2.30
5.45
2.20
50.70 1.8%
2.20
5.72
PMQ
2.05
49.84 1
PMQ
2.05
5.91
1.94
45.98 66.5%
1.94
6.49
1.81
41.67 20.8%
1.81
6.81
1.69
38.35 24.0%6
1.69
7.78
1.57
33.44 29.0%)
1.57
8.50
sign(x) [1
if x 2
[-1
(8)
others.
where W E RRd is the full precision weight and B E (-1,+1) Due to the elements range of being #1, we can not directly save the one-bit value into compact memory. Hence, we propose a simple transformation for B:
+1)dxm denotes the binarized matrix.
sign W) +
B
2
where B {0, 1)dxm. In this case, we can really use 1-bit memory to storage each element. During the inference stage, we need dequantize the binary weight and operate the matrix multiplication of each input vector follows:
8 xB = x,
Σ xy), for i =
(10)
j:B, j:B,==0
where x E R1xd denotes one set of input vector (token), and represents the scaling factor of each binary matrix, which is calculated from [ilee] (Rastegari et al., 2016). In this binarized weight format, we can achieve computation without minimal multiplication operation. As shown in Eq. (10), the original computation requires dm multiplications and (d 1 1)m additions, resulting in MACs consumption of dm and computational complexity of O(m²). ). In contrast, binary matrix operations require only multiplications and (d 1 11m additions, leading a MACs consumption of just m and computational complexity of ((m).
dxm
A.3 RESULTS OF DIFFERENT QUANTIZATION TECHNIQUES
As detailed in Sec. 3.2 the main text, PMQ focuses primarily on leveraging the significance differences between experts to construct determining the optimal allocation, it can be combined with various quantization techniques. this study, to efficiently validate the effect of mixed compression, employ GPTQ (Frantar et al., 2022), an efficient training-fit free post-training quantization (PTQ) strategy, which completes mixed-precist quantization the Mixtral 8 x 7b model in just 90 minutes.
optimal mixed mixed-precision bit-width allocation. After
16

---

Published as a conference paper at ICLR 2025
Uniform 3-bit
PMO 2.54-bit
PMO
Figure 9: Needle in a Haystack evaluation. Green squares indicates high retrieval success rate, the Y-axis represents the distance to the retrieved target.
Table 8: Performance of quantized Mixtral 7b on eight zero-shot benchmarks on GPTQ (Frantar et al., 2022) and Omniquant (Shao et al., 2023). w denotes "with".
Method
Bits
PIQA
ARC-e
ARC-c BoolQ HellaS.
Wino.
MathQA
MMLU
Avg.%
16.00
85.20
84.01
57.17
85.35
81.48
75.93
39.29
67.88
71.29
PMQ
2.54
80.52
77.10
51.28
82.54
79.03
73.95
39.18
56.37
67.50
W GPTQ
2.05
79.16
73.06
48.38
80.58
74.95
71.27
31.79
46.80
63.25
(Frantar al., 2022)
1.57
72.42
62.46
37.88
73.55
63.17
66.38
26.80
32.25
54.49
PMQ
2.54
81.63
78.66
52.91
82.54
80.17
74.51
39.20
59.83
68.80
Omniquant
2.05
79.77
74.24
48.65
81.09
75.76
72.48
33.01
47.15
64.01
(Shao et al., 2023)
1.57
73.33
65.28
38.54
74.06
66.61
66.59
26.74
35.20
55.79
In this section, replace GPTQ with another advanced quantization method, Omniquant (Shao et al., 2023), which uses a learnable weight clipping (LWC) for quantization calibration. For calibration, 256 sequences from the C4 dataset selected for gradient optimization. Omniquant requires approximately 480 minutes to quantize the Mixtral 8 x 7b model (see Tab. 8), but outperforms GPTQ across eight zero-shot benchmarks, owing to its precise search for quantizer factors via LWC. This further demonstrates the flexibility of our PMQ framework.
A.4 QUANTIZATION RESULTS ON CHALLENGING BENCHMARKS
benchmarks on more challenging datasets in Tab. 9, considering the importance performance testing on more complex long text reasoning tasks (Cobbe et al., 2021; Bai et al., 2023; Chen et al., 2021). We have observed that in challenging tasks like GSM8K, HumanEval, and long-context model compression becomes more pronounced. This phenomenon holds true in other MoE LLM compression methods (Frantar et al., 2022; Huang et al., 2024c; Shao et al., 2023; Lu et al., 2024) well. However, our PMQ method, compared the latest method like BSP (Li al., 2024) and HAWQ (Dong et al., 2020) with Hessian-basel approaches for MoE LLM, is still able to maintain state-of-the-art performance. Fig. 9 shows the NIAH results in different sequences.
In this section, we expand our
performance drop of
Recent studies on quantization performance losses (Jin et al., 2024; Liu et al., 2023a) were also explored, revealing that ARC-C and GSMSK primarily involves inference issues, categorized as chain-of-thought (CoT), while MMLU can be classified as in-context learning (ICL). CoT tasks, due to their intricate reasoning demands, pose significant challenges to various LLM types. Given that many open-source MoE LLMs and dense LLMs do not exhibit strong inference capabilities during pre-training, we anticipate larger performance losses when reducing model bit-width ultra-low scenarios. The results in Tab. 2 and Tab. 9 also indicate that there is the huge potential for future exploration of MoE LLM compression on complex tasks.
A.5 DETAILED RESULTS ON BIT- WIDTH ALLOCATION
In this section, we further visualize the different bit-width allocation results of PMQ on Mixtral 8 x 7b model, shown in Fig. 10. The results clearly show that the importance of MoE expert varies with different position. can be seen that at lower bits-width, our algorithm only selects a small part of the position for protection, which greatly improves calculation efficiency. With the increasing of the bit-width, the important positions from lower bit-width are leavening unchanged which further proves the effectiveness of the proposed method.
17

---

Published as a conference paper at ICLR 2025
Table 9: Comparison of different mixed-precision quantization methods challenging benchmarks. NIAH denotes the task in Needle-in-a-haystad which is a more challenging task for evaluating long-context ability.
Method
Bits
GSM8K1
HumanEval (pass @ 10)f
NIAHT
16.00
58.30
59.15
100.00
Uniform
3.00
38.13
29.88
98.48
Uniform
2.00
0.00
0.00
0.00
BSP
2.54
4.25
3.21
42.21
Hessian
2.54
33.59
25.49
100.00
Hessian
2.05
17.24
7.84
93.45
PMQ
2.54
37.67
29.34
100.00
PMQ+ODP
2.54
35.25
27.58
100.00
PMQ
2.05
19.97
11.83
100,00
PMQ+ODP
2.05
18.04
10.02
99.26
A.6 ABLATION ANALYSIS ON SYPEE-PARAMETEER OF EXPERT SIGNIFICANCE WEIGHT
In this section, we conduct experiments based on different hyperparameter settings for the expert significance factor weights, ie.. and 3 in Eq. 4. We evaluate these factors with values of 1, 1.5, and differentiate their relative significance on Mixtral 8 x 7B (2 bit). Since quantization eтror is a critical evaluation metric, we fix its weight 7 at 2 and vary the weights of the expert significance factors accordingly. The experimental results, shown Tab. 10, indicate that the overall accuracy remains stable, but exhibits slight decline when the combined value of a and 3 exceeds the quantization error weight.
Table 10: Ablation analysis Mixtral 7x8B model, evaluating different settings for the weights of the two significance factors, a and 33 (Eq. 4), with the quantization error weight fixed at 2, using the WikiText2 dataset.
1
1.5
3
1
1.5
2
1
1.5
2
1.5
PPI.
5.92
5.92
5.91
5.92
5.91
5.96
5.91
5.96
5.95
A.7 COMPARISON OF DIFFERENT TOKEN-DEPENDENT PRUNING METRIC
Regarding the dynamic pruming of experts, we note that most existing pruning methods for LLMs or other neural networks focus static weight pruning (Sun et al., 2023; Zhang et al., 2023), and cannot dynamically prune experts during inference based on tokens. Dynamic pruning during inference remains under-explored, with only one recent post-training MoE LLM dynamic pruning work (Lu et al., 2024) proposing a gating-sccre-baie compared with Wanda (Sun et al., 2023) method, highly effective static pruning method, and concluded that static pruning methods result in significant performance degradation when applied to dynamic MoE LLM experts. We incorporated additional metrics for dynamic expert pruning to expand the scope of our experiments. Specifically, we perform token-dependent expert pruning on token kurtosis, token var, and token mean, where 30% of tokens will be pruned from top-2 to top-1 (Tab. 11).
strategy for dynamic pruning. This work has already
A.8 ABLATION OF DYNAMIC EXPERT PRUNING THRESHOLD
We follow the setting from recent dynamic MoE pruming work (Lu et al., 2024), selecting it as the median value of #i which also theoretically and empirically demonstrates that this choice of we.
18

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of different token-dependent dynamic expert pruning strategies on Mixtral
8 x 7b. Avg.CH CP denotes the average compressed parameters ratio for each token. NIAH denotes the
task in beedle-in-a-haysttckk which more challenging task for evaluating long-context ability.
Method
/(w)/wo)
Avg.CP
WikiTex124 LM-Eval% GSMSK1
HumanEVall 10)1
NIAHf
Token kurtosis
0.3
15.62%
7.16
57.22
14.05
6.54
93.16
Token variance
0.3
15.62%
6.69
60.02
17.33
7.92
95.37
Token mean
0.3
15.62%
6.82
59.27
17.76
6.02
95.65
ODP
14.88%
6.22
63.25
18.04
10.02
99.26
threshold is comprehensive optimal setting. In this section, provide more comprehensive ablation on threshold ji in Eq. As demonstrated in Table 12, utilizing a manual threshold of 0.4, the PPL performance stands at 6.29 with mere 12.00% of experts pruned. In contrast, our proposed method, referred to as ODP, achieves a PPL. of 6.22 and prunes 14.88% of the experts. This not only showcases superior accuracy but also highlights enhanced efficiency.
Table 12: Ablation of different threshold hyperparameter.
((w//wo)
PPL (WikiText2))
Avg. Pruning Params.
0.4
6.29
12.00%
0.5
6.49
16.51 %
0.6
6.64
19.25 %
0.7
6.89
22.43%
Median
6.48
15.18%
ODP(Median Protection)
6.22
14.88%
A.9 COMPUTATION ANALYSIS OF ONLINE DYNAMIC PRUNING
During the ODP phase, compared to the significant reduction in the number of tokens and experts leading to large-scale matrix multiplications, the computational cost of token importance calculation can be negligible. Specifically, in Mixtral 8 x 7b, where the typical input token matrix size is R the token importance calculation involves three steps: summing attention weights, computing the E norm, and performing lop-k calculations. The overall floating-poin -pont operations per second (FLOPs) calculation amounts to n² + n + mm + alogn. the ODP inference phase, after dynamic pruning, an average of 15% of tokens in MoE layer will reduce experts inference (see Tab. 12). The FLOPs for these 15% tokens within an expert (an expert with 3 linear layers, the size is Rm Rm mm xm) are O. 1.15% (m my x m2 2 m. larger than n and m in Mixtral 8 x 7b. Therefore, the computational cost of importance calculation is usually low. As demonstrated in Tab. 4, when PMQ is combined with ODP, it further enhances computational efficiency. This indicates that the efficiency gain from experts' dynamic pruning outweighs the computational cost of token importance calculation.
2), where m, is typically much
Table 13: End-to-end latency comparison between FP16 and MC on Mixtral 8 x 7b under different [batch, input token length]. Each cell is the latency for one token generation speed (second).
Hardware [1.512]
[1,1024]
[1,2048]
[1.4096]
[8.2048]
[8.4096]
[16,2048]
[16.4096]
FP16 MC 2.54-bit Speedup(S))
AxA100
0.029
0.038
0.043
0.057
0.009
0.011
0.007
0.010
1xA100
0.015
0.018
0.019
0.025
0.004
0.005
0.004
0.004
48.3
52.7
56.2
56.1
55.4
54.3
47.6
60.1
In Tab. 13 and Tab. 14, we present the actual speed enhancements of deployment achieved by our MC method various hardware platforms. The speed enhancements in MC, as detailed in Tab. 13, originate from static compression during the PMQ phase and adaptations of the CUDA kernel (based on HQQ) along with ODP. Across varying batch sizes and input sequence lengths, speedup
19

---

Published as a conference paper at ICLR 2025
Table 14: Latency comparison of MoE and dense LLM under different hardware platform.
Model
Hardware
Loading Memory
Peak GPU Memory
LM-Eval9
Token/s
Mixtral 8b
A100
96.8 GB
112.6 GB
71.29
23
Mixtral 8x
1x 1x3090
0OM
OOM
LLaMA2-13h
AxA100
26.0 GB
33.4 GB
65.19
46
LLaMA2-13b
xx3090
0OM
00M
Mixtral 8x
1xA100
16.2 GB
20.7 GB
66.94
38
MC 2.54-bit
Mixtral 8x7b
1x3090
16.2
20.7 GB
66.90
52
MC 2.54-bit
ranges from 40% to 60%. The performance boost from model weight compression remains consistent regardless of input sequence length and batch size. However, with fixed batch size, we notice more pronounced speed advantage for our MC-MoE the sequence length increases, attributed to the increased efficiency demonstrated by ODP. As batch size increases, both the FP16 models and compressed models experience an overall increase in throughput, leading to accelerated average token generation speeds. In Tab. 14, with MC-MoE the RTX 3090 GPU, extreme compression allows for an average speed of 52 token/s, which is very cost-effective. this scenario, compressed MoE LLM outperforms dense LLM in memory, accuracy, and speed.
20

---

Published as a conference paper at ICLR 2025
Model Aug BitWidthh 1.57
Total Experts Bidth: 12
Model Bi-Widthh: 1.69
Total Experts Bit-Width: 13
Experts BitWidth:
Model Avg BitWidth:: 1:94
lotal Experts Wibwidth: 15
Model Avg Bi: Bit:Wihh:: 2.05
lotal Experts WidtWidth: 16
Model Width:
Total Experts Width:
Model Avg BR-Witth: 2.3
Total Experts Bit-Width:
Model Aug BitWidth: 2.42
Total Experts Bit-Width: 19
Model Bi-Width: 2.54
Total Experts Bit-Width: 20
Figure 10: Visualization different bit-width allocation. Color refers the bit size.
21


Reviews:
[{"review_id": "TWuMxkr0od", "reviewer": "Reviewer_vMGD", "summary": "The authors proposed two designs for efficient MoE inference: PMQ and ODP. PMQ conducts mix-precision quantization of experts, and ODP essentially prunes experts depending on certain critical tokens.", "questions": "-  I don't seem to find much connection between the PMQ and ODP part of the submitted work. Are they standalone to each other?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- MoE efficiency is a relatively under explored area in comparison to general, dense LLMs. This work is a welcomed addition.\n- Two proposed designs are able to deliver decent task performance, especially for PMQ over BSP in Table 2.", "weaknesses": "- The novelty of the proposed work is limited, as both mixed-precision quantization and token-dependent expert pruning are well-explored avenues for efficient MoE inference.\n- Potential lack of baseline: BSP is the only truly relevant comparison to PMQ due to its mixed-precision approach. No pruning comparisons are provided for ODP.\n- Most datasets used in Table 2 are common-sense intelligence tasks. Extensive literature across various fields has shown that such tasks (and ppl) are relatively robust and can achieve substantial compression gains. I would like to see more challenging tasks evaluated, such as GSM8k, HumanEval and LongBench.\n- The efficiency evaluation seems somewhat rough. The registered speedup does not correspond to the featured task. I would like to see comprehensive reports on latency, throughput (across different tasks, compression rates, and batch sizes), and memory.\n\nI also have a few formatting suggestions:\n- Please consider adding highlights in Figure 3, as you have done in Figure 4. It is difficult to identify the discussion substance by reading coordinates on a small diagram.\n- The LM-Eval column in Table 4 is not defined, though I understand it refers to the tasks in Table 2.\n- Not that it matters much, but the authors may want to know that the proper way to use left and right quotation marks in LaTeX is `` and '', respectively. Currently, only right quotation marks are being used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wdU47qiLkR", "reviewer": "Reviewer_knJE", "summary": "This paper proposes MC-MoE, a training-free Mixture-Compressor for MoE-LLMs that applies the significance of experts and tokens to perform deep compression, incentivized by optimization space to improve upon memory consumption of expert heads' parameters and redundancies of activated heads. To alleviate saving and loading overheads, the authors devised the Pre-Loading Mixed-Precision Quantization stage for adaptive memory allocation. Following up with Online Dynamic Pruning (ODP) stage, this method dynamically selects significant tokens to elevate inference efficiency while maintaining model performance, achieving extreme compression.", "questions": "While the performance scores of PMQ across benchmarks were mostly robust, those of ARC-c and MMLU were relatively meager. Would the authors like to provide an in-depth analysis of these two tasks, regarding potential challenges PMQ had on particular patterns/features?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Pre-loading is an intuitive yet effective method to cope with overheads in loading expert parameters.\n2. This method leveraged the uneven features learned by different expert heads as guidance to optimize quantization effort with integer programming while providing valid expert significance analysis to defend the assumption.\n3. This method introduced token relevance from the attention heat map to the criterion of parameter pruning, offering salient pruning instructions without utilizing external clues.", "weaknesses": "While this research adopts weight-only pruning, we encourage the authors to compare the effectiveness of other popular pruning methods in the second stage to demonstrate the weight-only pruning is sufficient and effective among all methods selected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5av97eqelY", "reviewer": "Reviewer_Z53p", "summary": "This paper introduces MC-MoE, a novel compression framework for Mixture-of-Experts (MoE) Large Language Models that combines static mixed-precision quantization with dynamic pruning. The approach consists of two main components: Pre-Loading Mixed-Precision Quantization (PMQ), which allocates different bit-widths to experts based on their importance, and Online Dynamic Pruning (ODP), which identifies and protects critical tokens while dynamically pruning less important experts during inference. Through extensive experiments, the authors demonstrate that their method can greatly compress the model without too much accuracy loss.", "questions": "1. The authors do not provide sufficient analysis of the robustness of their compression method under different input sequence lengths, which is crucial for practical deployment scenarios.\n2. The generalizability of the proposed methods to other MoE architectures with different routing mechanisms is not discussed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed PMQ method innovatively considers multiple factors (activation reconstruction error, routing scores, and frequencies) in determining bit-width allocation.\n2. The paper presents a comprehensive solution that addresses both static model compression and dynamic inference optimization.\n3. The authors provide extensive empirical validation across multiple benchmarks and model sizes, demonstrating the method’s robustness and scalability.", "weaknesses": "1. The paper lacks ablation studies on the impact of different hyperparameters (μ threshold, protection ratio) on model performance.\n2. The paper does not adequately address the potential compounding effects of quantization errors across multiple MoE layers, particularly in deeper networks where error propagation could be more significant.\n3. The paper lacks a comprehensive error analysis to identify which types of tasks or linguistic phenomena are most affected by the compression techniques.\n4. The computational overhead of the token importance calculation in ODP is not thoroughly analyzed, which could be significant for real-time applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSodyjRvxD", "reviewer": "Reviewer_uP5F", "summary": "The paper presents MC-MoE, a training-free Mixture-Compressor for Mixture-of-Experts LLMs, which is designed to address memory consumption and redundancy challenges. MC-MoE combines Pre-Loading Mixed-Precision Quantization and Online Dynamic Pruning to compress MoE-LLMs. Wxperiments show that MC-MoE maintains high performance.", "questions": "See Weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Conducts lots of experiments to verify the accuracy drop of the proposed method", "weaknesses": "My major conerns are the motivation and practical benefit of the proposed method:\n\n1. What is the \"metric\" for measuring the speedup in Table 4? Is it latency or throughput? What is the input length and output length in your benchmark setting?\n\n2. What is the target platform of the proposed method? Cloud or edge? The main advantage of SMoE models over dense models is that they scale better in term of computation [1]. Under the same \"act.\" param. size, SMoE is better than the dense counterpart (but non-act. experts still take GPU HBM to store). Thus it is very important to carefully design the distributed inference infrastructure to translate the theoretical benefits of SMoE to real system benefts. For example in [1], it discussed the expert load balance, pipeline parallelism, and large batch size to keep GPU busy. I did not see the discussion of the side effects of mixed precision, since it will worse the load balance, especially when the number of experts is large. If your targeted use case is on the edge, what benefits does an SMoE model offer over a dense model?\n\n3. In practical applications, it's extremely challenging to keep GPUs fully utilized. The techniques discussed in this paper—weight-only quantization and dynamic pruning—introduce irregular computation patterns. These irregularities lead to load imbalances across GPU cores, making it difficult to maintain GPU utility. Moreover, these methods tend to perform worse with larger batch sizes. However, using large batch sizes is essential in practice to achieve optimal performance with SMoE models [1]. Therefore, while these techniques aim to improve efficiency, they may actually hinder performance due to the introduced irregularities and their incompatibility with large batch sizes required by SMoE.\n\n\n[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)