Title: Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mixture-of-Experts

Expert Pruning
Abstract: Mixture-of-Experts (MoE) models achieve a favorable trade-off between performance and inference efficiency by activating only a subset of experts. However, the memory overhead of storing all experts remains a major limitation, especially in large-scale MoE models such as DeepSeek-R1 (671B). 
In this study, we investigate domain specialization and expert redundancy in large-scale MoE models and uncover a consistent behavior we term~\emph{few-shot expert localization}, with only a few in-domain demonstrations, the model consistently activates a sparse and stable subset of experts on tasks within the same domain.
Building on this observation, we propose a simple yet effective pruning framework, \textbf{EASY-EP}, that leverages a few domain-specific demonstrations to identify and retain only the most relevant experts.
EASY-EP comprises two key components: \textbf{output-aware expert importance assessment} and \textbf{expert-level token contribution estimation}. The former evaluates the importance of each expert for the current token by considering the gating scores and L2 norm of the outputs of activated experts, while the latter assesses the contribution of tokens based on representation similarities before and after routed experts.
Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99\times$ throughput under the same memory budget as the full model, with only half the experts. Our code is available at https://github.com/RUCAIBox/EASYEP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EASY-EP, a training-free pruning framework that (i) scores each expert by the product of its gating value and output norm and (ii) weights scores by token-level representation change, then retains only the top-ranked experts per layer. Applied to DeepSeek-R1 and DeepSeek-V3-0324, pruning 50 % of experts maintains ≥90 % of full-model accuracy across math, coding, science, finance, medicine, and agent tasks, while boosting 1k-token throughput by 2.99× and allowing single-node deployment. EASY-EP needs just one forward pass, no retraining, and also supports mixed-domain pruning with balanced performance, enabling memory- and latency-efficient use of gigantic MoE models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The empirical study spans two large-scale SOTA LLMs with MoE architecture and six downstream domains. The comprehensive ablations verify the necessity of both scoring factors that the paper proposed.\n\n2. The method is straightforward: one forward pass suffices, avoiding the combinatorial evaluations required by perturbation-based baselines.\n\n3. This paper provides a lot of helpful insights, such as MoE routers naturally localize to a small, domain-consistent expert set with only a handful of in-domain examples. It further shows that a single-node deployment with minimal accuracy loss is practically valuable. \n\n**Weakness:**\n\n1.\tThe proposed method is largely incremental and offers limited structural novelty; its computations rely primarily on similarity or norm-based measures.\n\n2.\tTo enhance interpretability, the authors could analyze expert capacities and token-routing distributions across different domains, comparing these patterns between the baseline and the proposed approach.\n\n3.\tThe study’s statistical reliability is weak: error bars and significance tests are absent, leaving uncertainty about the robustness of the reported improvements.\n\n4. It would be better to validate the performance of the method on at least one additional MoE family other than DeepSeek, e.g. Qwen or Mixtral. This would be helpful to show that the method can generalize beyond DeepSeek’s router design\n\nQUESTIONS:\n1. The paper acknowledges a larger accuracy drop when pruning with a single domain and testing on a different one. Could you add qualitative or quantitative analyses (e.g., expert-activation overlap heat-maps) to pinpoint why EASY-EP underperforms here and outline mitigation strategies such as hybrid expert pools or dynamic gating?\n\n2. In line 121, could the authors properly define \"demonstrations\"?\n\nLIMITATIONS:\nNo discussion on limitations. The authors could consider discussing whether the insights of this work can be extended to much boarder domains/modalities/MoE architectures.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the challenges associated with the memory overhead of large Mixture-of-Experts (MoE) models, particularly focusing on the DeepSeek-R1 model, which has 671 billion parameters. The authors introduce a novel pruning framework called EASY-EP, which leverages a few domain-specific demonstrations to effectively identify and retain the most relevant experts while discarding less important ones. \n\nThe key contributions of the paper include:\n\n1. Observation of Few-Shot Expert Localization: The authors identify a consistent behavior where, given a small number of in-domain examples, the MoE model activates a stable and sparse subset of experts that are highly relevant to the task at hand. This phenomenon is termed \"few-shot expert localization.\"\n\n2. Development of EASY-EP Framework: The proposed framework consists of two main components: output-aware expert importance assessment and expert-level token contribution estimation. This allows for efficient pruning of experts based on their relevance to the current task, significantly reducing memory requirements without sacrificing performance.\n\n3. Empirical Validation: The authors conduct extensive experiments on the DeepSeek-R1 and DeepSeek-V3-0324 models across various benchmark datasets. They demonstrate that their pruning method can maintain comparable performance while achieving a 2.99 times increase in throughput under the same memory constraints by retaining only half of the experts.\n\n4. Domain Specialization Insights: The paper provides insights into the domain specialization of experts within large MoE models, showing that experts are often highly specialized and can be reliably identified with minimal demonstrations. This finding underscores the potential for domain-specific pruning strategies in large-scale models.\n\nOverall, the paper presents a significant advancement in the efficient deployment of large MoE models, addressing both memory constraints and performance optimization through innovative pruning techniques.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper presents a well-structured analysis of expert activations in large Mixture of Experts (MoE) models, particularly focusing on the DeepSeek-R1 model. The methodology is robust, employing a novel pruning framework (EASY-EP) that effectively leverages few-shot demonstrations to identify and retain relevant experts. The experimental results demonstrate that the proposed method achieves comparable performance while significantly increasing throughput, indicating a high-quality contribution to the field.\n2. The document is clearly organized, with a logical flow from the introduction to the methodology and results. Key concepts such as few-shot expert localization and the components of the EASY-EP framework are well-defined, making it accessible to readers with varying levels of expertise in the subject matter. The use of figures and tables to illustrate findings enhances understanding.\n3. The research addresses a critical challenge in deploying large-scale MoE models—memory efficiency. By demonstrating that domain-specific experts can be effectively identified with minimal demonstrations, the paper contributes valuable insights that could lead to more efficient model deployment in real-world applications. The findings have implications for various domains, including math, science, and coding, which adds to the paper's significance.\n4. The concept of few-shot expert localization is a novel contribution to the field of machine learning and MoE architectures. The integration of output-aware expert importance assessment and expert-level token contribution estimation into the pruning process showcases innovative thinking and a fresh approach to expert pruning strategies.\n\nWeaknesses:\n\n1.While the experimental results are promising, the paper lacks detailed statistical significance reporting, such as error bars or confidence intervals, which could strengthen the validity of the findings. This omission may raise concerns about the robustness of the results.\n2.  Although the paper is generally well-written, some sections may benefit from additional elaboration, particularly in the explanation of the underlying mechanisms of the proposed pruning method. More detailed descriptions of the experimental setup and parameters could enhance reproducibility.\n3. While the findings are significant, the paper could further explore the long-term implications of pruning on model performance, particularly regarding generation capabilities. A discussion on potential trade-offs between efficiency and performance in different contexts would add depth to the analysis.\n4.  Although the paper introduces a novel framework, it could benefit from a more comprehensive comparison with existing expert pruning methods. Highlighting the unique advantages of EASY-EP over other approaches would strengthen the argument for its originality and effectiveness.\n\nIn summary, the paper presents a valuable contribution to the field of MoE models, with strengths in quality, clarity, significance, and originality. However, addressing the identified weaknesses could enhance its overall impact and acceptance in the research community.\n\nQUESTIONS:\n1. Statistical Significance Reporting: The paper does not provide error bars or any information regarding the statistical significance of the experimental results. This is a critical aspect of scientific reporting, as it helps to understand the reliability and variability of the results. The authors should include this information to strengthen their claims and provide a clearer picture of the model's performance.\n2. Reproducibility of Results: While the paper claims to provide sufficient information for reproducing the results, it would benefit from a more detailed description of the experimental setup, including specific hyperparameters, training procedures, and the exact datasets used. This transparency is essential for other researchers to validate the findings and build upon the work.\n3. Impact of Expert Pruning on Long-term Generation Abilities: The paper mentions that while domain capabilities may be preserved in pruned models, there are concerns about the long-term generation abilities being compromised. The authors should provide more empirical evidence or analysis to support this claim. It would be helpful to include experiments that specifically assess the long-term performance of the pruned models compared to the full models.\n4. Generalization Across Domains: The concept of few-shot expert localization is intriguing, but the authors should elaborate on how well the identified experts generalize across different domains. It would be beneficial to include additional experiments that test the model's performance on unseen datasets within the same domain to validate the robustness of the few-shot approach.\n5. Clarity and Structure of the Paper: The paper could improve its clarity by ensuring that the key contributions and findings are clearly articulated in the abstract and introduction. A more structured presentation of the results, including visual aids or tables summarizing key findings, would enhance the reader's understanding and engagement with the material. \n\nThese suggestions aim to enhance the rigor and clarity of the paper, ultimately contributing to its impact in the field.\n\nLIMITATIONS:\nThe paper presents a thorough investigation into the domain specialization of experts in large Mixture-of-Experts (MoE) models, particularly focusing on the DeepSeek-R1 model. However, there are several limitations that should be addressed:\n\n1. Generalizability of Findings: While the study demonstrates the effectiveness of the proposed EASY-EP pruning method across various domains, it primarily relies on specific datasets (e.g., AIME-2023, GPQA-main, LiveCodeBench-V3). The authors should discuss how well these findings generalize to other domains or tasks not covered in their experiments.\n2. Long-term Performance: The paper notes that while domain capabilities may be preserved in pruned models, there are concerns about the long-term generation abilities being compromised. This aspect requires further exploration, as it could significantly impact the practical deployment of the model in real-world applications.\n3. Statistical Significance: The paper does not report error bars or other statistical significance measures for the experimental results. Including this information would strengthen the validity of the claims made regarding the performance of the proposed method.\n4. Potential Negative Social Impacts: The authors state that there is no societal impact of the work performed. However, it is essential to consider potential negative implications, such as the misuse of the technology or biases that may arise from the model's training data. A more in-depth discussion on these aspects would be beneficial.\n5. Clarity and Detail in Methodology: While the methodology is described, there could be more clarity regarding the specific steps taken during the pruning process and how the calibration sets were selected. This would enhance reproducibility and understanding of the method.\n\nIn summary, while the paper makes significant contributions to the field, addressing these limitations and suggestions would enhance its robustness and applicability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates domain specialization in large Mixture-of-Experts (MoE) models. The authors observe that with just a few domain-specific examples, MoE models consistently activate a sparse and stable subset of experts (\"few-shot expert localization\").\n\nBased on this observation, they develop a pruning framework, EASY-EP, that combines output-aware expert importance assessment with expert-level token contribution estimation to identify and retain only the most relevant experts for specific domains.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The discovery of \"few-shot expert localization\" is compelling and well-documented. The systematic analysis showing that only 5-25 demonstrations are sufficient to identify domain-specific experts (Figure 2C) is a valuable empirical finding that advances our understanding of MoE model behavior.\n\n2. EASY-EP is well-designed with two complementary components. The method is computationally cheap, requiring only a single forward pass to calculate scores, making it practical even for models with hundreds of experts. \n\n3. The paper is well-written\n\n## Weaknesses\n1. All experiments use DeepSeek variants. It's unclear whether the findings generalize to other MoE architectures (e.g., Moonlight-MoE, OLMoE, Mistral). This significantly limits the generalizability claims.\n\n2. The comparison is mainly against simple baselines (random, frequency, gating scores). More sophisticated expert/model pruning methods from recent literature(e.g., MC-SMoE, Angular) could provide stronger baselines.\n\nQUESTIONS:\n1. The token contribution metric, $s_{t}^{l} = 1 - Sim(h_{t}^{l}, \\tilde{h}_{t}^{l})$ , measures the representational change induced by the entire MoE module for a given token. Have the authors investigated how this change correlates with the properties of the activated experts? For example, does a larger change (lower similarity) correlate with higher variance in the gating scores for that token?\n\n2. How does the method perform on other MoE architectures beyond DeepSeek models?\n\n3. Can the method be extended to dynamic expert selection during inference?\n\nLIMITATIONS:\nThe paper could benefit more by including a discussion of tasks that span multiple domains.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates domain specific pruning of the experts in large SOTA MoE models (e.g., DeepSeek-R1 (671B)). The paper confirms that for such models, different expert specializes to different domain. Based on this confirmation, the paper designs a method for identifying domain-irrelevant experts for pruning them in order to reduce inference cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe empirical analysis provided in this paper to demonstrate the domain-specific specialization of experts in large-scale MoE models with many experts validates (probably for the first time in this paper) the hypothesis, and useful to MoE research community\n2.\tThe proposed method of combining routing statistics with some expert’s output dependent metrics (output L2 norm and cosine similarity) outperforms some baselines provided in this paper\n3.\tThe paper is well-written and easy to follow\n\nWeakness:\n\n1.\tAlthough, the specific combination of metrics used in this paper for evaluating the experts’ importance is novel, most of these metrics are used before in literature.\n2.\tIt’s not clear why the specific combination of metrics works better than the baselines provided in this paper, i.e., no theoretical justification has been provided. Specifically, for the case of using expert’s output L2 norm, equation (6) shows that larger the L2 norm of the output of an expert, larger the contribution of the expert in the overall L2 norm of the final output. But, why the contribution to final output’s L2 norm is a good measure?\n3.\tThe comparison with existing baselines is inadequate. Although it has been mentioned that evaluating some of these baselines are computationally expensive (e.g., [1], [2]), still a comparison should be provided with those baselines to check that whether the proposed method performs worse than these methods. Specifically, those baselines should be included which doesn’t share any metric with the proposed combination of metrics (e.g., router-norm based method in [3]).\n\n[1] Lu, Xudong, et al. \"Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models.\" Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.\n\n[2] Cao, Mingyu, et al. \"Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning.\" arXiv preprint arXiv:2412.00069 (2024).\n\n[3] Chowdhury, Mohammed Nowaz Rabbani, et al. \"A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts.\" Forty-first International Conference on Machine Learning.\n\nQUESTIONS:\n1. How do you use the tokens of the pruned experts? Do you re-route them? Or, do you avoid computing the tokens?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}