MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts

Authors: Peng Jin, Bo Zhu, Li Yuan, Shuicheng YAN
Primary area: foundation or frontier models, including LLMs
Keywords: mixture of experts, large language models, efficient foundation models

Abstract:
In this work, we aim to simultaneously enhance the effectiveness and efficiency of Mixture-of-Experts (MoE) methods. To achieve this, we propose MoE++, a general and heterogeneous MoE framework that integrates both Feed-Forward Network (FFN) and zero-computation experts. Specifically, we introduce three types of zero-computation experts: the zero expert, copy expert, and constant expert, which correspond to discard, skip, and replace operations, respectively. This design offers three key advantages: (i) **Low Computing Overhead**: Unlike the uniform mixing mechanism for all tokens within vanilla MoE, MoE++ allows each token to engage with a dynamic number of FFNs, be adjusted by constant vectors, or even skip the MoE layer entirely. (ii) **High Performance**: By enabling simple tokens to utilize fewer FFN experts, MoE++ allows more experts to focus on challenging tokens, thereby unlocking greater performance potential than vanilla MoE. (iii) **Deployment Friendly**: Given that zero-computation experts have negligible parameters, we can deploy all zero-computation experts on each GPU, eliminating the significant communication overhead and expert load imbalance associated with FFN experts distributed across different GPUs. Moreover, we leverage gating residuals, enabling each token to consider the pathway taken in the previous layer when selecting the appropriate experts. Extensive experimental results demonstrate that MoE++ achieves better performance while delivering 1.1$\sim$2.1$\times$ expert forward throughput compared to a vanilla MoE model of the same size, which lays a solid foundation for developing advanced and efficient MoE-related models.

Decision: accept (oral)
Reviewer scores: 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730343399035

thread_type:
official_review

<thread_object>
creator: reviewer_lu5b

<content>
summary:
This paper introduces MoE++, a general and heterogeneous MoE framework that uses a fixed mixing mechanism for all tokens and optimizes computation allocation by assigning fewer FFN experts to simple tokens, allowing more FFN experts to be dedicated to challenging tokens. Extensive experimental results demonstrate the lower computational overhead and better performance of MoE++ than vanilla MoE.

strengths:
- This work improves the existing MoE framework in terms of both efficiency (throughput) and effectiveness (performance), making it impactful for real-world applications.
- This paper is well-written and insightful, with clear motivation and illustrations.
- Figures and tables are clear and easy to read.

weaknesses:
- While Table 1 shows the “complexity between the proposed MoE++ and MoE” and zero-computation experts enjoy a complexity of 0, they still likely lead to some extra computation overhead. Therefore, this work lacks real-world wall-clock time demonstrations of these zero-computation expert operators, especially regarding a batch of tokens.
- This work introduces three types of zero-computation experts (i.e. zero, copy, and constant) but provides minimal justification for this specific set. From Table 5, we can see they only conducted basic combinatorial ablations.

questions:
- How are these zero-computation experts specialized to input tokens? Specifically, it would be great to show some examples of tokens allocated to zero, copy, and constant experts, respectively.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685341815

thread_type:
official_review

<thread_object>
creator: reviewer_fdic

<content>
summary:
This paper proposes the addition of a new class of experts in the MoE architecture called "zero-computation" experts. The zero-computation type experts include a zero expert, a copy expert, and constant experts. Each of these experts is extremely computationally lightweight relative to a standard FFN expert. The authors demonstrate that incorporating this expert type is both GPU friendly due to the low overhead and can improve performance and throughput.

strengths:
The authors provide a fairly simple and effective addition to the standard MoE architecture which effectively allows more heterogeneous computing in different layers with minimal overhead. The paper is written clearly and a thorough set of ablations are performed.

weaknesses:
The empirical gains don't seem to be too significant and can only start to be seen at large scale ~7B parameters.

questions:
Are the baseline MoE models also trained with residual gating? For each model, many experts are activated in each layer? I could not really tell.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702657637

thread_type:
official_review

<thread_object>
creator: reviewer_sd7n

<content>
summary:
This paper introduces the MoE++, which adds zero-computation experts to enhance the efficiency of computation.
It utilizes zero-computation experts to minimize overhead by allocating fewer resources to basic tokens and concentrating on complex ones. MoE++ enhances expert selection stability by employing pathway-aware routing with gating residuals, resulting in higher performance and throughput than conventional MoE models at a reduced computational cost.

strengths:
With the proper hyper-parameters, MoE++ introduces zero-computation experts who can reduce the computational load by bypassing or simplifying processing for certain tokens, leading to efficient resource use.
These heterogeneous experts with suitable routing designs can improve the model's efficiency and effectiveness.

weaknesses:
1. Parameters such as $\tau$, which regulate token allocation between zero-computation experts and original, may complicate model tuning, as the performance and burden distribution are sensitive to them.  
2. The pathway-aware routing with gating residuals adds complexity to the expert selection process, which may require careful tuning for optimal results.
3. The dynamic routing mechanism can still result in load imbalances, especially under diverse data distributions, which may lead to underutilized or overloaded experts that affect efficiency.

questions:
1. How do individual components, like zero experts or gating residuals, contribute to MoE++'s overall performance?
2. Could certain components be removed or modified for specific use cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732110119225

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all PCs, SACs, ACs, and Reviewers for their time and efforts when handling our paper.

All reviewers appreciate the contributions of our method:
* All reviewers pointed out that our work improves the existing MoE framework in terms of **both efficiency (throughput) and effectiveness (performance)**.
* Reviewers FDic and LU5B all highlighted that our paper is **written clearly** and **extensive experiments are performed**.
* Reviewer LU5B commented that our method is **impactful for real-world applications**.

As suggested by the reviewers, we have revised the manuscript as follows:
* The real-world wall-clock time for zero-computation experts and FFN experts (Table A on Page 15).
* Comparison of computational costs between pathway-aware router and original router (Table B on Page 15).
* Effects of removing certain components (Table C on Page 16).
* The visualization of the number of zero-computation experts (zero experts, copy experts, and constant experts) activated per token at the token level (Figure A on Page 20).
* Modifying the caption of Table 2 on Page 6.

We highlight all the modifications using red color.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes MoE++, a heterogeneous MoE framework that enhances both the efficiency and effectiveness of MoE methods by integrating FFN and zero-computation experts (zero, copy, and constant experts). This design reduces computing overhead by dynamically assigning simpler tokens to zero-computation experts, improves performance by focusing FFNs on challenging tokens, and eliminates GPU communication overhead, making it deployment-friendly. Experimental results show that MoE++ outperforms vanilla MoE models of the same size while achieving up to 2.1× faster expert forward throughput.

Most reviewers acknowledge the efficacy and simplicity of the proposed algorithms. Most concerns lie in real running time and ablation analysis. During the rebuttal, the authors did a great job of addressing these concerns. We recommend an acceptance.