Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new token routing strategy for the MoE model architecture. The existing methods mostly focus on how each token chooses experts to be processed by, but this paper designs the opposite way which each expert chooses which tokens to process. By doing this, the authors claim that MoE models' intrinsic load balancing issue is resolved naturally. Furthermore, by applying more computation to more important tokens, the paper presents faster convergence and higher performance on a GLUE and SuperGLUE tasks.
strengths_and_weaknesses: MoE models are getting more widely used due to their efficient scaling property with higher quality than the vanilla dense models. However, there has been a load balancing issue from the beginning, and it is known to be critical to the model performance. Still, it is hard to solve and prevents models to be effectively trained such as under-trained experts. Given this situation, the paper proposes a very unique and noble approach to solve the problem naturally. This is very timely and makes sense intuitively. The paper is well-organized, clearly written and easy to follow.
questions: 1. In equation (2), can you clarify which dimension is used to do softmax?
2. Can you clarify why subsets of tasks from GLUE and SuperGLUE are selected? Are they all sequence classification tasks? If that's the case, I wondered how expert choice works for token classification tasks (ReCORD). Some of less importance tokens might not be fully processed and could potentially cause inaccurate perditions for them.
3. When experts choose tokens, do they make choices across different sentences in a batch? If so, how would it affect the training dynamics? Can a model prefer some specific sentences? If not, could you explain how a batch is handled?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper works on sparse MoE for large-scale Transformer.

Key idea: Rather than letting tokens select the top-k experts, the proposed method selects the top-k tokens for each expert.


strengths_and_weaknesses: Strengths:

1. The idea of selecting tokens for experts to experts is novel for MoE model training.

2. Better convergence speed compared with previous methods.

3. The method naturally solves the load balance issue of MoE.


********************************************
Weaknesses:

1. It is non-trivial to apply the proposed method to the autoregressive generation of GPT-like models, because all the tokens are not pre-given at the beginning. However, the most successful MoE-based models are GPT applications, which restricts the usage of the proposed method. This limitation is also mentioned in Sec 6.

2. The fine-tuning performance of dense models is not reported in Table 2. Only MoE-based models are evaluated under the transfer setting. It is straightforward to compare the finetuning numbers for dense models.

3. The results are compared with the T5 dense model, but the models use different training data. The numbers are not fairly comparable.

4. The training data are not publicly available. It makes future comparisons difficult for other papers.
questions: - How about the finetuning numbers of dense models in Table 2?

- "model training follows the setups of GLaM": is the model pretrained using the GPT task? 

- The results are compared with the T5 dense model, but the models use different training data. The numbers are not fairly comparable. How about the results using the same data?

- Are there numbers of training the model on public text data, such as CommonCrawl?

- How about the results for vision Transformers or machine translation instead of just language modeling?
limitations: There are two limitations mentioned in Sec 6.

- The proposed method can not immediately apply to auto-regressive text generation.

- Large memory footprint.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a new routing method called Expert Choice Routing for the Mixture-of-Experts (MoE) architectures. In this method instead of routing each token to a specific number of (top-$k$) experts, each expert receives a specific number of (top-$k$) tokens. Naturally using this approach, as mentioned in the manuscript, the following properties will be held:
- Perfect load balancing will be achieved between all of the experts, since every expert receives the same number of tokens
- Different tokens can be routed to various number of experts

Similar to GShard [1] and Switch Transformer [2], the authors apply the MoE and gating logic in the dense feed forward layer (FFN). The authors also use these works as baselines to compare their work against.
This comparison shows that using the same amount of computation cost the Expert Choice Routing method achieves superior results on GLUE and SuperGLUE benchmarks. Additionally, it is shown that using the expert choice routing method, the training efficiency is almost doubled. Lastly, it is shown that when using expert choice routing, training perplexity consistently decreases as the number of experts are increased.

References

[1] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.

[2] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
strengths_and_weaknesses: Strengths:
- The manuscript is well written. The method and background are explained clearly and are easy to follow.
- The paper introduces a novel routing technique which improves the training efficiency and accuracy of Mixture-of-Expert (MoE) architectures.
- The authors' central claims are sound, convincing, and well supported.

Weaknesses:
- There are few writing mistakes which should be fixed:
    - In Table 1, the 0.1B/128E model has 128 experts (not 64).
    - Incorrect citations (e.g. In [18] the paper name is missing. However, from pages 103 to 112 it can be inferred that the paper is GPipe)
    - In section 4.7, the term "We further pushes" should be changed to "We further push"
- In the last sentence of section 4.2, it is claimed that each GShard top-2 step is 20% slower than expert choice routing. However, this is not supported and explained anywhere within the manuscript. Additional clarification is necessary regarding this claim.
questions: - How different batch sizes affect the performance of expert choice (EC) routing? Are there regimes of batch sizes where using EC routing may not provide any advantages over token choice (TC) routing?
- What is the criteria for bolding numbers in Table 2? I assumed that highest number in each row is bolded. However, after inspecting the numbers, I found that this is not the case. For example, in the 100M/128 scenario on QQP, the highest number belongs to ST-Top-1 but the number belonging to EC-CF2 is highlighted.
- In Figure 2, it is clearly shown that EC routing improves the training efficiency compared to the GShard (GS). Was it also observed that eval perplexity at the end of the training is lower for EC? Does EC converge to a smaller eval perplexity compared to the GS at the end of the training?
limitations: The authors adequately addressed the limitations and potential negative impacts of this work in the manuscript.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents expert choice routing for mixture-of-experts models instead of the more common token-based routing strategies. The proposed method ensures ideal load balancing, resulting in faster training convergence as well as increased performance on downstream tasks, partly attributed through solving the under-spezialization problem in previous MOE approaches.
strengths_and_weaknesses: ### Strengths:

- Simple but effective method.
- Expert choice routing demonstrates faster convergence and improved downstream performance.
- Insightful comparison towards Hashing Layers, confirming that just balancing the load has some performance drawbacks.
- Results are well presented and and claims are well supported by experimental results.

### Weaknesses:

- **(small)**: I think the paper would still benefit from a Figure similar to Figure 4 in [28] to drive home the point of load balancing.
- **(small)**: Adding a rough wall clock time for the training time would be helpful.

### Minor points:

- Next time make sure to omit $\verb+final+$ and $\verb+preprint+$ in the NeurIPS $\LaTeX$ package which adds line numbers and hence makes reviewing easier.
- Try out the `sort&compress` option to natbib which sorts the references and makes things a little cleaner presentation-wise.
- Clean up references to be consistent in terms of the included information, i.e conference citations should have the same style and make sure to handle preprints consistently and add their arXiv identifier. E.g. [2], [4], [18], [30] all have different formatting. [2] doesn't even contain the title.
- Table 1 Caption "Session 4.1" -> "Section 4.1". Generally, I found that you are inconsistent with adding cross reference links to quickly jump to Sections, e.g. in Section 4.5, the reference to Section 3.3 doesn't have one. Switch to `cleveref` and use $\verb|\Cref|$ for consistency and to avoid typos.
- Just below Table 2 there seems to be a lonely (b) with no corresponding (a) that most likely shouldn't be there.
- Table 2: There are a few errors in the printing of bold numbers and formatting. I always assume that bold number highlight the best performance, even if it's not the presented method. Hence, I'd suggest adding that as well as fixing the errors where one of the other methods actually performs better.  **100M/128E**: BoolQ, QNLI, QQP, WNLI -> ST or GS outperform reducing the outperformed tasks from 10/11 -> 6/11. Here also GS Top-2 should be printed bold for RTE. **100M/64E**: GS Top-2 should be bold for WiC. **100M/32E**: Can we get the number for ST Top-1 on CB for completeness? GS Top-2 should be bold for MRPC, RTE. **8B/64E**: GS Top-2 should be bold for BoolQ, MNLI, MRPC, SST2
- Appendix Table 2 EC-CF0.5 last column has additional space that shouldn't be there.

questions: - If I choose an appropriate hashing function for my token-based routing (cf. Balanced assignment [28]), shouldn't that ensure a balanced expert load and as such contradicts your statement that previous work "leads to an imbalanced load of experts" ?  
limitations: Limitations are appropriately addressed.
review_title: null

Meta-review: metareview: This work introduces a new token routing strategy for MoE models.  Instead of allocating a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens, the proposed strategy adopts a heterogeneous mixture-of-experts employing an expert choice method. 

The proposed method is simple, training efficient, and empirically effective. The paper is well-written and easy to follow. Overall, it is a good paper. 

confidence: Certain
Decision: Accept