Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors provide theoretical and empirical study on the mixture of expert layer, which has recently become increasingly used and for which there is limited theoretical understanding. They show that cluster structure in the data is important for success of MoE, that under specific data distribution with cluster structure a single expert 2 layer CNN cannot generalize while a linear MoE can. They also show the advantages of non-linear MoE.  
strengths_and_weaknesses: 
*Strengths*

- Analysis on an important class of model/network components that are not well understood
- Theoretical analysis appears sound
- The writing is clear

*Weakness*
- The distribution studied is limited, i would like to see a more in depth discussion regarding why the distribution (with K centroids) is representative of real data. Indeed the experiments hint naturally many data does not have this underlying structure (e.g. CIFAR-10 performance is not improved by using MoE)
- (minor) There is some slight disconnect between the original question posed "Why do the experts in MoE diversify instead of collapsing into a single model? And how can the router learn to dispatch the data to the right expert?" and the work done, would be goo to make this flow smoother.



questions: none
limitations: None
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper engages in a formal study of a mixture-of-experts model (MOE) by considering a specific synthetic dataset with multiple clusters and a particular choice of the MOE model architecture. It outlines several theoretical and empirical results that show the importance of having multiple experts to properly fit the training data and the importance of nonlinearity in the architecture of an individual expert. It also mentions several typical problems that occur when training MOE models and discusses a set of techniques that address these issues. Finally, it extends empirical results to a more realistic rotated-CIFAR-10 dataset.

strengths_and_weaknesses: ### Strengths

1. This paper outlines a simple synthetic data distribution and a MOE model architecture that make it possible to conduct theoretical analysis of this problem and show a conceptual difference between a single- and multiple-expert scenarios. The proposed system is a simple model showing the essence of the advantage of the mixture-of-experts networks versus their simpler single-expert counterparts.
2. The empirical study goes beyond the simple synthetic setup and is extended to a rotated CIFAR-10 dataset showing the improvement provided by the mixture-of-experts model in this simple yet realistic case.
3. The paper outlines several practical techniques that altogether allow one to train mixture-of-experts models that are otherwise prone to various training issues.

### Weaknesses

1. The paper has multiple minor misprints and inaccuracies and may need another careful pass (numerous minor misprints, missing or unnecessary articles, missing parentheses in equations, for example in Lemma 5.2, etc.).
2. Several aspects of the theoretical analysis including the importance of expert nonlinearity for specific theoretical results may currently confuse the reader.
3. The problem setup is somewhat simplistic and cannot be easily extended to more realistic setups. But this limitation can of course be difficult to avoid if one wants to obtain meaningful theoretical predictions.

questions: 1. While Theorem 4.1 requires the $\alpha$ and the $\gamma$ distributions to match, empirical results discussed in Section 6.1 were obtained in the $\mathcal{D}_\alpha \ne \mathcal{D}_\gamma$ case. Are there reasons to think that it would be interesting to confirm expected results for a single-expert model and more importantly verify that mixture-of-experts models perform sufficiently well in this case (where the distributions of the signal and the noise multipliers are effectively indistinguishable)?
2. It is not clear if the result of Theorem 4.2 actually requires the experts to be nonlinear functions, can this question be clarified? If Theorem 4.2 also holds for linear experts, then perhaps it should not be summarized as "Nonlinear MOE performs well"? Currently it is a little confusing to first see authors mention that both linear and nonlinear MOEs are expected to fit the training data, yet only see the result seemingly only holding for nonlinear functions.

limitations: I did not identify any potential negative societal impacts of this work. While also being true for a vast majority of other theoretical studies, the proposed framework is somewhat limited by a number of simplifying choices that make it difficult to extend most results beyond the studied scenario.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper analyzes several features of mixture of experts (MoE) and derive a few conclusions such as the superiority of MoE in terms of the classification accuracy and it good loading balance.
strengths_and_weaknesses: Strength: The paper is well written and clear motivation. The results that show MoE is more effective on clustered data is interesting.

Weakness

The analysis about MoE is too superficial. In the experimental, the author shows MoE can beat single expert, which is actually a common sense. It is also not surprising that a non-linear MoE is better than a plain MoE. For most of MoE papers, they use the MLP as the expert, which is a non-linear module.

The experiments are too toy to reflect the real working mechanisms behind MoE. Nowadays, MoE is often deployed on ViT architectures that are trained with JFT 300M or ImageNet 22k data. It is really hard to say the analysis built on synthetic data and cifar 10 is still applicable to large models. For instance, in L34-35, the author posted a question about why experts are not collapsed into a single expert. From my view, it is due to the loading balance loss that is used during training, otherwise it will indeed convert to one expert. I hope the author should at least further clarify his/her point about this.
questions: The key limitation is the analysis is not deep enough. If the analysis is empirical, more experiments are needed. If the analysis is based on math, then concrete prove should be added. Most of the math shown in this paper is used to describe the design of the experiments and does not help me understand the working mechanisms of MoE. Then more analysis on larger model and datasets are also necessary.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The presented paper aims to crack open the blackbox of sparsely activated mixture-of-expert models and tries to understand what leads to the improved performance on many neural network tasks. Through experiments on a toy task, CIFAR-10, as well as a new CIFAR-10-Rotate dataset, the key-findings include that performance improvements are highly connected to the intrinsic cluster structure of the data  and dependent on the non-linear activation function in the experts.  
strengths_and_weaknesses: ### Strengths

- Intuitive visualisations on a toy task to understand the behaviour when training sparsely activated models and additional experiments on a created dataset named CIFAR-10-Rotate with strong underlying cluster structure.
- Empirical evidence that MoE's are heavily dependent on the non-linear activation function to learn the target distribution backed by visualisations for the router dispatch entropy in Figure 3.
-  Great effort of putting theory behind some of the empirical observations found in the field. Theory papers like this are essential for effectively moving forward in the empirical-driven field of deep learning.

---

### Weaknesses: 

### [Update]: Weaknesses have been addressed in the rebuttal and I've increased the score from 6 to 7. 

- **(medium)**: Table 1 + 2 only compare Single with linear + nonlinear activation function to the MoE models and shows that the Single model isn't able to learn the problem at hand to the same extend as the MoE variants. Since effectively MoE models have more parameters to play with (albeit them being in parallel branches), I believe adding total + effective parameter counts and adding one more line where we extend the Single model to the total number of parameters in the MoE model would be beneficial. This would verify that MoE's aren't just finding winning lottery tickets in their parameter initialization which allows them to perform better.
- **(small)**: The paper does consider other non-linear activation functions in the Appendix but here the setting is different than in the main paper. I would like to see Table 6 for the settings in Table 1, i.e. with $\alpha \in(0.5,2), \beta \in(1,2), \gamma \in(0.5,3), \sigma_{n}=1$ and  $\alpha \in(0.5,2), \beta \in(1,2), \gamma \in(0.5,3), \sigma_{n}=2$.
- **(large)**: Comparing the results of CIFAR-10 and CIFAR-10-Rotate, I find it surprising that MoE's don't seem to benefit CIFAR-10 although it should also have intrinsic clusters (similar to CIFAR-10-Rotate) which different experts should be able to utilize. In a paper that tries to understand MoE's and states that the cluster structure is important, I was hoping for a better explanation (or at least a hypothesis) in l.340 -- l.345 that would help to assess when MoE's are beneficial and what properties to look for in the dataset. In its current state, I do not know how I would transfer some of the findings in the paper to other real-world datasets e.g. multilingual machine translation. Maybe adding additional datasets from other domains or other computer vision datasets would help to solve this problem. 
- **(large)**: Continuing from the previous point, a nice analysis would be to visualise the latent embeddings on CIFAR-10 & CIFAR-10-Rotate similar to Figure 1 using t-SNE and see if we can distill any inherent characteristics about the clustering behaviour. My assumption would be that in CIFAR-10-Rotate the inter-cluster separation/distance is a lot higher than in regular CIFAR-10 which enables better expert learning. Thinking this further, maybe we could add additional regularization during the the training of MOE's on CIFAR-10 to enforce better separation and simplify the training for the different experts.


---

### Minor Improvement Suggestions:

- Related literature that seems relevant enough to cite: 
    - "*BASE Layers: Simplifying Training of Large, Sparse Models*" (Lewis et al., ICML 2021)
    - "*Hash Layers For Large Sparse Models*" (Roller et al., NeurIPS 2021)
    - "*Tricks for Training Sparse Translation Models*" (Dua et al., arXiv 2021)
- Typos & Formatting:
    - Table 1 can be made more easy to read and parse by using the $\LaTeX$ `booktabs` package with proper spacing. Similarly, Table 2--6 would benefit from it as well.
    - The definition the empirical loss after l. 175 & dispatch Entropy after l. 316 should have an Equation number
    -  l. 121 Radamacher -> Rademacher
- l. 165 refer to the experiments with other activation functions in Table 6 in the appendix where you consider more commonly used choices such as $\mathrm{ReLU}$ / $\mathrm{GeLU}$.
- l. 193 cite Adam paper
- Generally refer more to the Appendix in the main paper. It is really hard to follow the reading flow without it.

---

Apart from the mentioned points, the paper is well written and presentation, structure, and general layout of thoughts are up to NeurIPS standard. Related work is mentioned and cited accordingly apart from the previously mentioned papers.

questions: see above
limitations: Limitations are appropriately addressed in the conclusion and future extensions are given.
review_title: null

Meta-review: metareview: This paper analyzes mixtures of layers applied to DNNs, in order to understand their limits and how they benefit to DNNs. A significant theory and experiments support the claims of this paper, despite the doubts of one review on the experimental protocol. Consequently, I recommend the acceptance of this paper.
confidence: Certain
Decision: Accept