Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: SaMoE is a novel routing algorithm for mixture of experts (MoE) that allows different MoE layers select experts from a global shared pool. Well designed experiments and scaling law studies are reported in the evaluation section.

strength_and_weaknesses: Strengths:
1. I think the authors found the right critical bottlenecks for the MoE models, which are the trainability (model quality aspect) and the total number of parameter count (system performance aspect). The proposed solution with the empirical results shows it's on the promising direction,  although it's still far from completely addressing those foundational limitations of MoE, 

2. The ablation and scaling laws section are very helpful to the research community to understand how to set the hyperparameters. 

Weaknesses:
1. How the speedup in table 1 is evaluated?

2. The gains in the downstream tasks are marginal. It's better to report the variation of zero-shot results at nearby checkpoints as well. 

3.  I feel it's important to report the inference step time during autoregressive decoding to best demonstrate the gains from a smaller number of parameters. Because during decoding on accelerators, it's more often memory bound instead of compute bound. When decoding a single token given the prefix, the flops to compute each token is relatively small. However, the whole model parameters (in billions or even trillions) needs to send more HBM to the actual compute units during each decoded step. This HBM-cache communication is usually the dominant factor in the inference cost.

4. Double check the multi-rc results? It's a big jump from 1+ to 70+.

clarity,_quality,_novelty_and_reproducibility: The flow of this paper is crystal clear. The authors first identified the bottlenecks of an existing algorithm, found the root cause, proposed a solution, and finally demonstrated the effectiveness of the proposed solution with well prepared experimental evaluations. The text, tables, and figures are all of high quality.

summary_of_the_review: This paper worked on an important research topic: how to reduce the training and serving costs for large language models. The proposed algorithm is only marginally novel but empirically significant. The 5x reduction in the number of total parameters would improve the serving speed for MoE models by a lot. 

However, the gains in the downstream tasks are only marginal. So it would be better if the authors clearly demonstrated why the large reduction in the parameter count matters using the metrics people care about: the serving latency, the step time, etc. Alternatively, the authors can show the quality difference when matching the inference cost.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: MoEs have been reported to be parameter inefficient such that larger models do not always lead to better performance. This work proposes a parameter-efficient MoE models, by learning a soft combination of a global set of expert layers for each MoE layer. Experimental results show that SaMoE improves parameter efficiency by reducing up to 5.2x parameters while obtaining strong pretraining and zeroshot generalization results. 
strength_and_weaknesses: Strengths:
- The proposed method is simple in design and implementation but achieves reasonably good results. 

Weaknesses:
- The paper does not provide a fair comparison by fixing total parameters but ignore the computational cost (FLOPs) or activated parameters. In traditional MoE research, the general goal is to achieve better quality with a fixed computational cost (FLOPs), not with a fixed total parameters. The reviewer understand that this method provides a efficient way saving total parameters, but the reviewer suspects to achieve better quality, this method would also significantly increase the FLOPs compared to traditional top1 or top2 based routing used in switch transformer and GLaM. Table 1 does not provide any details on computational cost. According to multiple prior works including T5 and the Chinchilla [3], there is always a tradeoff between model capacity and training tokens, the larger the model is, the lower training data/steps can be achieved within the same computational cost budget.  According to Table 1, activated parameters are made fixed around 1B, however, it might not be clear about computational cost in this work's setting. 

- The paper misses fair comparisons with many significant related work including autoregressive sparse MoE, GLaM [1]. GLaM adopts a top-2 based routing, that can yield much better results than top-1 based routing. Various efficient routing functions should be compared with in this work, as intelligent routing functions achieve similar effects of improving parameter efficiency. For example, Expert Choice [2] routing achieves heterogeneous experts such that different tokens can utilize a variable number of parameters. 

[1] https://arxiv.org/pdf/2112.06905.pdf
[2] https://arxiv.org/abs/2202.09368
[3] https://arxiv.org/abs/2203.15556

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 
The proposed method is a minor improvement over traditional parameter sharing scheme like suggested in Universal Transformer. 
The paper can be reproduced but the results are not valid. 
summary_of_the_review: "No free lunch" in deep learning: reducing parameters and reducing training time will not come for free without sacrificing model quality. 
The paper's results are based on fixing total parameters but not on fixing computational cost (activated parameters and FLOPs), which can be unfair to many related works including GLaM. The reviewer would not believe in any results that purely relying on parameter sharing, we could improve quality without introducing additional computational cost. For example, whether this method increases activated parameters (experts per token) is unclear and should be explained. For example, the paper can be increasing the number of layers or expert width or number of experts per token compared to GLaM. All these increase activated parameters, thus inference time. The paper should be also more proactive in explaining why quality gains can be achieved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper:  This paper proposes a new MoE model Architecture to improve the parameter efficiency of MoE by learning a soft combination of a global set of expert layers. 

strength_and_weaknesses: Weaknesses: 

(1). This paper carried out analysis first and listed three challenges from analysis. However, I did not know which MoE model does this paper study. In Figure 1, it shows “MoE” but I don’t know which MoE model is used to carry out experiments. There are plenty of MoE models such as Gshard (Lepikhin et al., 2020), Switch Transformer (Fedus et al., 2021), Base Layers (Lewis et al., 2021), HASH Layers (Roller et al., 2021), and etc. Different MoE models may lead to different conclusions. The author needs to announce which model they used for analysis and add citations. 

(2). This article used only one MoE model to draw analysis conclusions, which I cannot agree with. Because different MoE models may have different performance, analysis conclusions need to conduct experiments with at least two representative MoE models when talking about common challenges with MoE models.

(3). I am very suspicious about the expert pool method proposed in this article. How to choose the size of the expert pool. I speculate that the amount of experts required by a MoE model may be related to the diversity of the dataset. Table 1 in BASE layers paper [1] shows similar words usually gathered to the same expert unit. However, this article only uses one dataset for pretraining, and does not use multiple datasets to test the required expert pool size.

(4). Followed by the third problem, this paper selected the Pile dataset as the pre-training dataset. However, the Pile dataset is full of duplicate documents (see [2] page 2), and this paper does not perform additional de-duplication processing. Because the dataset selected in the article has a lot of repetition and the tokens are not diverse, the size of the expert pool does not need to be large. The conclusion is likely to change when changing to a different (diverse) pre-training dataset.

(5). As a MoE model, it is basically necessary to control the number of flos and compare it with the dense models and sparse models with the same number of flops, but this paper does not report total training flops number and total train computer (PF-days). In addition, this paper doesn’t compare it with a dense model with the same amount of flops in table 1. 

(6). I also have some questions about the experimental results of table 2. When we compared SaMoE (350M-128E) with dense model (350M), SaMoE should have more flops since it needs additional all2all communication cost. However, I notice usually a dense model (350M) could get a score of 70.2 on piqa. This SaMoE with more flops achieves a score 68.9. 

(7). Minor suggestion: usually we reported pretraining perplexity instead of validation loss in figure 3.


*References:*

[1]. Lewis, Mike, et al. "Base layers: Simplifying training of large, sparse models." International Conference on Machine Learning. PMLR, 2021.

[2]. Zhang, Susan, et al. "Opt: Open pre-trained transformer language models." arXiv preprint arXiv:2205.01068 (2022).

clarity,_quality,_novelty_and_reproducibility: Quality:

Due to the above weaknesses, this paper does not have a high quality.

summary_of_the_review: This paper proposes a new MoE model. However, in the analysis part, it only carried out analysis experiments with one MoE model, which is hard to tell if findings applied to all MoE models. In addition, this paper proposes to have a fixed number of global MoE layers, which is probably not suitable when a pre-training dataset has very diverse tokens. It happens that this paper selects the Pile as the pretraining dataset, and Pile is widely considered to contain many repeated sentences. (see [2] page 2). 

*References:*

[1]. Lewis, Mike, et al. "Base layers: Simplifying training of large, sparse models." International Conference on Machine Learning. PMLR, 2021.

[2]. Zhang, Susan, et al. "Opt: Open pre-trained transformer language models." arXiv preprint arXiv:2205.01068 (2022).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a parameter-efficient version of Mixture of Experts (MoE), which is known for requiring a large number of parameters to perform well. The proposed model, called SaMoE, uses a soft combination of a global set of expert layers for each MoE layer, which reduces the number of required parameters by up to 5.2 times. The experimental results show that SaMoE not only is more efficient in terms of parameters, but also performs well in pretraining and zero-shot generalization tasks.

The reviewers raised concerns on the evaluation metrics (such as serving latency and step time), comparison fairness regrading baselines, which remain after the discussion. The paper could be significantly strengthened if these concerns can be addressed.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject