Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper develops techniques, systems and models to leverage conditional computation, greatly scaling model size while only requiring sublinear computation with respect to model size.   As a result, a sparsely-gated MoE model with 600B parameters for NMT tasks has been trained efficiently (4 days on 2048TPUs) with new state-of-art accuracy.

Merits of the paper:
- Provide an effective strategy to address the expensive computational cost of training giant models
- Offer a comprehensive solution covering techniques, systems and models 
- Well-designed APIs and implementations to express a wide range of parallel computation patterns with  minimal changes to the existing mode code
-  New state-of-art models have been trained using the strategy and systems, as a great demonstration on the effectiveness and efficiency of the approach.

Places to improve
-  It is helpful to further clarify the difference of this work comparing with the prior work Shazeer et al. (2017) beside transformer vs RNN.  For example, are the features like load balancing, efficiency at scale, auxiliary loss, random routing new contributions of this work, or similar as the prior work, or with certain incremental improvements?  It would also be helpful for readers to understand, which techniques are generic and which are model-type specific.

- Both this work and the prior work Shazeer et al. (2017) apply conditional computation on neural machine translation tasks.  It would be helpful to comment on the generality and effectiveness of the solution on other types of tasks.

review_title: Great work with breakthroughs on techniques, systems and models

final_score: rating: 7: Good paper, accept
review_content: ## Summary
This paper addresses the training efficiency issue of large-scale models.
The authors proposed to use GShard API to implement Transformer model with Sparsely-Gated Mixture-of-Experts, allowing sublinear scaling of the computation cost.
Experimental results on a multilingual machine translation task show that the proposed method can use computational resources efficiently.    

## Strong points
* The training efficiency of giant models is one of the significant issues in the field and will attract practitioners' attention.
* The proposed model architecture is reasonable to increase efficiency. The empirical results prove their favorable scalability/efficiency trade-off.
* The results in Section 3.2, such as "scaling depth brings quality gains only after the capacity bottleneck is resolved.", suggest several interesting research directions.

## Weak points
* The readers may confuse because, although the title and abstract of the paper focus GShard, the details of GShard describe in the appendix.
* Notations are not clearly defined, so that the detail of the proposed algorithm is difficult to follow.
* Experiments using more than one trillion model weights are not ready and not shown in the paper. 

## Decision reason
The empirical results are suggestive and valuable to discuss in the community.

## Questions
* What M and H stand for in the equation above Algorithm 2?
* What is the number of groups, G, and expert capacity, C, used in the experiments?  If they can be computed from other constant values, please provide the formula.

## Additional Feedback
* The operation to sets in Algorithm 1 is not clear. 
  * Do Line (3) means m_e = \frac{1}{S} \sum_{s=1}^{S} g_{s, e} for all e in E ? 
  * Is m_e in Line (13) defined as I described above?
  * Although other similar operations, such as c_E <- 0, are easy to guess, I recommend to define the meaning of these set operations for clarity.
* Before explaining its efficient implementation on page 3, it would be nice to explain why a sequential implementation of the gating function is required.   I guess the reason is that the assignment of an expert depends on the assignment of other experts.
* It is better to have a reference to A3.3 in the main body.  Without the reference, readers may wonder why the communication cost is the square root of the number of devices.
* Typo: 
  * page 3: "our software Stack" --> "our software stack" 
  * \sum_{s=1}^{s} in Line (3) of Algorithm 1 should be \sum_{s=1}^{S}

review_title: Interesting results

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper applies mixture-of-experts (MoE) [1] to the Transformers to significantly increase the number of parameters in the model while keeping the total computational cost feasible. The main differences with [1] are: (1) Only choose 2 experts at each timestep; (2) Set a capacity upper bound for each expert to make sure that no expert becomes a lagger. The paper also presents a convenient library to make implementing MoE models easier. The proposed method is tested on a large multilingual machine translation dataset and shows performance gains over models trained on a single language pair and a model trained without MoE.

Reasons for score: 

Although the improvement of translation performance presented in the paper is significant, my overall feeling is that the improvement over [1] is incremental and not clearly justified. See the following pros and cons for more detail.

Pros: 
1. The improvements in translation accuracy are significant and show the effectiveness of MoE for Transformers. This should be an interesting result for the community and beneficial for people working on scaling Transformers.
2. If publicly released, the GShard library should be useful for implementing MoE or more general model parallel models.

Cons: 
1. The novelty of the paper, especially compared with [1]. In [1], MoE is implemented as a position-wise feed-forward layer, which can be directly applied to Transformers with no modification. Also, there already exist previous works applying MoE on Transformers, such as [2].
2. The main improvement over [1] is to add a capacity bound on each expert to mitigate the lagger issue. However, this improvement is not thoroughly evaluated in the paper. I would expect an ablation study to measure the impact on the speed and accuracy of this improvement.
3. The paper proposed a library and a set of APIs for the XLA compiler to make the implementation easy. However, there are previous works (e.g. [3]) already working on a similar but more general set of APIs. How does this work differ from these works, especially [3]?
4. The performances on MT are great. However, all of the evaluations are not on any standard MT test sets (e.g. test sets of WMT). I understand it’s impossible to evaluate the performance of all 100 languages with WMT, but including the results of the standard benchmarks will make the performance gain claimed in the paper more intuitive and stronger.
5. For models of different sizes, it would be better to include a comparison of models training with a similar compute budget (e.g. training the models with the same TPU-days) to better show the benefit of large models.

Minor comments: 
- At the bottom on page 2: “in sufficiently” -> “insufficiently”.
- Figure 2: There is an extra black solid line in the first multi-head attention layer on device E (at the bottom-right of the figure).
- Second paragraph from the bottom of page 3: “software Stack” -> “software stack”.
- Algorithm 2: Please define H and M.

References:

[1] Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." arXiv preprint arXiv:1701.06538 (2017).

[2] Shen, Tianxiao, et al. "Mixture models for diverse machine translation: Tricks of the trade." arXiv preprint arXiv:1902.07816 (2019).

[3] Palkar, Shoumik, and Matei Zaharia. "Optimizing data-intensive computations in existing libraries with split annotations." Proceedings of the 27th ACM Symposium on Operating Systems Principles. 2019.


review_title: Great performance but lack novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: Claims:

-Architecture based on transformer + mixture of experts

-Number of parameters can scale 5-10x larger than transformer-only architectures, while simultaneously achieving 10x shorter training time

-Increased number of parameters allows improvements in BLEU score 

-MoE architecture allows sub-linear computational scaling with increasing parameters, even when mixed with transformer layers

-Argues that conditional computation, via the GShard module, is a good way to tackle computation cost and complexity of parallel programming

Pros:

-600 billion parameters is - as far as I am aware - a new record for the size of transformer-related models.

-It is very promising to see that increasing the number of parameters (via the density of the transformer layers, or the depth of the model), continues to yield BLEU score increases on very large datasets.

-Decreasing computational cost and programming complexity is an important goal - however, I think this should be studied not just at the rarefied high end of extremely large models and datasets, but on more typical scales that would be accessible to more research teams.

-The paper is quite thorough about implementation details, and comes with a comprehensive appendix. This will greatly help with reproducibility. However, I have concerns that the huge size of the sole dataset/task unfortunately work in the opposite direction, hurting reproducibility

Cons:

-The paper only studies one task, on one data set, at one (very large) scale, and only compares to one baseline (standard transformer architecture). Although the paper deserves interest due to its potentially record-setting model size and scale, due to the lack of variety in tasks, data and scale, it comes across as a singular engineering effort rather than a study into the benefits of conditional computation. This creates two issues: (1) it is unclear how well the model generalizes to other tasks (does it only work for really big problems? or will it also be competitive for smaller problems); (2) even if the dataset were made public, would anyone be able to reproduce the results given that 23 TPU core-years is an impractically-high cost for most teams?

-The theme of conditional computation is interesting, but the MoE design isn't too much different from Shazeer et al., 2017. It was a little disappointing that different variations of conditional computation were not explored, perhaps through ablation studies.

-GShard is mentioned as a contribution, but the API design, particularly the ability to specify a dimension of parallelism, is not novel. My recollection is that Jia et al., SysML’2019 already covers the claims the authors are trying to make.

-Regarding the claims on page 7 about model quality improving with greater model depth and denser transformer layers - while intuitive, these claims are based on rather shallow evidence that compares two results points at a time. Consequently, the argument about "increased bandwidth for transfer" seems speculative, and deserves more substantiation. I think a well-designed ablation study would really help back up these arguments.

-There isn't much discussion about related work and how the paper makes novel contributions. Beyond the 600b parameter scale, it seems the key original contribution is to fuse transformers with MOE, but I did have a hard time understanding what other contributions were significant and novel.

Recommendations for improvement:

1) Having experiments on smaller model sizes, particularly in the 10m-10b range, would help make the results reproducible by typical research/engineering teams. Smaller, publicly-available datasets (ideally in the billions of tokens, rather than trillions) would go a long way too. If, on top of that, the authors studied a second task (other than multilingual translation), I think the paper would be very solid.

2) Variations of the MoE architecture, or perhaps ablation studies, could really help to bring out the conditional computation theme, improve the novelty of the paper, as well as provide stronger evidence for the quality-gain claims on page 7. For example, the authors talk about Load Balancing, Auxiliary Loss, and Random Routing, which I understand are not present in the original MoE paper. Perhaps these could be a starting point for designing a set of ablation experiments.

3) It would be good to understand what is novel/improved about GShard, compared to existing systems-ML literature


review_title: The paper demonstrates a neural architecture that combines transformers with mixture-of-experts, and shows that it can be trained on up to 600 billion parameters, a roughly 5-10 fold increase over pure transformer architectures. The increased number of parameters allows improved BLEU scores on a large-sclae multilingual translation task.

Meta-review: This paper is a study of neural network scaling, with models containing hundred of billions of parameters. To that end, the paper introduce a new module called GShard, consisting of annotations APIs on how to split computations across accelerators, which is integrated in the XLA compiler. This enables the training of models with hundreds billions of parameters. To scale efficiently to very large models, the paper proposes to use transformer networks, where every other feed forward sub-layer is replaced by a sparse mixture of experts (similar to Shazeer et al. 2017). This model is then evaluated on a multilingual machine translation task, from 100 languages to English.

On the one hand, I believe that the contributions of the paper are significant: scaling to 600B parameters, and showing that this leads to better translation quality are important achievement. The analysis of transformer networks scaling could also have an important impact. Finally I think that GShard and its integration in XLA could be very valuable. On the other hand, I agree with some of the concerns raised by the reviewers, regarding the writing of the paper and the reproducibility. I found the paper not well written, and hard to identify the differences with previous work. As GShard is one of the main contribution, I would expect a better description of it in the main text (compared to the MoE which seems more incremental). Regarding reproducibility, I do not think that the authors provided a good reason not to evaluate on standard benchmarks: the test sets could be excluded from the train set through various deduplication heuristics. 

To conclude, I am leaning toward accepting the paper, but believe it is borderline. The reason is that the contributions are significant, and worth publishing. But I would not oppose a rejection based on the reproducibility and writing issues.
Decision: Accept (Poster)