Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Paper studies the routing problem in sparsely gated mixture of experts model for text. The central contribution of the paper is experimenting the efficacy of deterministic routing strategies that are not learned over the course of training but fixated at the beginning of training by utilizing hash functions. Various hash functions are experimented on four different tasks, and compared against prior art methods (BASE layers and Switch Transformer).

- Overall well written paper with a focused contribution
- Conclusion has a more compelling and striking story which would be great to hear early on: learning free approaches for routing are quite competitive hinting that the learned routing algorithms may not be doing what we expect them to be doing. 
- Paper suggests the proposed algorithm should be considered as a baseline to other methods, and given the results I cannot disagree with the proposal. 
main_review: Uses a hash function on the "tokens" to decide the expert being routed throughout the network, where each expert being represented by a bucket in the hash map. 

Various different types of hashing functions experimented/proposed: random, balanced, bi-gram hashing, previous-token hash or hashing based on position, clustered or dispersed hashes etc.

**Questions**
- What is the effect of vocabulary size, vocab composition to the final quality?
- If we were to switch to character or byte level vocabularies, how could we apply the proposed algorithm?
- What would be the extension you could propose if we were to extend this to vision? While you criticize the learned routing methods, they are also widely applicable to other domains. Could you please elaborate on this. 
- If we were to use multiple moe layers, then all the tokens are always going to be sharing the same expert for each layer right?
 Multihash layers, --> caveat, you may end up activating the entire parameter space, will have a downside on the inference, but also each token will have the chance to touch all the experts, slowing down the training (practically making it a fully dense model if $N$ is equal to the number of experts, and each hashing method assigns a token to a separate expert). 

**Typos**
- line 74: advanced -> advance
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies a simple hash-based approach to route input tokens in sparsely-gated mixture-of-experts models. The proposed approach for routing is a static hash-based approach, input tokens are hashed to 1 of the several experts in the MoE layer without the need for any learnable routing network parameters  which are typically used in MoE layers. This approach is compared against Switch Transformers and BASE layers [1, 2] on a generative dialogue task on the Pushshift.io dataset, language modeling on Roberta+cc100 and Wikitext-103 datasets and on fine-tuning on the BST dialogue tasks. On all these tasks hash-based routing is either competitive with or shows slight improvements over the previous (more complex) routing approaches.

In addition to the comparison against Switch and BASE layers, the paper also conducts some analysis on:
1. Several approaches for hash-based routing, and finds that simple balanced or random hashing performs better than approaches that take into account the preceding tokens or predicted future tokens for routing. 
2. The effect of the number of experts (or the amount of sparsity) on the performance of Switch layers and Hash layers, and demonstrate that learnt routing layers are more competitive at lower levels of sparsity.

References:

[1] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus et al.

[2] BASE Layers: Simplifying Training of Large, Sparse Models, Lewis et al.
main_review: Pros:

1. The paper makes an original and valuable contribution, demonstrating that a simple hashing based approach can be competitive with more computationally expensive and complex routing approaches for MoE layers. 
2. Provides a strong and simple baseline for future work on routing in MoEs, while highlighting a major weakness of MoE models being studied in literature.
3. Clearly written, easy to understand and contains several interesting ablations and analysis.

Cons:
1. The paper is limited in scope to a particular type of architecture and task. The analysis focuses on MoE layers in either decoder-only LMs, or encoder-decoder models with very light-weight encoders. The increased capacity is also limited to layers with causal attention, which might affect the need for more complex routing strategies. It's not clear how well hash layers or non-learned routing strategies would perform on tasks where MoE layers have shown major improvements in recent work, like NLU tasks with span-mask pre-training [1] or multilingual MT [2,3].

Other questions/comments for authors:

1. For figure 3b, is there a corresponding plot with position vs performance for switch layers as well? Are the trends any different from hash layers?
2. Line 200: "we hypothesize ... work": Or perhaps this indicates that routing functions are easier to optimize with top-k style gating at lower levels of sparsity / fewer experts?
3. Line 205: "We find ... work well" this finding is also similar to that observed for PKM. [4]

References:

[1] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus et al.

[2] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, Lepikhin et al.

[3] Exploring Routing Strategies for Multilingual Mixture-of-Experts Models, Kudugunta et al.

[4] Large Memory Layers with Product Keys, Lample et al.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose using hashing layer to replace the routing mechanism in the methods of mixture-of-expert such as Switch
Transformers and BASE Layers. The main benefit of the proposed method is that no routing parameters need to learn and achieve quite good performance on the language modeling tasks. The authors also have a wide exploration of different hashing mechanisms, such as random hash, balanced hash, bigram hash, clustered hash, etc. 
main_review: Overall, the proposed method is interesting and easy to use. It's promising for the community to further boost the performance of super-large models. The authors have also done a solid comparison between the switch-transformer and wider/deeper models. However, my major concern is why this method can be better than switch-transformer which theoretically can learn more knowledge to route than simple hashing. I think there are some methods to make a better hashing function, such as balanced hashing. Could these methods also apply to switch-transformer to further boost performance?

Other comments:
1. The proposed model is only tested on language modeling tasks. I am curious about the performance of the downstream tasks, such as GLUE after pertaining with cc100 corpus.
2. The hashing function has been adopted to build sparse attention in Transformer by the works of Reformer and Routing-Transformer. The authors adopt a similar method but focus on the MLP layer in SwitchTransformer. I would say this is new and interesting, but not very novel.
3. The authors have proposed many different hashing methods. However, the simplest hashing method still works almost the best. The relatively new methods don't work well. Most experiments on other datasets are required if the authors want to claim novelty on different hashing functions. Currently, it is only evaluated on Reddit which is also not a popular LM data.

Questions:
1. Are all layers using hashing layers? What are the detailed structures of using hashing layer?
2. Any challenges when training hashing layers? Is it easy or stable to start training?
limitations_and_societal_impact: 1. Hasn't proved to be a general framework for different NLP tasks, only testing on language modeling tasks.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: **Summary And Contributions:**

This paper is interested in routing strategy in MoE Transformers in NLP where each tokens of the sequence are “routed” to a specific FFN (also called expert). The authors argue that the difficulty of learning parametrized routing strategies (especially as the number of experts grows) calls for simpler baselines. The experiments show that a simple hashing baseline (pre-defining the expert routing for each token based on some token-frequency rules) is an easy-to-implement, robust and strong baseline that is either on par or outperforming learned routing strategies. Most of the comparisons are done with the perplexity on a variety of datasets and the authors also analyze a series of different hashing strategies.


main_review: **Strengths:**

- The paper is well organized and written, and it’s easy to follow.
- Experiments are quite exhaustive and convincing, giving to a strong and easy to implement baseline for future research on Transformer MoE

**Weaknesses:**

- While the differences in perplexity are noteworthy, it is less convincing that there is a difference in downstream fine-tuned performance. It is not straightforward that further gains in perplexity are interesting, especially from the perspective of memorization (that the authors briefly touched upon in the appendix).
- It would be valuable to extend the comparisons to more fine-tuning setups which would bring a more useful signal from the perspective of applications (compared to perplexity), especially since it is not clear that gains in perplexity necessarily translate to gains in downstream performance (fine-tuned or zero-shot).

**Questions:**

- Training very large dense language models (more than a billion parameters) is known to be unstable (the loss can diverge unexpectedly during the training). Did you notice any differences in training stability in hashing vs learned-routing strategies?
- How would this approach be extended to multilingual setups? Any thoughts on how to scale the number of experts? The dictionary size?
- Any thoughts on the memorization abilities of the obtained models? Is it possible that improving the routing makes it easier for the model to memorize random facts and training instances?
- Is the balancing additional loss used during fine-tuning too?

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This work presents a surprising result that a deterministic routing strategy for sparse MoE models can yield competitive results for language tasks compared to learnable approaches. The experiments conducted are extensive and well thought. Regarding using perplexity as the only comparison metric, I suggest the authors either consider additional metrics (e.g., qualitative/human evaluation) or using multiple perplexity values, e.g., in Figure 2 or 3, to make a conclusion. The reason is a single small difference of < 1 ppl point (at the range of ppl > 20) might not be conclusive. Nevertheless, I agree with the reviewers that this paper is well written and the proposed approach serves as a simple and strong baseline. Hence, I recommend Accept.
Decision: Accept (Spotlight)