Hash Layers For Large Sparse Models
Authors: Stephen Roller
Sainbayar Sukhbaatar
Arthur Szlam
Jason E Weston
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: large-scale
sparsity
Transformers
hashing
MoE

Abstract: We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence. We show that this procedure either outperforms or is competitive with learning-to-route mixture-of-expert methods such as Switch Transformers and BASE Layers, while requiring no routing parameters or extra terms in the objective function such as a load balancing loss, and no sophisticated assignment algorithm. We study the performance of different hashing techniques,  hash sizes and input features,  and  show that  balanced and random hashes focused on the most local features work best, compared to either learning clusters or using longer-range context. We show our approach works well both on large language modeling and dialogue tasks, and on downstream fine-tuning tasks.

Hash Layers For Large Sparse Models
Stephen Roller
Sainbayar Sukhbaatar
Arthur Szlam
Jason Weston
Facebook Al Research
Abstract
We investigate the training of sparse layers that use different parameters for different inputs based on hashing large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence. We show that this procedure either outperforms or is competitive with learning-to-rout mixture-of-expert methods such Switch Transformers and BASE Layers, while requiring no routing parameters or extra terms in the objective function such as load balancing loss, and sophisticated assignment algorithm. We study the performance of different hashing techniques, hash sizes and input features, and show that balanced and random hashes focused on the most local features work best, compared to either learning clusters or using longer-range context. We show approach works well both on large language modeling and dialogue tasks, and on downstream fine-tuning tasks.
Introduction
Recent studies of Transformer models have shown clear trend towards improvements with scale in data and model size [1]. mirroring the same trend in Machine Learning more generally. However, when architected naively, larger (in terms of parameter count) models are slower train and to evaluate; and at extreme scale, with current computer systems, necessitate complex engineering to facilitate communication between workers. To address these challenges, researchers have studied cettures-of-Experrt (MoE) models [2] 3
g where "gater" routes computation through a sparse subset the weights of model (the "expert modules"). Specifically in the setting of Transformers for Natural Language Processing (NLP), recent approaches have led to state of the art performance in language modeling [8). MoE models allow increasing the number of parameters in the model while holding steady the number of computations that affect given sample.
key component a MoE model is the routing (gating) strategy. While MoE models can be computationally advantageous per parameter compared to dense model, they might be functionally less powerful per parameter.
poor routing strategy might lead to expert modules that are not properly specialized (essentially making stochastic ensemble model); overly specialized, using the data assignment function to overfit. Meanwhile, the routing strategy itself must be efficient.
standard approach is to train a layer of weights that makes the routing decision based upon the input the layer be routed. Classically, this may have been implemented with softmax over the choice of expert modules, and fitted via backpropagation. However, a dense softmax requires all expert modules to run on all data points at train time, which negates the computational savings. Several works have shown that sparsity can be maintained during training, e.g. 7 g 10 In particular, Switch Transformers g select the top expert per token using a softmax over the token's hidden state, but require load balancing term in the objective function or they can become imbalanced or degenerate, giving poor results. BASE Layers [10] employ a linear assignment algorithm to try to resolve the same problem.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

hr'
Layer
MoE FFN
FFN
FFN
FEN
FFN
FFN,
FFN,
FFN,
Layer
self attention
h
"We"
"eat"
"every"
"taco"
Figure 1: Overview of the Hash Layer. Tokens are routed to fixed expert modules based on their hash.
In this work, we describe a simple, sparse, efficient routing strategy based on hashing input tokens that is effective in the Transformett-fo--NLPP setting. We show this approach is effective on a number of datasets, comparing favorably to both Switch Transformers and BASE Layers, As the routing strategy requires no extra parameters, no change to the objective function or assignment algorithm, its simplicity means it is robust, fast and easy to implement. We provide detailed analysis to explain why our method works, and in which conditions. Given that when training very large models one may typically have only one shot given the required compute budget, and experimenters will be unable to try many parameter choices, we hence advocate our approach as a strong candidate for such a setting.
Background
Let us first introduce the Mixture-of-ENee of-pxet setting where we apply hash-based routing strategy.
We use the same setting as [1] 8 10] where a feedforward network (FFN) in a Transformer is
replaced by its MoE version. Given a tokenized input sequence {21, L21,22...... IT of T tokens,
representation for each token is computed in parallel by standard Transformer [12]
h hy,,.,, 1i,,..,,,,, h = TRANSFORMER SERAFORRER(EE,120
The Transformer consists of T. layers that computes final hidden states for each token, and each layer is composed of self-attention and FFN sublayers, where FFNs are two-layer fully connected networks
h SelfAtn(hh(-))
= FFF(())
(2)
Here we omit skip-connections and normalization for brevity. We can then replace one or more the FFN sublayers with expert modules. Replacing the FNN at layer with K expert FFNs, their output then mixed with some gating function 9()):
K Eg
h = (FN(
h -
(F)) 1= 1=
(3)
where importantly each token routed different mixture of experts, as the gating function depends on the token's 's specific hidden state h/-
Sparse MoE methods assume gating values gi often zero, only few experts need to be computed for better efficiency. As expert FFNs do not share parameters, the number of parameters increases with K while the amount of computations per input token stays the same if the MoE FFN only routes to a single expert, and computation of 9; is cheap. While this allows training of large capacity models with small compute budget, optimizing 9i in the sparse setting can be tricky.
3 Method
In this paper we propose a simple gating mechanism that is especially efficient because only one expert is active, and has no routing network parameters to learnt. Recent work [ g 10] has to learn parameters that determine the routing to expert modules based on hidden states, which have be optimized tandem with the expert weights themselves. This can potentially cause difficulty because during training membership for each expert is changing while it is trying to learn the mapping

---

for those members. We instead advocate for fixed mapping to experts. Namely, by hashing the tokens into a fixed number of buckets, each bucket corresponding to an expert:
h aa (hi), 1 1=.....,7
(4)
While the FFN still takes the hidden state hj as input, our routing function uses the original input token T2 rather than the hidden state, see aeelFi 1 for graphical depiction. We are free to choose from various possible hash functions, which we will consider below. However, for training purposes, the hash function is fixed in advance, and in this way, our routing mechanism requires no training and has no adjustable parameters.
3.1 Hash Functions
Hash functions have long been employed throughout Computer Science 13. of forms. In our work, we generally employ pre-computed hash functions, which use a lookup table during learning - precomputed in advance to map tokens to expert modules.
and can take a variety
We consider several kinds of hash functions as possible choices for routing tokens to expert modules. The simplest Random Hash, wherein we assign every token fixed, random expert at initializa- tion. Due to the Zipfian distribution of token frequency, this naturally produces imbalance across the different expert modules. As balancing has been previously shown to be important for training MoE models g [0], we also consider Balanced assignment. In this method, we build the lookup table before training the model using the training data distribution by greedily assigning the most frequent tokens to the emptiest buckets. The resulting assignment structure is significantly more balanced than Random Hashing, but not perfect, as the frequency of some tokens exceeds the ideal distribution.
Random and Balanced hashing exploit the inductive bias of auto-regressive models and hash on the input token, but we also consider other possibilities: Bigram Hash uses the current and previous token (z,-1, xτ)) rather than only the current token, while Previous Token Hash uses the previous token 1;-1, ignoring the current input. We also consider sanity check which hashes based on the Position in the sequence, which we expect to have little impact, as absolute positions carry little information in natural language. Each of these hash functions is used to assess the value of the information being routed-on in our subsequent experimental analysis.
As upper baseline, we also evaluate using an Oracle Future Hash, which hashes based on the output loken 11+1, rather than input token. This Oracle Hash checks how powerful routing decisions can be solving task. Similarly, we also consider Predicted Future Token Hash, which utilizes a baseline Transformer to make a prediction of the output token, and then hashes over this prediction.
Clustered Hashes Based on the intuition that similar tokens may want to be routed to the same expert, we also experiment with Clustered Hashes. We obtain clusters by performing k-means clustering with a fixed number of clusters using token embeddings from a baseline Transformer model. Each expert assigned a centroid, and tokens are assigned to their closest cluster.
Dispersed Hashes We also consider the opposite hypothesis: that similar-tokens should be placed in different buckets, where the assumption is that very similar tokens need fine distinctions which requires more model capacity (hence assigning to different experts). To do this, we use the same k-means clusters as before, but distribute all tokens within each cluster equally across all buckets.
3.2 MultiHash Layers
In the standard FFN MoE approach, all K expert modules have independent parameters, but here we consider another option. is known in the hashing literature that multiple hashes can provide better allocations in many contexts [14. We consider such schemes in the context of sparse routing. Let us assume we are given N different hashing functions, and for given input token x we compute these hashes, denoted as ki hash.. (x).
1, ..... N. Assuming the usual expert FFN is a function B B(relu(A( ((h))) where A : Rd + RD and B IR + Rd, we split the linear layers into N segments, Am RRd + IR D/N and Bm D + Id/NN Then we compute:
relu([As, (b), (h).... AKN (h))
FFNMH(h) = [Bx, (v),.... ... Bky (v)).
3

---

That is, use hashing to select the parameters we are going to use for each segment, and then concatenate them together. The advantage is that we are now no longer reliant on the quality of single hash function, but have multiple chances to produce good quality partitions. This perhaps can also be seen as analogous to the multi-head attention process already used in Transformers.
Related Work
Sparse MoE. models, where only few expert modules are active for any input, in particular in the context of NLP, have been studied recently in [6 IT. In these works, the gating is learned via backpropagation, perhaps with regularizer to encourage load balancing across experts. [8] showed that models in [[] can be successfully trained with each input assigned to exactly one expert. Another such approach for Transformers, where the routing is learned via solving linear assignment problem, studied in [10]. [15] uses a different approach, where product keys enable nearest neighbor search to select parameters. More generally, using MoE to trade off compute time (at the cost of possible data fragmentation) has a long history, see e.g. 3..7
The approach in this work is different from all of these in that the assignments use no learning whatsoever, and instead make use of the inductive biases possible in the setting of natural language. use the fact that n-grams are themselves decent language models [16]. Thus
In particular, this work is related to previous work attempting to combine neural and n-gram language models
18 19 20 21
Our work is also related to feature hashing in linear models and kernel methods 23
24 where
word or n-gram features are hashed to provide new lower dimensional feature space. 23 showed that when performing such feature hashing the interaction between random subspaces negligible
with high probability. [25] uses hashing to compress neural networks, rather than increase their parameters as we do here. Work on long-context Transformers has recently used hashing techniques to speed up access to long-range token history via sparse self-attention patterns, particularly in (26) and the Reformer [27]. In contrast, our work uses hashing access a large set of parameters via sparse routing, rather than sparse access to input features.
Routing Transformers
5 Experiments
5.1 Tasks
Pushshift.io Reddit We use a variant of Reddit discussions, which has also been used in several
existing studies, see c.g. [28] 29 30 31]. Following (32), we use a previously cxisting Reddit dataset extracted and obtained by third party and made available pushshift.io [33], training generate comment conditioned on the full thread leading up to the comment, spanning 1.5B training examples.
We use the same BPE dictionary
34] comprising of 8008 tokens.
RoBERTa+cc100es 100en Data We use the same data used to train BASE 10,, which consists of approximately 100B tokens, combining corpora used in RoBERTa 351 with the English subset of the CC100 corpus [36]. The GPT2 dictionary, of size 51200, is used for tokenization. For our seq2seq experiments, arrange this data splitting by sentence predict the next turn. We consider it as the originally intended language modeling task in our experiments comparing with BASE [0].
Wikitext-103 Wikitext-108 103 is smaller language modeling benchmark (37 consisting of collec- tion of Wikipedia articles of over 100 million tokens, and a fixed vocabulary size of 270K tokens is provided. We view this seq2seq task in our experiments, again splitting by sentence.
Downstream BST tasks Finally, we use the Blended Skill Talk (BST) dialogue tasks used in (34 after iushshift.io Reddit pre-training evaluate fine-tuning performance of dense sparse models.
5.2 Experimental Setup
Seq2Seq Setup The majority of our experiments are carried out in ParlA platform using an encoder-decoder Transformer framework. We first train several standard (dense) Transformers, with
httpp//parl..a

---

Table 1: Comparison of Models on pushshift.io Reddit. We show three sizes of dense Transformer compared to Switch Transformers and using Hash Layers with various numbers of modules and sparse layers, e.g. 5x16 means 5 sparse layers with 16 modules each. All Switch and Hash Layer modules are built to the same computational complexity as the 11 layer baseline Transformer, but have more parameters; the larger dense models have similar total parameters, but use more compute.
Model
Configuration
Params
Valid PPL
Test PPL
Baseline Transformer
layers=11, 1024, D=4096
222M
24.90
24.96
Wider Transformer (more compute)
layens= 2=2048 D=6144
755M
23.32
23.38
Deeper Transformer (more compute)
layens=22, 1=1336, D=4096
755M
22.72
22.78
Switch Transformer
bal=0.1
751M
23.65
23.73
Hash Layer
layers=1 1,modules=xx644
751M
23.16
23.23
Switch Transformer
layers= Lmmduleeelllssseee22222 load_bal=0., bal=0.1
1.28B
23.52
23.58
Hash Layer
layers= l_modules=1x122
1,28B
22.89
22.95
Switch Transformer
layers= I.modules=5x16. load bal_000=0000
852M
23.19
23.25
Switch Transformer
laycrs= L_oodules=5x116 load bal=0.1
852M
23.00
22.93
Hash Layer
layers=
1.modules=5x116
852M
23.21
23.27
Table 2: Comparison of Models RoBERTa++cc10en Data. We compare dense transformer with the same parameters as our sparse models, except with sparse layer with 64 modules (1x64).
Model
Configuration
Params
Valid PPL
Baseline Transformer
layers= d=1024, D=4096
266M
28 85
Switch Transformer
layers=11, modules=1x64, -al0001
795M
27 77.41
Hash Layer
layers=11, modules=1x64
794M
26.99
2 encoder layers and either 11 or 22 decoder layers, following the structure in 34 for training on pushshift.io Reddit, We refer to the one with 11 layers and embedding size of d = 1024 and FFN hidden layer size of D 4096 our Baseline Transformer. We also train a "Wider" model with D 6144, and a "Deeper" model with 22 decoder layers, and D = 4096. The Baseline model has 222M parameters, and the "Wider" and "Deeper" are selected both have 755M parameters each. These models are compared to the Hash Layer methods detailed in |section 3 and to Switch Transformers of the same sizes and settings. The load balancing for Switch is optimized on the validation set. For both Hash and Switch we use the "Baseline" Transformer size detailed above as the architecture that we add sparse routing layers to by replacing one or more of the original dense layers. All experiments are run for 100k updates; table of hyperparameters is provided in subsection B.
BASE Comparison While most of our analysis takes place in the setup described above with models up to 1.28B parameters, to test our methods at scale on larger sparse models, we adopt the BASE Layer setup [10] and code base instead where we compare 4.5B parameter Hash and BASE Layer models. This setting uses pure language models rather than the Seq2Seq setup above. We use the architecture, data (RoBERTa+ccc CaNeERRTTccc000ee 100en), and hyperparameters directly from [T0], using either single sparse routing layer consisting of 3 stacked FFNs (D = 8192) on the middle layer of a 25 layer network, or 3 routing layers evenly spaced in the network. In order to compare with BASE directly, we keep all hyperparameters fixed and only change the routing method; we use a balanced assignment Hash Layer in this case. We trained until 40k steps had been reached. A table of hyperparameters is provided in subsection B.2]
5.3 Results and Analysis
5.3.1
Comparison between Hash, Switch and Dense models
Hash vs. Switch routing on a single layer We first compare a Hash layer (with balanced hash) to a Switch layer, on an otherwise dense Transformer, where sparse routing is performed on layer 7 of the decoder. Both methods use 64 expert FFNs with 751M total parameters. Results on pushshift.io Reddit are given in Table I] (rows 4 and 5) and on the RoBERTa+ce 100en data (Talle (rows 2 and 3). We find Hash Layers outperforming Switch both datasets by about 0.4-0.5 perplexity.
Made available within Fairseq (38].
5

---

Pushshift.io Reddit
RoBERTa + cc100en
30
16
Transformer Switch
BASE
28
15
Hash Layer
Perplexty
Hash Layer
Perploxity
14
Hash Layer
26
13
24
22
30000 Number of Updates
50000
70000
90000
10000
20000
30000
40000
Number of Updates
Figure 2: Comparison of Hash Layers with other models. (left) Validation perplexity of baseline Transformer, Switch Transformer, and Hash Layer on the pushshift.io Reddit dataset with 128 modules. (right) Validation perplexity of BASE, Hash Layer, and deeper Hash Layer model on the RoBERTa+cc100en dataset. All sparse models have the same number of parameters.
Performance by Number of Experts
Position of Hash Layer
24.00
24.00
Hash Layer
23.75
Perplexity
23.75
23.50
23.50
23.25
Switch
23.25
23:00
Hash Layer
23.00
16
32
64
128
5 Layer
Number of Experts
Figure 3: Comparing Different Number of Expert Modules and Layer Position. We compare (left) the validation perplexity wrt. the number of expert modules on the pushshift.io Reddit task for Hash or Switch Layer on layer 7 of an 11 layer decoder in Transformer. The baseline Transformer obtains a perplexity of 24.9. We compare the performance when adjusting the layer position of 64 module Hash Layer on the same task (right). Placing on later layers works best.
Dense vs. Sparse Models Both Hash and Switch sparse models outperform the dense Baseline (222M parameters) they are hased on, as well as the Wider Transformer (755M parameters). However, the Deeper Transformer (755M parameters) outperforms the sparse models which have similar number of parameters. However, we note that due to its dense rather than conditional compute is slower in inference speed. We see this as general trend: good dense models can get more power out of the same number of parameters than sparse models. However, sparse models, although more wasteful in memory, give better perplexity for the same speed (i.e, we should compare to the Baseline Transformer in this case, which has roughly the same amount of computation).
Hash layer module size We conduct the same pushshift.io Reddit experiments as above, but altering the number of expert modules in both Hash and Switch. Increasing from 64 to 128 modules (1,28B parameters total) sees even larger improvement of Hash over Switch (about 0.6 perplexity). see Table T] (rows and 7), and Figure (left). Trying smaller numbers of modules, 16 and 32, and plotting all the results in Figure 3 (left) see that for small numbers of modules Hash and Switch perform similarly, but the gap grows larger as the number of modules increases. For small numbers of modules, we hypothesize that learning route, as Switch does, would be more important to be performant with those choices, but with larger numbers of modules many routing choices could work. Hence, Hash layers work well in that setting, and learning route becomes less important.
Hash layer position We also experiment to find the best position layer-wise for the sparse routing to take place. In In/Figure (right) we plot perplexity for the 64 module Hash Layer, placing different layers of the decoder. We find that later layers perform better, but even the worst performing choice (layer 1) is still performing well compared to other baselines: as good as Switch Transformers using later layers in fact. We note that analysis of BASE Layers [10] showed a similar trend that later layers work well. Hypothesizing that conditional compute gives the ability to make fine-grained specializations, it follows that it is worth making those distinctions after more obvious features have first been extracted. We will return to this argument in later experiments.

---

Table 3: Different Hash Layering Methods on pushshift.io Reddit.
Model
Hashing Type
Valid PPL
Test PPL
Baseline Transformer
24.90
24.96
Hush Layer 1x64
Balanced ussignment
23.16
23.23
Hash Layer 1x64
Fixed random assignment
23.22
23.27
Hash Layer 1x64
Token clustering (using Bascline Transformer)
23.90
23.99
Hush Layer Lx64
Dispersed Hash (within token clusters)
23.17
23.22
Hash Layer 1x64
Hash on position
25.07
25.14
Hash Layer 1x64
Bigrams
24.19
24.28
Hash Layer 1x64
Previous token
24.16
24.22
Hash Layer Lx64
Future token predictions (using Transformer Buseline)
25.02
25,09
Hash Layer 1x64
Future token (Oracle)
1.97
1.97
Hash Layer 5x16
Same hash layer (balance assignment) Different Hash per layer
23.74
23.81
Hash Layer 5x16
23.21
23.27
Random Hash Bucket Frequency
Balanced Hash Bucket Frequency
Bucket Frequency
Bucket Frequency
Ideal balance
Ideal balance
Hash Bucket
Hash (sorted)
Figure 4: Relative frequency for 64 expert modules with Random Hash (left) and Balanced Hash
(right). The Zipfian distribution makes perfect balance impossible, but Balanced Hash is closer.
Multi-layer routing We evaluate placing sparse routing every other layer, 16 different modules each in Table (rows 8-10). Switch and Hash perform similarly in this setting, with Switch outperforming with the optimal choice of 0.1 load balancing (23.00 vs. 23,21), and the same performance (23.19) for balancing parameter 0.01. Given the results of Figure 3 (left), the small number of modules in this case may make performance close.
Downstream fine-tuning We compare several of the pushshift.io Reddit models for the goal of fine-tuning on downstream tasks. We experiment with either fine-tuning the whole model, or freezing some parts of the model during fine-tuning, as well as altering the load balancing for Switch at fine-tune time. Results are given in Apppeddix A. We find that the fine-tune results generally agree with the original performance on the pre-training pushshift.io Reddit task, and the order of methods retained. Hash outperforms Switch slightly, both outperform the Bascline model, and the larger dense models perform better, as expected. Freezing parts of the model generally hurts fine-tuning, unless the part frozen the sparse part of the model. appears that case just fine-tuning the dense parts of the model is sufficient for good performance. Only tuning the sparse part of the model, on the other hand, hurts performance, perhaps because the majority of the capacity of the model lies there.
5.3.2 Hash Function Analysis
We evaluate the different choices of hashing function detailed in ssubsection 3.1 The overall results are given in Table 3 on the pushshift.io Reddit dataset using a 64 module Hash Layer.
Random and Balanced Hash Functions We find that fixed random assignment (row 3) and balanced assignment (row 2) perform similarly well in terms of perplexity (23.22 vs. 23.16 valid perplexity). However, balanced assignment, as its name suggests, is more balanced, see Figure 4 which may render more efficient in terms of distributed training schemes.
Interestingly, using cluster based hashes ("Token clustering", row 4) performs clearly worse than randomized hashes (23.90 vs. 23.22). We hypothesize that if the goal of conditional computation is make fine distinctions, then those distinctions are more likely to appear between tokens within the same cluster, hence they should be in different hashes (parts of the compute graph), not the same one. We provide partial evidence for this by hashing within token clusters instead ( ("Dispersed Hash", row 5), which restores the performance be similar to random
Clustering Hash Functions
7

---

Table 4: Comparison of Models on Wikitext-103. We compare a baseline dense Transformer to our sparse models, which have sparse layer with 16 modules (1x16). We show results with two different dictionaries, the BB [4] BPE dictionary (8008 tokens) and the standard one for the task (267,739 tokens). As these are different dictionaries, perplexities are not comparable across columns.
Std. Dict
BB Dict Valid PPL
Model
Configuration
Valid PPL
Baseline Transformer
layers=8, d=512, D=512
33.09
12.58
Switch Transformer Hash Layer
layers=8, modules=1x16, load_bal-d0 bal=0.1 layers=8, modulcs=1x16
31.76
11.67
32.32
11.58
hashes (23.17 vs. 23.22). We note that Icarn-to-route methods such as Switch Transformers and BASE use simple functions of the hidden state to perform routing, which generally provide clustered expert modules [0], which could hence be disadvantage for those methods.
Position-based Hash Function We conduct experiments hashing based on sequence position only. We consider this experiment as a samity check, we did not expect choosing conditional compute based on position in the output sequence help. Indeed, it turns out that this is no better than the dense Transformer baseline. Thus it appears that routing based on input content is much more important.
Bigram Hash Function Hashing based on the last two tokens (bigrams) performs worse than using only the last token (24.19 vs. 23.16 16). We hypothesize there are two reasons for this: (1) first, the last token clearly the most pertinent, and bigrams add less relevant feature; (2) this creates too many hashes, which performs less well. Subsequent experiments will help test these claims.
Previous Token Hashing Hashing based on the previous token is clearly worse than using the current token (24.16 vs. 23.16), and gives similar performance to using bigrams, helping confirm the first part of our above bigram hypothesis.
Dictionary size We perform experiments on Wikitext- 103 in two settings: using the given dictionary of 267k tokens, using the 8k dictionary we use in our pushshift.io Reddit experiments, following 34.. The results, comparing to Switch and a baseline Transformer, are given in TTall 4) We find that Hash works well for the small dictionary, slightly outperforming Switch. However, on the larger dictionary, it performs worse than Switch. As this is the same data but just the tokenization has changed conclude the hashing induced from the smaller dictionary is easier to learn from, helping confirm the second part of our above bigram hypothesis.
Oracle Future Token Hashing We evaluate hashing using the oracle next token that is to be predicted. This yields a perplexity of 1.9. Using oracle information just to choose between modules is sufficient to essentially solve a task.
Predicted Future Token Hashing The last result poses the question: we can predict the next token. and hash based on that prediction instead - will it be better than hashing on the current token? We thus tried hashing using the Baseline Transformer to predict labels, yielding a perplexity of 25.02 -which does not actually beat the Baseline itself. It appears that the bias of the token predictions limits the ability of the sparse routing to improve.
Multi Multi-hashing We evaluate the multi-hashing technique described in subsectionn 3.2 Results given in Appendix A] comparing to Switch and standard hashing. Even though the same number of parameters is used in all cases, we see improvements for splitting the hash into 2, 4 different hashes compared to single hash, with steadily improving results for both 16 or 32 modules.
5.3.3 Switch Transformer Analysis
Switch load balancing We show the performance of Switch for different values of the load balancing parameter on pushshift.io Reddit in Appendix A Clearly the choice of parameter is important, with results varying over a perplexity point range.
Switch with Token-based Routing Given our analysis of oracle and predicted token hashing in subsection 5.3.2 we hypothesize that the hidden representations in layers of the Transformer, being biased towards the predictions of the model, may be suboptimal for routing. We therefore experiment with hybrid between Switch and Hash Layers: the sparse layer, instead of using hidden state as
8

---

Table 5: Multi-hashing experiments on pushshift.io Reddit. When multi multi-hashing, the same number of parameters is used, but the FFN weights are split and indexed into multiple hashes and then concatenated together for the forward step.
Model
Configuration
Params
Valid PPL
Test PPL
Switch Transformer
layers= 11 1.aduulessmmll _ad_bal=0.1
483M
23.79
23,84
Hash Layer
layers= 11 1.mdullesmmmmsl......
483M
23.58
23.65
MultiHash Layer
layers= 11 1ooduless11322.uusbeee..
483M
23.48
23,53
MultiHash Layer
layers= .mw.umms=1xx32.mmmme.. 4
483M
23.38
23,45
MultiHash Layer
layers= 1, L.l.clmmlls-1133322hhseeee
483M
23.28
23,34
Table 6: Switch Transformers with Token-Based Routing on pushshift.io Reddit. We compare
standard Switch which routes based on the hidden state to token feature-routing ("Token Switch').
Model
Configuration
Params
Valid PPL.
Test PPI.
Switch Transformer
layers= 1.w.ullemmmmeeee644444444 load bal=0.1
751M
23.65
23.73
Token Switch
layers= 1 load_ bal=0.1
751M
23.43
23.43
Switch Transformer
|ayers=||,modulesee Imodules=111222 load_bal=t)
1.28B
23.52
23.58
Token Switch
[.trs==|..odule=1112222 load_bal=O.,
1.28B
23.26
23.32
the Switch router input, we use the current token instead. To convert the token vector we use an extra lookup table, i.e., an extra set of learnable parameters that is the size of the dictionary. These parameters are independent of the hidden state and are only used by the router to learn the best route. Results are given in TTbll 6) We find this brings some small improvements to Switch for 64 and 128 modules on a single layer, affirming the usefulness of token-based routing.
5.3.4 Comparison to BASE Layers
We next compare BASE Layers. Using the BASE Layer code base, we implement Hash Layers in exactly the same setup, changing only the routing method, and leaving everything else fixed. Figure (right) shows results comparing Hash with BASE for 4.5B parameter models. Across the entire run, we see that Hash outperforms BASE at each training step. During early parts of training, Hash would presumably have advantage in being able specialize expert modules earlier, while BASE must learn membership for each of the expert modules. Later in training. BASE becomes mildly unstable presumably expert assignments shift, while Hash performance continues improve smoothly.
Additionally, to demonstrate Hash Layers remain performant when stacked, we trained a model with 3 Hash Layers (using random hashes), but fewer parameters per expert module the total parameters remained constant at 4.5B (see subsection B.2). We find that using multiple Hash Layers gives a small but consistent improvement, suggesting Hash Layers will be effective at more depth.
In addition to performance gains compared BASE, we also find that Hash Layers are more efficient in total computation. In particular, BASE requires two all-to-all communications: the first de-correlates batches order to make assignment balancing more stochastic, and the second routes states to their assigned expert. As Hash Layers use fixed, pre-computed assignments they avoid the de- correlation step. In practice, we find this gives an improvement of about 11% in podates-per-seconn As the number of expert layers increases, this difference will become more exaggerated.
6 Conclusion
We have introduced a simple and efficient approach to sparse models in the Transformers-for-NL.P setting based on hash layers. We showed on a variety of datasets and with analysis in various settings that this approach is highly competitive with existing methods such as Switch Transformers and BASE Layers, whilst being robust and far simpler requiring no extra learning parameters, assignment algorithm or changes to the objective function. Given that researchers typically have only one opportunity to train very large models, this makes our approach strong candidate for such runs. While our experiments scale up to 4.5B parameters, we do not reach the scales of large industrial works such as [8) and we hope to see future work conduct such experiments. Finally, given that our routing approach is learning free, our results perhaps suggest that none of the current approaches are routing partieularly well. We thus believe learning-to- tooiingg---ruteee should continue to be the study of future work, and consider our work a strong baseline for such research.

---

References
[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
[2] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-8)) 87, 1991.
[3] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilisstc language model. The journal of machine learning research, 3:1137-1155, 2003.
[4] Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts.
IEEE transactions neural networks and learning systems, 23(8)1177--113, 2012.
[5] David Eigen, Marc' Aurelio Ranzato, and llya Sutskever. Learning factored representations in deep mixture experts. arXiv preprint arXiv:1312.4314, 2013.
[6] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experts layer. arXiv preprint arXiv: 0.6538, 2017.
[7] Sam Gross, Marc' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 6865-6873, 2017.
[8] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021.
[9] Ronan Collobert, Yoshua Bengio, and Samy Bengio. Scaling large learning problems with hard parallel mixtures. International Journal of pattern recognition and artificial intelligence, 17 (03):349-365, 2003.
[10] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. arXiv preprint arXiv:2103. 16716, 2021.
[11] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
[13] Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2009.
[14] Andrei Z Broder and Anna Karlin. Multilevel adaptive hashing. In Proceedings of the first anmual ACM-SIAM symposium on Discrete algorithms, pages 43-53, 1990.
[15] Guillaume Lample, Alexandre Sablayrolles, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. In Advances Neural Information Processing Systems, pages 8548-68 8548-8559, 2019.
[16] Reinhard Kneser and Hermann Ney. Improved backing-off for m-gram language modeling. In 1995 international conference on acoustics, speech, and signal processing, volume 1, pages 181-184. IEEE, 1995.
[17] Tomák Mikolov, Martin Karafíát, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. Recur- rent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
[18] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiy preprint 11Xiv:1112.3005, 2013.
10

---

[19] Graham Neubig and Chris Dyer. Generalizing and hybridizing count-based and neural language models, arXiv preprint arXiv: 1606.00499 2016.
[20] Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv: MoXKv::412.2007, 2014.
[21] Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, al. Efficient softmax approximation for gpus. In International Conference on Machine Learning, pages 1302-1310. PMLR, 2017.
[22] Anton Bakhtin, Arthur Szlam, Marc' Aurelio Ranzato, and Edouard Grave. Lightweight adaptive mixture of neural and n-gram language models. arXiv preprint Frxiiv/804.07700 2018.
[23] Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning, pages 1113-1120, 2009.
[24] Bing Bai, Jason Weston, David Grangier, Ronan Collobert, Kunihiko Sadamasa, Yanjun Qi, Olivier Chapelle, and Kilian Weinberger. Learning to rank with (a lot of) word features. Information retrieval, 13(3):291-3144 2010.
[25] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294. PMLR, 2015.
[26] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content- based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53 68, 2021.
[27] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint 1:XXi:2001.04455 2020.
[28] Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learming for NLP, pages 164-174, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[29]
Baareeemmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium, October- November 2018. Association for Computational Linguistics.
[30] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. CTRL: conditional transformer language model for controllable generation. arXiv preprini arXiv: 1909:v990900088 05858, 2019.
[31] Kurt Shuster, Da Ju. Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents, 2019.
[32] Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In Proceedings of the International Conference on Learning Representations, 2019.
[33] Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. arXiv preprint arXiv:2001.0844 2020.
[34] Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004. 3637,, 2020.
[35] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1672v 2019.

---

[36] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek. Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: 191 1,00116, 2019.
[37] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0000 2016.
[38] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint 1rXiv:1904.01038, 2019.
12


Reviews:
[{"review_id": "u6qfiy4Xu2X", "reviewer": "Reviewer_S1J2", "summary": "The paper studies a simple hash-based approach to route input tokens in sparsely-gated mixture-of-experts models. The proposed approach for routing is a static hash-based approach, input tokens are hashed to 1 of the several experts in the MoE layer without the need for any learnable routing network parameters  which are typically used in MoE layers. This approach is compared against Switch Transformers and BASE layers [1, 2] on a generative dialogue task on the Pushshift.io dataset, language modeling on Roberta+cc100 and Wikitext-103 datasets and on fine-tuning on the BST dialogue tasks. On all these tasks hash-based routing is either competitive with or shows slight improvements over the previous (more complex) routing approaches.\n\nIn addition to the comparison against Switch and BASE layers, the paper also conducts some analysis on:\n1. Several approaches for hash-based routing, and finds that simple balanced or random hashing performs better than approaches that take into account the preceding tokens or predicted future tokens for routing. \n2. The effect of the number of experts (or the amount of sparsity) on the performance of Switch layers and Hash layers, and demonstrate that learnt routing layers are more competitive at lower levels of sparsity.\n\nReferences:\n\n[1] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus et al.\n\n[2] BASE Layers: Simplifying Training of Large, Sparse Models, Lewis et al.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The paper makes an original and valuable contribution, demonstrating that a simple hashing based approach can be competitive with more computationally expensive and complex routing approaches for MoE layers. \n2. Provides a strong and simple baseline for future work on routing in MoEs, while highlighting a major weakness of MoE models being studied in literature.\n3. Clearly written, easy to understand and contains several interesting ablations and analysis.\n\nCons:\n1. The paper is limited in scope to a particular type of architecture and task. The analysis focuses on MoE layers in either decoder-only LMs, or encoder-decoder models with very light-weight encoders. The increased capacity is also limited to layers with causal attention, which might affect the need for more complex routing strategies. It's not clear how well hash layers or non-learned routing strategies would perform on tasks where MoE layers have shown major improvements in recent work, like NLU tasks with span-mask pre-training [1] or multilingual MT [2,3].\n\nOther questions/comments for authors:\n\n1. For figure 3b, is there a corresponding plot with position vs performance for switch layers as well? Are the trends any different from hash layers?\n2. Line 200: \"we hypothesize ... work\": Or perhaps this indicates that routing functions are easier to optimize with top-k style gating at lower levels of sparsity / fewer experts?\n3. Line 205: \"We find ... work well\" this finding is also similar to that observed for PKM. [4]\n\nReferences:\n\n[1] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus et al.\n\n[2] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, Lepikhin et al.\n\n[3] Exploring Routing Strategies for Multilingual Mixture-of-Experts Models, Kudugunta et al.\n\n[4] Large Memory Layers with Product Keys, Lample et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "7BOle3OswO5", "reviewer": "Reviewer_eK9M", "summary": "Paper studies the routing problem in sparsely gated mixture of experts model for text. The central contribution of the paper is experimenting the efficacy of deterministic routing strategies that are not learned over the course of training but fixated at the beginning of training by utilizing hash functions. Various hash functions are experimented on four different tasks, and compared against prior art methods (BASE layers and Switch Transformer).\n\n- Overall well written paper with a focused contribution\n- Conclusion has a more compelling and striking story which would be great to hear early on: learning free approaches for routing are quite competitive hinting that the learned routing algorithms may not be doing what we expect them to be doing. \n- Paper suggests the proposed algorithm should be considered as a baseline to other methods, and given the results I cannot disagree with the proposal. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Uses a hash function on the \"tokens\" to decide the expert being routed throughout the network, where each expert being represented by a bucket in the hash map. \n\nVarious different types of hashing functions experimented/proposed: random, balanced, bi-gram hashing, previous-token hash or hashing based on position, clustered or dispersed hashes etc.\n\n**Questions**\n- What is the effect of vocabulary size, vocab composition to the final quality?\n- If we were to switch to character or byte level vocabularies, how could we apply the proposed algorithm?\n- What would be the extension you could propose if we were to extend this to vision? While you criticize the learned routing methods, they are also widely applicable to other domains. Could you please elaborate on this. \n- If we were to use multiple moe layers, then all the tokens are always going to be sharing the same expert for each layer right?\n Multihash layers, --> caveat, you may end up activating the entire parameter space, will have a downside on the inference, but also each token will have the chance to touch all the experts, slowing down the training (practically making it a fully dense model if $N$ is equal to the number of experts, and each hashing method assigns a token to a separate expert). \n\n**Typos**\n- line 74: advanced -> advance", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "1j3Z2-LO0ly", "reviewer": "Reviewer_sntW", "summary": "The authors propose using hashing layer to replace the routing mechanism in the methods of mixture-of-expert such as Switch\nTransformers and BASE Layers. The main benefit of the proposed method is that no routing parameters need to learn and achieve quite good performance on the language modeling tasks. The authors also have a wide exploration of different hashing mechanisms, such as random hash, balanced hash, bigram hash, clustered hash, etc. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed method is interesting and easy to use. It's promising for the community to further boost the performance of super-large models. The authors have also done a solid comparison between the switch-transformer and wider/deeper models. However, my major concern is why this method can be better than switch-transformer which theoretically can learn more knowledge to route than simple hashing. I think there are some methods to make a better hashing function, such as balanced hashing. Could these methods also apply to switch-transformer to further boost performance?\n\nOther comments:\n1. The proposed model is only tested on language modeling tasks. I am curious about the performance of the downstream tasks, such as GLUE after pertaining with cc100 corpus.\n2. The hashing function has been adopted to build sparse attention in Transformer by the works of Reformer and Routing-Transformer. The authors adopt a similar method but focus on the MLP layer in SwitchTransformer. I would say this is new and interesting, but not very novel.\n3. The authors have proposed many different hashing methods. However, the simplest hashing method still works almost the best. The relatively new methods don't work well. Most experiments on other datasets are required if the authors want to claim novelty on different hashing functions. Currently, it is only evaluated on Reddit which is also not a popular LM data.\n\nQuestions:\n1. Are all layers using hashing layers? What are the detailed structures of using hashing layer?\n2. Any challenges when training hashing layers? Is it easy or stable to start training?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Hasn't proved to be a general framework for different NLP tasks, only testing on language modeling tasks.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "1SjGVOdRHEj", "reviewer": "Reviewer_UiAT", "summary": "**Summary And Contributions:**\n\nThis paper is interested in routing strategy in MoE Transformers in NLP where each tokens of the sequence are “routed” to a specific FFN (also called expert). The authors argue that the difficulty of learning parametrized routing strategies (especially as the number of experts grows) calls for simpler baselines. The experiments show that a simple hashing baseline (pre-defining the expert routing for each token based on some token-frequency rules) is an easy-to-implement, robust and strong baseline that is either on par or outperforming learned routing strategies. Most of the comparisons are done with the perplexity on a variety of datasets and the authors also analyze a series of different hashing strategies.\n\n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n- The paper is well organized and written, and it’s easy to follow.\n- Experiments are quite exhaustive and convincing, giving to a strong and easy to implement baseline for future research on Transformer MoE\n\n**Weaknesses:**\n\n- While the differences in perplexity are noteworthy, it is less convincing that there is a difference in downstream fine-tuned performance. It is not straightforward that further gains in perplexity are interesting, especially from the perspective of memorization (that the authors briefly touched upon in the appendix).\n- It would be valuable to extend the comparisons to more fine-tuning setups which would bring a more useful signal from the perspective of applications (compared to perplexity), especially since it is not clear that gains in perplexity necessarily translate to gains in downstream performance (fine-tuned or zero-shot).\n\n**Questions:**\n\n- Training very large dense language models (more than a billion parameters) is known to be unstable (the loss can diverge unexpectedly during the training). Did you notice any differences in training stability in hashing vs learned-routing strategies?\n- How would this approach be extended to multilingual setups? Any thoughts on how to scale the number of experts? The dictionary size?\n- Any thoughts on the memorization abilities of the obtained models? Is it possible that improving the routing makes it easier for the model to memorize random facts and training instances?\n- Is the balancing additional loss used during fine-tuning too?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a surprising result that a deterministic routing strategy for sparse MoE models can yield competitive results for language tasks compared to learnable approaches. The experiments conducted are extensive and well thought. Regarding using perplexity as the only comparison metric, I suggest the authors either consider additional metrics (e.g., qualitative/human evaluation) or using multiple perplexity values, e.g., in Figure 2 or 3, to make a conclusion. The reason is a single small difference of < 1 ppl point (at the range of ppl > 20) might not be conclusive. Nevertheless, I agree with the reviewers that this paper is well written and the proposed approach serves as a simple and strong baseline. Hence, I recommend Accept.
Decision: Accept (Spotlight)