Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose to equip transformer with stochastic experts ( i.e., a number of FFN layers in parallel)  to boost model capacity without increasing much computation. Moreover, they propose a consistency regularization between a pair of experts in the training, which can alleviate performance deterioration of random expert selection in the inference.

main_review: Strength
1 The paper is well written with good structure
2 The method is concise and novel to some degree.

Weakness
1 The number of experts is limited to 2. Indeed it would maintain the computation load. But the power of experts may not be exhibited sufficiently. It would be great to investigate the performance by changing the number of experts.
2 I am wondering if this mechanism can be used in the vision transformers? 
summary_of_the_review: Basically, it is an OK paper for me. The novelty and experiments are good to show effectiveness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new routing mechanism for sparse models in the context of language tasks. Rather than learning a parametric router that learns how to assign tokens to experts, the proposed algorithm (THOR), randomly selects two experts per mini batch, and applies those experts independently to every input. A consistency loss is used to force experts to provide similar predictions. A number of experiments are provided suggesting the algorithm outperforms previous works.
main_review: This paper studies sparse models in the context of deep learning: neural networks that only activate a subset of all parameters depending on the input. This is a field that is gaining lots of attention recently, as models have grown large, expensive and inference time has increased accordingly.

So far, most works [1, 2, 3] learn a router (usually a linear map + softmax + maybe noise) to assign tokens to experts. This approach has some downsides (lack of differentiability, expert collapse that requires additional auxiliary losses, etc). The paper proposes a new approach that simply samples the experts at random --and enforces some coherence among them. 

Section 3 presents some experiments related to standard gating mechanisms, Section 4 presents the algorithm, and Section 5 shows the experimental results.

I have concerns in the three sections.

**Section 3.**

The number of experts is 2 --and I assume the model selects 1--, which seems maybe too few to me, in order to see generalizable effects.

If I understand Figures 1-3 correctly, two MoE models were trained (MoE(dec) and MoE(tok)). The first one collapses (only Expert 2 ends up being used) in Figure 2, and the second one seems to reach an equilibrium, in Figure 3. I think the interpretation provided by the paper for Figure 2 is generally wrong and misleading. The fact that both experts are used equally often (i.e. load ~ 0.5 in both cases), and the fact that the *average* routing confidence is 0.5 does *not* imply random routing is happening, as suggested in the paper. If uniformly random routing was happening, we'd see this. However, there are good routing schemes that would lead to this averaged outcome too. To give a simple example, suppose we have a dataset containing sentences about sports and about animals, half and half. The router may send all sports tokens to Expert 1 with weight 1.0, and all animal tokens to Expert 2 with weight 1.0. The average gating weight is 0.5 for each expert, and they get half the tokens. The routing, however, is very far from random.

I'd not agree with the general statement (and, more importantly, I don't think this paper contains enough evidence to back it up): "The widely-used routing method based on the gating mechanism does not work better than randomly routing inputs to experts".

There are published experiments (see [3], Figure 27, layer 21) that replace a trained router with a random router and, as expected, performance goes down. Of course, this does not imply that if routing is random while training, final performance will be worse (but we'll end up with an ensemble of similar experts). Moreover, there's evidence that, under learned routing, experts specialize (even in a human understandable way), see [3] Figure 7. These facts should probably be acknowledged in the paper. Also, note that even fixed routing [4] is fundamentally different from random routing, it still allows for expert specialization if done well.

**Section 4.**

The proposed algorithm seems to weaken the sparse models advantage, that is, having lots of capacity which the model can specialize per input, while keeping cost kind of constant. By randomly selecting experts in every batch (+ explicitly adding a consistency loss), we are forcing all experts to learn the same thing --and how to deal with all possible inputs/tokens--, which directly counteracts the sparse models core idea. In other words, we are training a computationally-cheap ensemble. This can still be a very good model, nonetheless; but I'd be surprised if it beats a properly tuned MoE transformer (more on this in the next section). As mentioned before, there's evidence that experts specialize [3].

Also, I was wondering if there's concrete evidence to support this claim in page 5: "Although these experts are learned to make consistent predictions, they converge to different (local) optima given the randomness introduced in training [...]". I'd assume there's such type of statements in the ensemble literature.

**Section 5.**

I have some fundamental concerns with the experimental section. If my understanding of the experiments is correct, the comparison between THOR and the Switch Transformer [2] is not fair. Basically, the number of epochs is fixed (or say, datapoints seen by the models during training, batch size * number of steps). However, THOR applies two times the backbone network to each datapoint, leading to a massive mismatch in terms of total FLOPs or runtime (with equivalent hardware).

In order to have a meaningful comparison, all the tables in the paper should include two additional columns: total training FLOPs and total runtime. This will unhide compute differences across models.

Switch uses k=1 (i.e. one expert selected per token). Note that even for k=2 (as used in [1, 3]) previous sparse models would be cheaper (and significantly so) than THOR, as all the common --non-experts-- layers are only applied once, regardless of k. One baseline that must be in the paper tables is a Switch model that runs for a FLOPs- or runtime-matched amount of steps, that is, it probably runs for around twice as many steps. I'm open to change my score/reviews if those experiments are added, and are still favorable to THOR.


Another couple of comments regarding experiments. First, for non-ensemble methods to shine (i.e. Switch), probably more than 2/4 experts would help. Using so few experts probably helps mitigate the lack of specialization effect, and benefits the ensemble-like THOR approach. 

Second, I don't understand the following sentence at the end of section 5.3: "Similar to what is observed in low-resource translation, the Switch Transformer does not outperform the vanilla Transformer". Looking at Table 2, it seems BLEU for Switch is (quite?) better than for Vaswani et al (i.e. "vanilla" Transformer)?



[1] - GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

[2] - Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

[3] - Scaling Vision with Sparse Mixture of Experts

[4] - Hash Layers For Large Sparse Models
summary_of_the_review: Experiments devote significantly more compute to THOR than to Switch. Accordingly, it is hard to interpret the results.

------

I've raised my score given the authors rebuttal and updated results (3 --> 5).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed to use randomly selected experts for Mixture-of-Experts models instead of gating function based selection methods. To avoid large performance variance of the random selection during inference, the paper proposed to add a consistency regularization which drives the similarity between different experts. The experimental results show its superior performance over existing MoE models such as switch transformers. 
main_review: The paper is easy to follow and the proposed methods are simple and novel. The experiments and ablation study are extensively conducted. The results look promising and the method should be easy to be adopted. 

I have a few questions and comments.

1. In Table 2, Transformer-base (Vaswani et al.,2017) architecture is used for Europarl datasets, while a smaller model is used on IWSLT datasets. I am wondering why not apply both model architecture to both datasets. 

2. In Table 2, it seems to me that the size of Switch transformer is larger than transformer-base but they have same inference flops? However, I also see this statement `The results confirm that SAMs do not outperform densely activated models with similar model sizes.` in Sec. 5.2. So I am wondering if the model size of switch transformer is same as the size of transformer-based model in Table 2. 

3. In page 5, the paper states `During inference, we can also select a pair of experts to activate at each layer for each input, similar to that in training.`. I thought only one expert is activated during inference to gain the inference speed. Also, if two experts are selected, how the final predictions are selected? Is the average embedding used? 

4. The paper mostly compares the inference FLOPS. I am wondering if the authors can provide the real inference time for transformer-base model and the THOR model (with same FLOPS) in both batch inference mode and real time inference mode. 
summary_of_the_review: The paper presented a simple and novel MoE method. The experiments and ablation study are extensively conducted. The results look quite promising and the method should be easy to be adopted. I vote for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes THoR, an approach towards training MoE-like models (called SAMs in the paper) that have multiple internal experts which are chosen in a discrete fashion. Unlike other common methods, which generally incorporate a form of router which maps input to an  specific router choice, THoR makes random, global, per-minibatch decisions on experts, and includes a cross-expert regularization term to help align experts to each other. This paper reports that THoR outperforms other methods, including the recent Switch Transformer MoE-style model, on several classes of multilingual translation tasks.
main_review: The paper is very well written and clear, proposing the THoR method, including ablations on aspects of its design, and performing three classes of evaluations, comparing to other methods, reporting that THoR outperforms all other methods and establishes at least two state-of-the-art results on WMT. Fundamentally, I am more than willing to believe the key results which show that THoR outperforms Switch Transformer. 

However, MoEs are complex models, for which it is very easy to subtly apply an incorrect comparison, or to evaluate improperly. There are several results in the paper which claim that Switch Transformer underperforms or just equals the performance of a dense, standard transformer. I find this difficult to reconcile with the results of the Switch Transformer paper [1], which not only claims that Switch outperforms dense models, but has several results (Figure 1A, and Appendix D) specifically saying increasing the number of experts from 1 to larger numbers monotonically improves performance. A general result which has been mirrored in other recent MoE-like papers (Figure 3 in [2], Figure 6 in [3], Figure 4 in [4]). This inconsistency makes me skeptical of the hierarchy of methods suggested by this paper.

Fundamentally, the main request of this rebuttal is that the authors address this inconsistency between their MoE/Switch results and those reported by others. I have three concrete asks:

* Are methods FLOP-matched when being compared, or step-matched? My read of sections 4 and 5 are that methods are step-matched, but this seems substantially unfair since THoR (and the other regularization methods) require twice the compute per step. I believe a proper comparison across all the datasets and evaluators should be at a fixed number of FLOPs (i.e., with Switch/Dense running for double the number of steps). In line with this request, the inclusion of validation curves showing the performance of THoR, Switch and Dense Transformers as training progresses would be very valuable.

* Are the authors able to reproduce Figure 1A from [1]? An exact reproduction isn't necessary, differences in model details or dataset are fine, but it would be extremely valuable to know whether or not the author's implementation of Switch mirrors the result that increasing the number of experts should increase the performance of the network. If yes, what is your explanation for Switch not outperforming a Dense transformer on some evaluations? If no, do you have an explanation for the difference? Does THoR exhibit improvement with more experts?

*  This paper uses just 2 or 4 experts for the majority of its experiments. While Switch does already claims performance improvements with so few experts, it is also much smaller than the normal regime which is tested, with number of experts in the 16-128 regime being standard [1, 2, 3, 4, 5]. What do the initial experiments, as well as the later Switch vs THoR evaluation comparisons, look like with a larger number of experts? Does the comparison hold up?

If the authors address these three points and either explain or reconcile the difference in their reported Switch performance with those of other papers, then I would have no problem recommending an accept (8). In the current form, I feel this paper is borderline due to these unanswered questions.




Additionally, I have several smaller nits and comments:

* **[General]** There are several recent methods which claim to improve over switch ([2] and [5] come to mind). I don't necessarily think you need to compare against them, but as the claimed performance improvement is high, it seems important to acknowledge.

* **[General]** The authors prefer the term SAM, but this is a very general term (there are lots of different sparsely-activated models out there) and this term doesn't seem standard in the literature. Maybe something more specific would be more apt?

* **[Section 2, "Transformer"]** "the model" should be replaced with "our model", not all transformers are encoder/decoder models.

* **[Section 2, "Sparsely Activated Models"]** A nit: Hash Layers don't help solve the load balance issue, they completely obviate the need for a routing layer (also, from Section 4, worth noting that similar to THoR, Hash layers also don't need to introduce new parameters).

* **[Appendix A.2]** I think the conclusions you're reaching are too certain. While it is true that random routing is a possibility based on these results, it is not a foregone conclusion (what manifests as a slight preference in the router might be key). A conclusive experiment you could run would be to replace the routing decisions *only at evaluation time* by random decisions, and see how much the evaluation performance of the networks are impacted.

* **[Section 4]** The idea that THoR proposes, that experts should be similar and consistency-regularized towards each other, has a very reasonable interpretation. However, it is directly in opposition to the standard interpretation (that experts should be diverse). I think it would be good to discuss this difference more.

* **[Section 4, "different experts on different GPUs in parallel"]** Can you elaborate on this? I understand expert parallelism, but a key aspect of expert parallelism applied to most MoE models is the expectation that for any training minibatch, we expect to be using all experts more or less equally (which means we can share the compute equally across a bunch of different devices, each with their own experts). In THoR, for any given step we're fixing the number of experts we use to two, so I don't see how this can be effectively parallelized across more than 2 devices (without replication).

* **[Section 5.4, Table 3]** Having numbers from a dense transformer would be useful here.

* **[Section 5.5, Figure 8]** Switch should have input jitter disabled at evaluation time (Figure 15 in [1]). This should make the network at eval deterministic. What is the explanation for the variance seen in this figure?


[1] *Switch Transformer* https://arxiv.org/abs/2101.03961

[2] *Hash layers* https://arxiv.org/abs/2106.04426

[3] *GShard* https://arxiv.org/abs/2006.16668

[4] https://arxiv.org/abs/2109.10465

[5] *BASE Layers* https://arxiv.org/abs/2103.16716
summary_of_the_review: The paper is well written and the proposed ideal seems novel. But my inclination to recommend accept (8) is tainted by some doubts about the primary comparison given with Switch Transformer.  I've outlined three key questions/additions which could assuage these doubts.
review_title: null

Meta-review: In this paper, the authors introduce a simple mixture-of-experts model, by greatly simplifying the routing mechanism: experts are randomly activated both at train and inference time. A consistency loss function is added for training the proposed models, enforcing all experts to make consistent predictions. The proposed method, called THOR, is evaluated on machine translation tasks, including multi-lingual MT, and outperforms the recently proposed Switch Transformer MoE.

The reviews note that the paper is well written and easy to follow, and that the proposed method is simple. While the results look promising, the reviewers also raised concerns regarding comparisons to previous work, some of which were addressed in the rebuttal. Finally, a reviewer raised the concern that this method is related to ensembles, which work well for machine translation, but are not discussed or compared to. For these reasons, I believe that the paper is borderline, leaning toward acceptance.
Decision: Accept (Poster)