Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In the context of neural networks with Mixture of Experts (MoE) layers, the paper proposes to decrease the number of activated experts per each input as training progresses, by means of decreasing the temperature in the softmax that is typically used to weight the contribution of the different experts in the MoE layers. The proposed approach is compared with fully Dense MoEs and the recent Switch-MoE baselines in a language modeling task.
main_review: The paper proposes a simple and easy to implement idea. However, the experiments do not clearly show that Dense-To-Sparse (DTS) is better in terms of accuracy-compute trade-offs than fully sparse baselines (such as Switch-Moe).

In particular, although Figure 3b should show this (FLOP vs. Accuracy), but the lowest perplexity is achieved by Switch-MoE with $8 \cdot 10^{33}$ FLOP and DTS-MoE only matches Switch-MoE at around $6.5 \cdot 10^{13}$ FLOP. Most importantly, FLOP is not very well correlated with actual training time (nor energy consumption), since if some sort of "masking" is used to implement DTS, rather than true sparsity, some compute is wasted to compute the experts outputs that are then masked.

To this reviewer, it is not clear if masking or some thresholding was used to evaluate DTS. Figure 1 depicts a "threshold", which could mean that DTS avoid computing the output of some experts (the ones whose weight falls below the threshold), and that the FLOPs depicted in Figure 3b and thorugh the rest of the paper are more correlated with actual runtime. However, this "threshold" is not mentioned anywhere else in the paper, so it's very likely that when the temperature is very low (i.e. only 1 of the expert's weights is effectively non-zero) all the experts are being actually used. In any case, notice that even if the reported FLOPs correlate well with runtime, the observation regarding Switch-MoE is still true.

Moreover, many of the experiments do not report results on a test set, not even a validation set. Figures 3 (arguably the main plot) and 5 report *training* perplexity, rather than the perplexity on a separate validation or test set. However, Figure 4 and 6 do report results on the validation set. It's unclear to this reviewer what's the criterion used to report results on one set or the other. In any case, validation perplexity, or even better test perplexity, should have been used through all the paper (training measuraments are relevant sometimes, but not in the context of this paper).

There are some results from the "effect of balanced loads" analysis (Section 4.2.4) which are also not clear to this reviewer, as well. In particular, Figure 6a shows that the use of balanced loads adds an overhead of approximately $5 \cdot 10^{11}$ FLOPs in the openwebtext dataset. Why? Why only in this task but not in the Wikipedia task? In addition, notice that one of the main practical reasons to use a balanced load (e.g. in Switch-MoE) is to efficiently use all available compute, which is not considered in the analysis presented.

Minor typos and comments:

- Abstract: "**an** neegative impact" -> "**a** negative impact".
- Page 4: "Here we train a 24-layer transformer decoder model, with **and** every other FFN" -> "Here we train a 24-layer transformer decoder model, with every other FFN".
- Section 4.2.3: There seems to be notes left from the draft of the paper. "Need We analyze it in token-level's weights?".
- Section 4.2.4: "We directly load**s**" -> "We directly load".
summary_of_the_review: The main claim of the paper (the current approach of jointly training experts and the sparse gates introduce a negative impact on model accuracy) is not well supported by the experiments. In addition to that, many of the experiments report training measurements rather than test or validation measurements. Comparing training perplexity is not enough to show the superiority of the proposed approach. Some of the details of the implementation are not clear to this reviewer and affect the FLOP vs. Accuracy comparison. Actual runtime comparison (on a particual hardware architecture) would be much better.

Given all these issues, I recommend to not accept the paper. 


**Update after rebuttal**
The authors have addressed many of my concerns during the rebuttal period satisfactorily. My concern about the claimed "speed-up" is still present, but the authors have stated why they decided to use FLOPs rather than actual runtime on a given hardware and implementation. Based on all this, I'm slightly increasing my score for this submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studied the problem of training the gating network for the mixture-of-experts based model architectures. To make the gating network training more stable and robust, the authors proposed a dense-to-sparse gating training algorithm that uses Gumbel noise and temperature tuning. The authors conducted experiments on large NLP datasets.
main_review: Strengths
The proposed scheme to train dense gating then making it sparser makes sense and could potentially stabilize and improve the training of gating networks. 

The authors conducted extensive experiments to evaluate both the effectiveness and efficiency of the proposed algorithm.

Weaknesses.
Stabilizing the training of the sparse gating network has been the main challenge of the MoE based model architectures, there exists some research with similar ideas trying to make the training of MoE more robust. 
1. This paper uses gumbel noise to stabilize the learning of softmax gating. (https://arxiv.org/abs/1910.04915)
2. This paper uses a binary gating with similar noise and a l0 regularizer to control the sparsity. (https://ojs.aaai.org/index.php/AAAI/article/view/3788)
3. This paper uses a binary encoding and diffienable operators to smooth the learning of sparse gating, with initializing to be dense and later converging to sparse. (https://arxiv.org/abs/2106.03760)

Given these existing literatures, I feel like the novelty of this paper is quite limited. 

In the meantime, since the sparsity of the gating is controlled by the annealing of temperature of the softmax, only Top-1 gating is supported, this also limits the capacity flexibility of the MoE based morel architecture. 

summary_of_the_review: Overall, I think the training scheme from dense to sparse for sparse MoE gating makes sense, and this is supported by experiments in the paper. However, I also think the technical novelty of this paper is limited given existing work with similar ideas on improving the gating network training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes the Dense-to-Sparse Gate or DTS-Gate. It’s a simple idea of starting training with soft (continuous) routing to all experts and then gradually reducing to hard (discrete) routing. The authors claim that this improves the quality of traditional approaches. The work proposes the novel idea and details how to schedule the sparsity adjustment. The authors suggest this improves traditional approaches by 5% FLOPs-efficiency.
main_review: Notes:
* The idea is an interesting one. If the quality of sparse models can be significantly improved through using more compute only early on in training, this could become a widely adopted technique.
* Reasonably well-written paper.
* The authors use a reasonable baseline of a 24-layer MoE Transformer, 16-experts for every other layer.
* Visualization of the model dynamics, specifically gate weights distribution, is a nice addition beyond plots and tables of numbers.

Improvements:
* I suggest a different graphic for the “experts” in Figure 1. Our field already anthropomorphizes neural networks enough; I’d prefer to not see a little human with an etched brain as a stand-in for two matrix multiplications and a non-linearity.
* A style recommendation. Make each of your captions self-contained with description and result so that the reader can quickly scan your work figure-to-figure. Elaborate over the short phrases you’re using (e.g. “Figure 5: Different Temperature Scheduler”).
* On page 8, don’t use the abbreviations w/ and w/o.
* The significance of the gate values in Figure 7 will likely be lost to many readers. As stated above, please elaborate in your captions to describe what is being shown and why it’s important to substantiate your contribution.
* p8. “Though these tokens are spatial proximity” -> Fix.

Questions:
* Figure 3. Why are there periodic dips in the training perplexity curves? If this is from repeating the data in the same order, I’d recommend in the future that you shuffle data each epoch.
* Figure 4 demonstrates that MoE-DTS improves with more experts and expert-layers. How does this scaling compare to MoE-Switch?
* “experiments results show that the FLOPs-efficiency can be optimized into time-efficiency” -> Where was this shown? All the plots and data I see are for FLOPs, not wallclock time on a specific hardware.
* “It is amazing that DTS-Gate can learn to load balance without extra loss.” -> Was this verified to be a property only of DTS-Gate? Please confirm that naive approaches without load balancing do not learn this, or else, perhaps remove this. Also, as a nit, I’d avoid subjective claims like in using the word “amazing”.
summary_of_the_review: I’m positively disposed towards this work, however, my main concern is on the lack of a clear demonstration of the magnitude and the significance of the gains.

Please clearly show me the asymptotic quality is worse using vanilla approaches (e.g. Lepikhin et al., 2020, Fedus et al., 2021). As far as I can tell, the only substantiating plot for this is 3(b), but in my opinion, this doesn’t clearly showcase the improvement of the method. At the very least, these curves should be extended until MoE-DTS is conclusively better than MoE-Switch on a FLOPs basis. Further, though the idea is interesting, showing stronger empirical results than a 5% FLOPs-efficiency gain is likely needed to convince practitioners and researchers. If this is the case, I will improve from a 6->8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an algorithm, DTS-Gate, to warm-up experts in an MoE learning setup. Rather than applying standard top-k selection per input, the algorithm initially applies dense routing (i.e. k = total number of experts), to avoid quick collapse, and gradually relaxes it to become sparser (and cheaper). Experiments and ablations in language tasks are provided to support the advantages of the algorithm.
main_review: This paper tackles an interesting aspect of conditional computation: in the context of MoEs, how do we start training from scratch when individual experts can easily break down and collapse?

The paper provides an intuitive answer: we can start by training all experts together (i.e. as a dense model), and relax this over time, hopefully, once all individual experts are enough developed. In some setups (which, interestingly, coincide with some of the original motivation ones for MoEs) this may not be possible, for example if the number of experts is just huge.

Some recent work has shown the effect of increasing "k" a lot (see [1], Figure 10). It seems that the effect of increasing "k" plateaus somewhat quickly. While in those experiments "k" doesn't change over time, I think this may suggest there's little to gain in doing so (especially as it's quite expensive).

Overall, I think the experiments in the paper do not show the proposed algorithm offers any advantage over standard top-k routing (Switch, in the paper).

Figure 3 seems to be the only comparison between Switch ([2] i.e. no warm-up) and DTS-Gate (expert warm-up). First, the reported metric is training perplexity. I'm not sure why this is the case: test (or validation) perplexity should be reported. In the second plot (3b), which is the most relevant in my opinion (FLOPs or runtime, rather than "steps" as in 3a, as these hide very different costs), it seems that Switch dominates DTS. For any FLOPs budget (vertical line), Switch offers better performance than DTS. Similarly, for any attainable performance level, Switch gets there at a lower FLOPs cost. Thus, I'm not sure how the paper justifies the use of DTS. I may have misunderstood something though. It's not clear to me why the DTS run (in red) was halted, and didn't continue training. There's a mismatch between both plots for the yellow curve (MoE-Dense), in the left one (3a), perplexity is below 14, while in the right one (3b), it's above 15. Why? Was the x-axis (FLOPs) trimmed?

In the previous to last paragraph in page 6, there's this sentence: "DTS-Gate can obtain greater than 5% improvements in FLOPs-efficiency compared with state-of-the-art Switch-Gate [...]". Where can this be seen?

Also, and as the authors acknowledge in the conclusions, the fact that DTS doesn't impose balance (if I understand correctly) even when using a load balancing loss (as opposed to having a maximum expert capacity that enforces it), can lead to an inefficient use of current hardware (some experts waiting for others).

For Figure 6, as expert id's are independent across layers, it would be nicer to sort experts per row according to their load, so that clearer patterns emerge.

Alternative idea: One could try something even simpler. Initially train a dense model (with one MLP only, rather than E different MLPs + router) for a number of steps. Then, replicate the dense MLP to all the experts, and add a router from scratch, and apply top-k as usual. In other words, pre-train a dense backbone for a bit, and then initialize the expert model from this. Have the authors tried this? I can see this working well.


**Typos:**

Second paragraph of page 6: "set tokens of each sample as 1024" -> "set tokens' hidden size to be 1024"?

"comparsion" --> "comparison"


**References:**

[1] Scaling Vision with Sparse Mixture of Experts

[2] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
summary_of_the_review: While the algorithm is reasonable, the experimental results (only one plot, Figure 3b) do not suggest it provides any additional benefit compare to the standard baseline.

------------

Note: After the rebuttal, I increased the score from 3 to 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the idea of using dense-to-sparse (DTS) training gates in mixture-of-experts (MoE) based on a gradual sparsification process. The novel produced method is evaluated on a Transformer model. Overall, it looks to me that the paper has rather limited novelty. While the paper is well-written, in my opinion, the empirical evaluation support is not perfectly aligned with the paper claims.
main_review: **Strong Points:**
- using gradual sparsification for the MoE gates of a transformer model seems to offer a final performance (perplexity in this particular case) close to the MoE dense gates, while having lower computational requirements. At least in theory, as I believe that a binary mask is used to emulate sparsity. Is my understanding correct?

**Weak Points and suggestions:**
- The paper claims are not perfectly aligned with the empirical evaluation. If the proposed gradual sparsification with DST is intended to be for any MoE model than other neural network models besides transformers shall be studied. If the proposed method is particularly designed for transformer models then perhaps vision transformers shall be also studied. Based on the chosen direction, new algorithmic baselines and datasets have to be added for comparison. Otherwise, the broadness of the claims needs to be adjusted accordingly.
- The related work discussion is missing very important work. Particularly, gradual sparsification for dense-to-sparse training has been introduced in [1], up to my best knowledge. From that moment, a large body of works has been released on this topic. In parallel, sparse-to-sparse training has been studied for vision transformer in [2]. Please add a consistent paragraph to discuss the above directions and to highlight clearly what is novel in this paper in comparison with those works. Otherwise, the proposed method, broadly speaking, would be just a simple application of [1] (or follow ups) on MoE gates for transformers. A recent survey can give more details on these topics [3].
- Can you please add an algorithm to better illustrate the proposed method?
- Is the lower bound from the “Adaptive Capacity” paragraph fixed across training? How can one choose it?
- Why in Figure 3b, the dense model minimum PPL is higher than in Figure 3a? Can you train all models from Figure 3a for the same number of iterations? Now, somehow, it seems that the complete overview picture is missing
- I suggest to carefully proof read the whole paper. The English usage and general appearance can be improved. Also, there are a number of typos, e.g., “Comparsion with…”. “…As shown in Figure 3.2….” 

**References:**
- [1] Christos Louizos, Max Welling, Diederik P Kingma. Learning Sparse Neural Networks through 𝐿_0 Regularization. ICLR 2018, https://openreview.net/forum?id=H1Y8hhg0b 
- [2] Tianlong Chen , Yu Cheng , Zhe Gan , Lu Yuan , Lei Zhang , Zhangyang Wang. Chasing Sparsity in Vision Transformers: An End-to-End Exploration. NeurIPS 2021, https://arxiv.org/abs/2106.04533 
- [3] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks, JMLR 2021, https://www.jmlr.org/papers/v22/21-0366.html

summary_of_the_review: While there are some clear advantages highlighted by this paper in obtaining more efficient transformer models, overall, I believe that the paper is not ready yet for publication.
review_title: null

Meta-review: This paper proposes a simple approach to improve the robustness of training a sparsely gated mixture-of-experts model, which at a high level simply consists in training initially as a dense gated model, to better warm start a final phase of sparse training. Results are presented to highlight the potential benefits of this approach.

The authors have provided a detailed response and updated results, in response to the reviews. Each reviewer has also responded at least once to the author response. Despite that engagement, all reviewers are leaning towards rejection (though there is one reviewer with a rating of 6, they regardless state that "I'm confident this will make a great resubmission at a future venue", indicating they actually support rejection).

The reviewers point out that the proposed method is not really novel, pointing to an existing recent paper. Even without that prior work, I would also argue that the proposed approach is conceptually straightforward and has benefits that were fairly predictable and not particularly surprising. Given the generally lukewarm reception from the reviewers, I think there is a legitimate concern to be had here about this work's potential for impact.

Though the review process has definitely improved the paper's manuscript since its submission, I unfortunately could not find a reason to dissent from the reviewers' consensus that this submission is not ready to be published. Therefore recommend it be rejected at this time.
Decision: Reject