Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel plug-and-play strategy for training large transformer models, which leverages sparse MoEs in a dropout-like manner to scale transformers to better performance in their full capacity without collapse. The method is simple, and the experiments are thorough.
strength_and_weaknesses: Pros
-	SMoE-Dropout demonstrates an attractive “self-slimming” property during inference and downstream fine-tuning, which delivers a once-for-all in-situ trade-off between efficiency and performance
-	Like classical dropout, SMoE-Dropout is able to mitigate the representation collapse that standard MoEs usually suffer from, i.e., activating more experts do not improve or even hurt performance.
-	Applying SMoE-Dropout is extremely cheap as only random router is adopted
-	Transfer study is strong, which provides another evidence that more pre-training information is effectively retained by the full transformer capacity. It is further validated in ablation studies 4.4

Cons:
-	The performance gain of SMoE-Dropout is sometimes marginal, such as on BeRT (section 4.1). Moreover, as seen from Figure 1 and Figure 3, it seems SMoE-Dropout does sacrifice performance at low parameter counts compared to Learnable MoE, why?
-	When breaking a single-stream models into MLP MoEs (the modularization step in section 3.2), how to decide the number of MLP experts needed, i.e., N? This seems to be an important hyperparameter but not discussed. Also, why not applying SMoE-Dropout to MLP layers but not self-attention? 
-	For training with Learnable SMoE, it is further unclear how the authors selected its k?
-	Why training time is the same between SMoE-dropout k=N/2 and k=N, in Table 1?
-	Table 1 again: are all those numbers averaged across three datasets? The authors didn’t explain

clarity,_quality,_novelty_and_reproducibility: All look good to me. Authors promised to release models and codes: it would help if the authors could respond to be clearer what they plan to release: all experiments versus some, pre-trained model versus training script, etc.
summary_of_the_review: Overall, I would tend to accept this paper. If the author can address my concerns. I would be more convinced.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: SMoE-Dropout is motivated by the representation collapse and expert scalability issues observed in learned MoEs, by dynamically adapting number of activated experts as needed. It consists of a randomly initialized and fixed router network to activate experts and gradually increase their number as training progresses over time.
strength_and_weaknesses: Main Strength Points
(1)	SMoE-Dropout facilitates randomly and sparsely activated structure of network modules, playing an implicit regularization role similar to dropout – but more scalable to large transformer models. It has very low overhead and is very easy to use.
(2)	SMoE-Dropout naturally provides a “self-slimmable” property offering consistent performance boost for transformers with an increase in activated experts during inference and downstream fine-tuning. The transfer gains are especially obvious.
(3)	Experiments are very extensive and thorough, including a variety of tasks, architectures, datasets, and strong baselines beyond vanilla dropout. Both standard training and transfer learning are reported, accompanied with in-depth ablation studies. The experiment and analysis are very well organized and read convincing.

Weakness and/or clarification:
(1)	Does SMoE-training cost higher memory overhead, compared to dense training?
(2)	It is very interesting to see that the SMoE-trained model can be distilled into a single-expert backbone. But details are lacked on how the distillation was done: is it common logit matching or something else? Also, how other MoE-variants will performance under distillation?
(3)	For learnable MoE overfitting and collapse, it is an already well-known phenomenon that can also be mitigated by various diversity-regularized routers. It would be nice to compare SMoE-dropout with some of them.

clarity,_quality,_novelty_and_reproducibility: It seems the proposed method is not a new training recipe for MoEs, but rather a new way to train big dense models by leveraging MoEs as an auxiliary tool. This confused me initially. Clarifying this point better in abstract would help readers set their expectation. 
summary_of_the_review: The paper proposed a new method for dynamically adapting number of activated experts as needed and the overall quality is good. If the  questions as described above are well addressed, I tend to increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Sparse Mixture-of-Experts (SMoE) have been widely used to increase the representation capacity of transformers with similar computational cost as dense transformers. SMoE consists of many experts, only few of which are activate for any given input example. This increased representational power comes with two issues: 
(a) Representation Collapse -- as the latent space representations get centered around experts leading to lots of redundancy and sub-optimal performance
(b) Poor Scalability -- During inference and downstream fine-tuning, increasing the number of active experts does not increase the performance due to overfitting to the active experts during training

This work addresses these issues by introducing SMoE-dropout. It consists of randomly initialized and fixed router network that activates the experts. It gradually increases the number of experts during the training. This yields SMoE that have "self-slimmable" property, i.e., offering consistent boosted performance for transformers with an increase in activated experts during inference and downstream fine-tuning. Empirical evaluations show that SMoE transformers trained with SMoE-dropout yields consistent improvements over their dense counterparts on various reasoning tasks.
strength_and_weaknesses: 
Strengths:
- "Self-Slimmable" -- the consistent boost in transformer performance with an increase in activated experts during inference and downstream fine-tuning
- Extensive empirical evaluation that demonstrates the improvements on pre-training and reasoning tasks

Weaknesses:
- Does not explain the proposal in details (see below)
- Limited novelty ( progressive activation of experts by https://arxiv.org/abs/2112.14397v1, other works have looked at progressive activation of kernels in CNNs such as Once-for-All networks  )


Questions for Authors:

(1) SMoE-Dropout needs to be explained in more detail than what is presented in 3.2. What does it mean to say that the router network is 
 - randomly initialized
 - and fixed 
 - and implicitly optimized during training

 Either it is fixed or you train the network. May be elaborate any points that I have missed.

(2) How do you integrate the Concrete-Dropout and DropBlock in the proposed SMoE-Dropout? A simple pseudo-code that shows these steps end-to-end would help a lot to clarify things. 

(3) Why is only the linear schedule used to gradually increase during the training? How about other ways to progressively activate the experts? How does dropout and expert activation schedule interplay?

(4) How is Figure~1 generated? 
   - For SMoE-Dropout --> for the same SMoE, you increase the number of activate experts --> yielding in higher parameter count.
   - How are green and black curves generated? 
   - Since black represents the dense network --> it should be a collection of different dense models
   - Why is the proposed scheme worse in the low parameter count regime?

clarity,_quality,_novelty_and_reproducibility: Paper would benefit from explaining the Dropout integration along with the expert activation schedule a bit more. It is unclear why the linear activation schedule would be the optimal in this setup. 

Novelty of this work is somewhat limited given that others have tried out similar notion of gradual expert activation (see https://arxiv.org/abs/2112.14397v1). 


Nit-picks 
- Page 2, Para 3, "... SMoE-Dropout ican be ... "
- Page 3, Para 4, "... inferences. Meantime, the presented .."
- Page 4, Eq.1, 'TopK( softmax(G(x), k) )' should change to 'TopK( softmax(G(x)), k )'
- Term "progressive" might suit better in this context rather than self-slimmable





summary_of_the_review: Overall the self-slimmable property is indeed important for a sparse mixture of experts model and the proposed architecture helps in achieving the same. The paper is its current state needs to add more details to the method and explain some of the design choices. It is also lacking in novelty as mentioned earlier. 


----


I've read other reviewer comments and author's response. I'll increase my score accordingly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: While many MoE research focused on improving routing policies to encourage expert diversity and specialization, this paper studies another aspect of expert scalability, i.e., gradually increasing the portion of experts being activated per time. The authors show that by making MoEs smoothly scalable to the number of experts, it can turn into a new structured dropout approach that effectively trains larger transformer models in their full capacity. 
strength_and_weaknesses: First of all, it is very interesting to see the connection drawn between the modern MoEs and the “old-fashioned” dropout. The former typically refers to both sparse training and sparse inference (at the same or similar sparsity), while the later conducts sparse training yet dense inference in full model capacity. Despites their formation similarity, traditionally they pursue different goals: this paper makes a revisit of them and neatly bridges their gap.

Their proposed solution, SMoE-Dropout, simply re-parameterizes a transformer with MoE layers for their MLPs, and then use a fixed random router to activate experts with a “curriculum learning” strategy, i.e., the number of activated experts gradually improved. Such training naturally yields a once-for-all in-situ trade-off between efficiency and performance when considering the deployment – a bonus that most current MoEs do not naturally enjoy. The method is also potentially generalizable to other non-transformer models.

The authors report very extensive empirical studies across various combinations of network architectures and datasets, as well as some transfer learning settings. They consistently demonstrate the significantly improved performance as well as training time savings from the SMoE-Dropout over a number of baselines. The experiment analyses are particularly well done and provide many useful insights on how SMoE-Dropout recipe works.

The following weakness points are identified, and clarifications are requested:
-	Is random routing really the major contributing factor in your recipe? What will happen if we apply the same curriculum learning of k to training learnable MoEs? Will they also become self-slimmable and suffer less collapse?
-	Another main critique I hold against this work is that in many experiments, the empirical gains of SMoE-Dropout seem just marginal over other baselines, although the performance increase trend seems consistent as more experts are activated.
-	 In Figure 1, it also appears that SMoE-Dropout performs worse than learnable MoE when the model is small.
-	I am also curious that, given the “self-slimming” property displayed on the same task, whether and how SMoE-Dropout could help address the compositional generalization challenge, that most MoEs nowadays can only solve with the assistance of clever prompting? For example, it would be very interesting to see the authors testing their trained models on multi-step commonsense or algorithm reasoning cases, perhaps with or without prompting.
clarity,_quality,_novelty_and_reproducibility: The writing clarity and technical quality are great. The method is simple but revisits MoE/dropout from a novel angle.
summary_of_the_review: See the strength and weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the issues of representation collapse and scalability in sparse MoE transformer models. It proposes SMoE-Dropout that uses randomly initialized and fixed router network to address these issues. Empirical improvements are moderate but consistent. All reviewers appreciate the novelty of the idea and contributions of the paper, and the author response has addressed the reviewers' concerns well. 

justification_for_why_not_higher_score: Empirical improvements are moderate. No theoretical justification for the approach. 
justification_for_why_not_lower_score: Efficient training/inference of/with large models is an increasingly relevant topic and sparse MoEs are an important direction for this.
Decision: Accept: notable-top-25%