Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel yet simple method to reuse pretrained dense Transformer checkpoints to initialize larger sparse
models. 
The authors showcase that using this technique will result in a more performant MoE model versus continuing training the dense counterparts on vision and language domains up to 3B scale. 
Extensive ablations are provided for the design choices. 

strength_and_weaknesses: Summary Of Strengths
- the paper is clearly written and presented;
- the efficacy of the sparse upcycling is verified both across domain (vision & language), transfer (downstream) and scales (up to 3B); 
- extensive ablations are provided (router, #dense pretrain, #MoE layers, #experts, MoE locations, and initialization) to illustrate the design choices, which may pave the road for future researchers and practitioners in the efficient computing area. 

Summary Of Weaknesses
- Fairer comparison: it is clearly presented in figure 2 that a sparsely upcycled model outperforms a continual dense counterpart on vision, and language pretraining. However, the sparsely upcycled model also has more capacity, I am wondering if will it be a fairer comparison for:

  (i) upcylce a dense model to a large model (though it will introduce more inference cost) as in [1,2,3,4], especially [1,2] show that staged warm starting may also help the efficiency of training; 

  (ii) continue to train a MoE model as shown in Figure 4, and why the Appendix Table 2 seems to disagree what presented in Figure 4?
  7+7 extra epochs of MoE models seem to outperform a sparsely upcylced model with 7+ extra epochs. 

- Why continual training of a dense model turns to hurt the downstream performance on SuperGLUE; 
- How sensitive are the design choices like in the following will affect (1) the initial transition; (2) the ending performance? 

    (i) the learning rate schedule: resuming/restarting [1], (also in Appendix A.1.1 suggest using a learning rate schedule where the dense checkpoint left off, does that suggest you will initially plan for more epochs, what if the learning rate has already decay to 0)

    (ii) MoE losses: [5] uses half loading balance + half importance and strong regularization like noise jitter and larger capacity ratio to make the V-MoE stable. Will all the techniques (except for the BPR) also be used here and will all these hyperparameters also be sensitive. 

- the intuition in Figure 4 that why will a upcycled model will outperform continue-training MoE

   In Figure 3 and Appendix that the initial transition of the sparsely upcycled model underperforms a dense model, which I assume will also underperform a MoE model. Then using the same schedule, will the sparsely upcycled model consistently underperform a MoE counterpart that trained from scratch?


- Is it possible to upcycle MoE models?

[1] Rae, Jack W., et al. "Scaling language models: Methods, analysis & insights from training gopher." arXiv preprint arXiv:2112.11446 (2021).
[2] Shen, Sheng, et al. "Staged Training for Transformer Language Models." ICML 2022.
[3] Gu, Xiaotao, et al. "On the transformer growth for progressive bert training." NAACL 2021.
[4] Gong, Linyuan, et al. "Efficient training of bert by progressively stacking." ICML 2019.
[5] Riquelme, Carlos, et al. "Scaling vision with sparse mixture of experts." Advances in Neural Information Processing Systems 34 (2021): 8583-8595.


clarity,_quality,_novelty_and_reproducibility: - The paper is clearly presented, and extensive experiments verified the efficacy; 
- the idea of using dense for upcycling is novel and effective; 
- the author promises to release the upcycle code upon acceptance. 

Typo - Appendix B.1 "on a per step basis, provided we also use Batch Priority Routing (BPR)" -> "on a per step basis when using Batch Priority Routing (BPR)"
summary_of_the_review: it is no doubt that sparse upcycling tries to leverage the inference & scaling benefits from MoE and pre-trained weight from a smaller Dense model, and it achieves better results given the more introduced capacity; 

The major concerns for me are the inherited instability of training MoE also appears/worse in upcycling and will that affect the initial transition and how will the initial transition affect the end performance; 

More details for the hyperparameter sensitivity will also be valuable to add.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an approach for initializing sparse Mixture of Expert (MoE) models from dense checkpoints. The approach called upcycling reuses an already trained dense checkpoint by copying all the parameters from the original checkpoint (except for the MoE router parameters). The experts in this new model are replicas of the original MLP layer(s) in the dense model. The experiments demonstrate that the upscaled models outperform the sparse models trained from scratch when using the same computation budget as the initial dense pretraining. 
strength_and_weaknesses: The paper is generally clear and easy to follow. The problem of using already trained dense checkpoints for initializing sparse models is important, especially given the promising results of sparse MoE models in different domains. I found the idea of replicating the MLP layer(s) and randomly initializing the routing mechanism simple yet powerful. I have the following questions and comments:
1. How do the distribution of token to expert assignment look like? Are we seeing few experts which dominate the probability distributions? 
2. How many experts are involved in processing each token (on average)?
3. How do the self-similarities and intersimilarities of inputs between experts look like?
4. The related work, is missing some works on converting a dense model to its corresponding MoE version. See for instance:
    1. MoEfication: Transformer Feed-forward Layers are Mixtures of Experts 
    2. MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
    3. DEMIX Layers: Disentangling Domains for Modular Language Modeling (not directly related but uses a similar idea of replicating the MLP units). 
5. It would be nice to add a comparison to one of the aforementioned approaches to show the advantages of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and the idea is novel. 
summary_of_the_review: Overall, I like the idea and the fact that the paper covers a relatively comprehensive set of experiments for both vision and language domains.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an interesting and useful idea for reusing pretrained dense checkpoints to initialize larger sparse models for better performance on different vision and language tasks. This method helps to improve performance with some additional computational budget but does not require training large sparse models from scratch. The paper conducts analysis on T5 and ViT models with various sizes and discusses different design decisions for upcycling.
strength_and_weaknesses: ### Strengths
- Sparse upcycling clearly shows improved performance for both vision and language pretraining, with significant improvements when some non-trivial computation budget is available for further training.
- importance of this paradigm is in general great to reuse/recycle old models
- Performance improvement is fairly consistent irrespective of how long the dense pretraining was done, which is promising.
- Authors discuss various design choices for upcycling clearly. Ablation experiments are thorough and clearly justifies different design choices.

### Weaknesses
- Initializing MOE models from dense checkpoints is not novel and is somewhat a known practice. 
- Figure 3 is missing the finetuning performance for XL models for language tasks. 
- The performance on downstream language tasks has a lot of variance which increases with larger models and somewhat seems that upcycling does not help in SuperGLUE tasks to a large extent. Is this dependent on downstream tasks? Is this behavior same on other downstream language tasks as well?

clarity,_quality,_novelty_and_reproducibility: The proposed idea is not entirely new and it has been a practice to initialize MOE models with dense checkpoints. However, the insights from this work are of high quality which provides some important design choices when there is limited computation budget and we want to get better performance. The ablation studies have clearly demonstrated the design choices. 

In terms of reproducibility, for the sparse models in this work, will depend on the availability of the pretraining datasets like JFT. However, the learnings from this work will be widely applicable and the thorough discussion of the design choices will help reproducing this work on other datasets. I would encourage the authors to release the code/model/data as far as possible to make this study reproducible.
summary_of_the_review: Overall, this is an interesting work which shows that we can get improved performance by initializing sparse models with dense pretrained checkpoints when there is additional computational budget available. Initialization of sparse models from dense checkpoints is not new to the community but this work shows some good design choices which are important and will be useful. Hence I recommend this rating, and look forward to the author discussion phase.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes sparse upcycling -- copy the model into a sparse model for faster and more efficient training. They conduct experiments on JFT-300M and English C4 dataset. The empirical results show its superiority.
strength_and_weaknesses: The paper proposes a training strategy to make MoE training faster. The proposed sparse upcycling is very reasonable. The experiments also show the superiority of this training strategy. 

I have the following concern and doubts:

1. The proposed sparse upcycling only increases training speed, but not performance. As shown in Figure.4 right, upcycling and moe from scratch would finally converge.

2. The proposed method seems to take as long as the moe from scratch to finally converge. In Fig.4 and Fig.5, the training curve is not finally converged so it occurs to me that sparse upcycling are just faster in the early stage but could take as long as the moe from scratch to converge.

3. The training speed still seems to be at the same level as training a normal moe. The improvement in time is not a game changer.

4. In the ideal case, MoE only increases the params size but not computation cost or computation speed. (which depends on topk) So the dense model training time should be similar to the sparse model training time even though the sparse model is much larger. I am not sure if that really matters to increase the training speed. A demonstration of the relation between params size, computation cost, and the training speed using normal training or the proposed upscale training is missing from the paper. In another word, under what circumstances (params size, FLOPS) do we care about the training speed, and how much this method can improve?

Above all, the method is simple and solid. My major concern is that the speed improvement does not seems to be significant and crucial in a real application and the paper does not demonstrate why this is useful. I am also not sure if upcycling is on a different speed level of a normal MoE with decent optimization. 



----------Afrter Rebuttal---------------

I have read the response from the authors. I still have doubts about the speed improvement: not likely to be significant and mostly comes from the slow MoE implementation. I would keep my rating.

clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: see above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a simple method to reuse pretrained dense transformer checkpoints to initialize larger sparse models.

After author rebuttal, this paper received scores of 5688. All the reviewers agree that this paper is clearly written and presented. The idea is simple but effective. Experiments and ablations are comprehensive. On the other hand, one major concern raised by the reviewer who gave a score of 5 is that the proposed sparse upcycling only increases training speed, but not performance. During rebuttal, the authors have tried to make this clear that sparse upcycling is primarily useful when the compute budget is constrained.

Overall, given the overall positive reviewer comments, the AC would like to recommend acceptance of the paper.
justification_for_why_not_higher_score: One major concern raised by the reviewer who gave a score of 5 is that the proposed sparse upcycling only increases training speed, but not the final performance if one has unlimited computation budget. This could limit the impact of the work, so the AC thinks that it can be  a good poster paper, but not to the level of spotlight. 
justification_for_why_not_lower_score: This paper is overall well written and presented. The idea is simple but effective, experiments are also extensive. So, the AC thinks that it is above the ICLR acceptance bar.
Decision: Accept: poster