Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Authors: Aran Komatsuzaki
Joan Puigcerver
James Lee-Thorp
Carlos Riquelme Ruiz
Basil Mustafa
Joshua Ainslie
Yi Tay
Mostafa Dehghani
Neil Houlsby
Venue: ICLR 2023 poster (2023)
Keywords: mixture of experts
sparse
vision
language
deep learning
superglue
imagenet

Abstract: Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality and computation cost, sparse models remain data-hungry and costly to train from scratch in the large scale regime. In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language models and Vision Transformer Base and Large models, respectively, significantly outperform their dense counterparts on SuperGLUE and ImageNet, using only ~50% of the initial dense pretraining sunk cost. The upcycled models also outperform sparse models trained from scratch on 100% of the initial dense pretraining computation budget.

Published as a conference paper at ICLR 2023
SPARSE UPCYCLING: TRAINING MIXTURE-OF-EXPERTS FROM DENSE CHECKPOINTS
Aran Komatsuzaki*
Joan Puigcerver Google Research
James Lee-Thorp
Georgia Institute of TTehhnloog
Google Research
Carlos Riquelme Google Research
Basil Mustafa
Joshua Ainslie
Google Research
Google Research
Yi Tay
Mostafa Dehghani
Neil Houlsby
Google Research
Google Research
Google Research
ABSTRACT
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality and computation cost, sparse models remain data-hungry and costly train from scratch in the large scale regime. In this work, we propose sparse upcycling a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of- -f-uurertt---xrtts model from a dense checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language models and Vision Transformer Base and Large models, respectively, significantly outperform their dense counterparts on SuperGLUE and ImageNet, using only ~ 50% of the initial dense pretraining sunk cost. The upcycled models also outperform sparse models trained from scratch on 100% of the initial dense pretraining computation budget
INTRODUCTION
Increased scale is of the main drivers of better performance in deep learning. From BERT (Devlin et 2019)) to GPT-3 (Brown et al., (002)) to PaLM (Chowdhery et al., (022) in natural language processing, or from AlexNet (Krizhevsky et al. 2017) ViT-G (Zhai 2022 in vision, break- throughs in performance have been obtained from larger hardware, datasets, and architectures. This
trend holds true in many other domains too, including speech (Baevski et al. 2020). reinforcement learning (Schrittwieser et al. 2020). multimodal learning Yu et al. 2022). and scientific applications of deep learning (Jmpper et 20.T).
However, most state-of state-of-the-art neural networks are trained from-scratch, that is, starting from randomly initialized weights. The cost for training such networks is growing rapidly. For example, in language, BERT-Large (345M parameters, proposed in 2018) required an estimated 0.5 ZFLOPS to train, while GPT-3 (175B parameters, from 2020) required 314 ZFLOPS (Brown et al. 2020). and PaLM (540B parameters, from 2022) required 2527 ZFLOPS (Chowdhery et al. 2022). As result of these computation costs, research into new large language models is often limited to a small number of teams with access to lots of resources. To enable significant further progress, we must develop cheaper ways of training giant models.
In this paper, we explore model upcycling: upgrading an existing model with relatively small additional computational budget. In particular, we focus on upcycling dense models into larger, sparsely activated ofxture-of-Expeer (MoEs). We do not use any new unique sources of data (Wil
Work done while interning Google Research.
Contacts: ar33123432 @gmai com, [juuigcerver, jawpeeeeeerrraeeeeeeeeeee
Jamesl /ggoogle com. (Vision) and https::/
Code is available at http://wwwaaaaaa///a/ooooooooooa
language).

---

Published as a conference paper at ICLR 2023
et al. 2021 Ouyang al., 2022).
We assume the existence of pretrained dense Transformer checkpoint (e.g. (Wolf et al. 2020)), that we then use to warm-start the training of a MoE. By obtain an MoE model more performant
leveraging the additional capacity of from the MoE layers, than the original model, at smaller cost than was used to train the original model. Across all model sizes that we study for both language and vision, with less than 40% additional budget, upeyeling improves the network's 's performance beyond what would be achieved by continued training the original Transformer model.
Sparse upcycling may be particularly valuable in two scenarios: (i) One has access to pretrained Transformer (there are many publicly available) and wants to improve with a modest or constrained computational budget. (ii) One is planning to train a large model, and do not know whether dense or MoE model would be more effective (the latter often being more performant, but more technically challenging to train): one can have both by first training the dense model, then upcycling it into an MoE. model once the dense model saturates.
A central challenge in model upcycling is overcoming the initial performance decrease entailed by changing a trained network's structure. We present model surgery recipe that is effective both vision and language, and numerous ablations for the key components that make it work well. In experiments on Vision Transformers (Dosovitskiy et al., 202T) and T5 language models (Raffel et al. 2020), show that upcycling is highly effective when the computation budget lies between + +10% and +60% of the cost to train the original (dense) network. For example, increasing the performance of ViT-B/16 by at least 1 1% on ImageNet 10-shot requires an additional 58% extra training time (relative to the original checkpoint) if we continue training the dense model; however, it only takes 13% extra training time with the upeycled version. Similarly, upcycled T5-Large and T5-Base models outperform their dense counterparts by 1.5-2 absolute points on SuperGLUE using 46% and 55% extra training, respectively.
BACKGROUND
In this section we recap of the main components used in sparse upeycling: Transformer-bris based language and vision models, and sparsely activated Hixture-of-Experte (MoEs).
2.1 SPARSELY ACTIVATED MIXTURE-OR- (MoE)
Dense models apply all parameters to every input. Accordingly, growing the model capacity results in increased computational cost. Sparse models attempt to alleviate this fundamental issue by only activating a subset of parameters for each input. Sparsely activated tixture-of-Expert (MoE) models are an accelerator friendly family of sparse models that allow training of models with up to trillions of parameters (Shazeer et al. 2017 Fedus et al. 2022).
MoE. models typically alternate standard dense Transformer blocks with MoE. blocks. In particular, we usually replace the MLPs in a Transformer block with a number of "experts" (typically themselves MLPs) with different learnable parameters and a router-a small neural network- decides which expert is applied to each individual token. A number of routing algorithms have been developed, for
example Top-K (Shazeer al. (017)). BASE and Sinkhomn-BAr layers (Lewis et al. [02 1 Clark
ct al. 2022), Hash layers (Roller et al. 2021), and Expert Choice routing (Zhou et al. 2022).
We generally focus on Expert Choice routing, which works follows. Let F denote the total number of experts in a MoE. layer, and n the total number of tokens. The router outputs a matrix R E gaxE with the routing probabilities. where row T; E E corresponds the i-th token and is distribution over E' experts (rg 2 0 and Σ3 rij 1). Then, every expert e independently chooses the 7 tokens with highest probabilities for e (i.e., we perform top-T per column) and processes them. We parameterize T as T C(n/ E)) where C' is capacity factor that we control to choose more or fewer tokens per expert. When C' = 1, each expert processes exactly n/ E tokens; note that some tokens may be processed by several experts, while others by none. This allows for model parameter count increase with minimal FLOPs overhead 2 Letting C' > usually leads to higher performance higher compute cost.
"The FLOPs overhead comes from the (relatively modest) router computation of R.
2

---

Published as a conference paper at ICLR 2023
Original Dense Block
Layer Norm
Attention
Layer Norm
MLP
copy weights
copy weights
coppy weights
MoE
MLP 1
Layer Norm
Altention
Layer Norm
Router
MLP 2
Weighted Sum
fro
MLP E
Upcycled MoE Block
Figure I: The upeycling initialization process, All parameters, and optionally their optimizer state, are copied from the original checkpoint, except those corresponding the MoE router, which does not exist in the original architecture. In particular, the experts in the new MoE layer are identical copies of the original MLP layer that is replaced.
2.2 ARCHITECTURES
We apply the same sparse upcycling recipe to both language and vision tasks, focusing on the T5 (encoder-decoder) (Raffel et al. 2020 Narang et al. 202T and Vision Transformer (encoder)
(Dosovitskiy et al. (021) architectures, respectively. We generally adopt the same gating function and MoB hyperparameters in the encoders of both models. See Section 3.1 for specific design choices and Appendix A for differences between the vision and language upcycling setups.
Vision. Vision Transformers (ViT) are encoder-only Transformer architectures (Liu et al. 202
Radford et al. 2021] Touvron et al. 2021 He et al. 2022) which tokenize and embed images. We upcycle models based on the B/32, B/16, L/32 and L/16 variants. The resultant MoEs broadly follow Vision MoE Transfomers ("V-MoE")) (Riquelme et al.) (021). with two differences; we perform
global average pooling (Zhai al., 2022) and use Expert Choice routing.
Language. We experiment with the T5 (Raffel et al. 2020) encoder-decoder our archetypal language model. We upcycle the Base, Large, and XI. variants of the model. We sparsify both the encoder and decoder. As in our vision setup, the model encoder applies Expert Choice routing. We use Top-K K routing in the decoder with K 2; see also Section 3.1
THE UPCYCLING ALGORITHM
The algorithm is illustrated in Figure T To upcycle a model, we need a dense model's parameters (i.e. a checkpoint). The number and shape of Transformer blocks in the new model is identical that in the original dense model. A subset of the of the MLP layers are expanded into MoE layers. The remaining MIP layers, along with all the layer-norm and attention layers, and the embedding and output layers are copied across from the original model to the new model. Each MoE layer contains a fixed number of experts. Each expert is initialized as a copy of the original MLP. In addition, we add router whose weights are randomly initialized. In Section 4.2.2 we experiment with different variations on this basic recipe. After the new model loaded and initialized, we continue training for number of additional steps depending hyperparameters: same batch size, learning rate schedule, and weight decay leading to the original checkpoint; see also Appendix for full training details.
the available budget and resources. We use the original
3

---

Published as a conference paper at ICLR 2023
3.1 DESIGN DECISIONS
An upeycled model's performance is heavily influenced by the configuration of the MoE layers. Increasing the model capacity by increasing the number of upcycled layers, number of experts or expert capacity will generally lead to a higher quality model, but will also increase the computational cost and/or result in a greater initial quality drop, due the more drastic reconfiguration of the layers.
Router type. For upcycled vision models and for the encoder of upcycled language models, we use Expert Choice routing with capacity factor C = 2. To avoid train time (full batch teacher forcing) versus inference time (single token auto-regressive decoding) discrepancies, we use Top-K (K = 2) routing in the language decoder. Section 4.2.2 we show that Expert Choice routing outperforms standard Top-K routing for upcycling, while both beat dense continuations.
Number layers to upeycle. Adding more MoE layers increases the model capacity dramatically, at the expense of increasing the model's cost, and also causing the quality of the upcycled model to initially drop further relative to the original dense model. Based on our ablation in Section 4.2.2 and prevailing conventions in the MoE literature (Lepikhin et al. 2021), unless otherwise specified, replace half of the MLP layers in our upcycled models with MoE layers.
Number of experts to add in upcycled layers. Each new expert provides new learnable parameters that extend the model capacity. The expert capacity-thh the number of tokens expert processes- is inversely proportional to the number of experts, thus adding more experts does not significantly affect the FLOPS or the run time of the model. However, with a very large number of experts, the upcycled model experiences a larger initial quality drop relative to the baseline dense model. Given sufficient upcycling compute, this initial drop can be overcome. In our studies, we upcycle with +20% to +100% of the initial dense baseline model's computational cost, and in this regime we find that 32 experts provides a good compromise. We explore varying the number of experts in Section 4.2.2
Expert capacity. By tuning the expert capacity, C, we control the number of experts that process each token on average Larger expert capacity generally yields larger quality but also increases the FLOPS and run time. Although increasing the expert capacity yields quality gains on a per step basis, we find that C = 2 generally offers good quality on compute time basis. We ablate through different capacity factors Section 4.2.2
Resuming optimizer state (vision only). When upcycling a model, we can resume the optimizer state from the original dense checkpoint together with the model parameters. Appendix [B.6 we find that reusing the optimizer state gives performance boost for vision models. We did not, however, see any improvement from reusing the dense model optimizer state in our language experiments, so we only reuse the optimizer state for vision models.
Normalize weights after routing (vision only). In an effort to reduce the performance drop when applying the upcycling model surgery, we attempted to normalize the router combine weights of each token to 1. This follows that intuition that each token was previously only processed by a single "expert" MLP in the dense model. Appendix shows that router weight normalization helps upcycled vision models, but hurts the performance of upcycled language models. One hypothesis for this different behavior is that the vision models use Expert Choice routing everywhere, but the language models use Expert Choice in the encoder and Top-K routing in the decoder.
EXPERIMENTS
In this section, we present the main experimental results of the paper. We also share the takeaways of number of ablations aimed at identifying the key aspects of our algorithm; full results are included in Appendix B Most of the results are presented as quality vs. cost plots, where we use the upstream or downstream performance to measure quality, and training time in terms of TPU-core-days (as prominent cost metrics (Dehghani et al.) (202T) or training steps (when the cost per step is the same for all the compared models) to measure computation cost.
For Expert Choice routing, more capacity means that each expert choose more tokens. For standard Top-K K routing, more capacity means that each token is more likely fit into the huffer of its desired expert.
4

---

Published as a conference paper at ICLR 2023
C4 Validation Token Accuracy
58%
JFT Validation Precision at
method
75%
method
Dense
Dense
Uprycling
Upcycling
XL
variant
B/32
Base
B/16
L/1
73%
Large
53%
L/16
XI.
72%
Large
49%
B/16
70%
43%
38%
B/32
68%
Base
10
10a
101
102
10a
Extra Pretraining Time (TPU-core-days) Extra Pretraining Time (TPU-core-days)
Figure 2: Pretraining performance achieved by the dense continuation and upeycling methods, for different Transformer variants. The left plot shows the performance on the vision task and the right the text task. The x-axis shows the extra pretraining time (TPU-core-days), with respect the total time needed to train the original dense checkpoints, for each size. The horizontal lines indicate the quality (y-axis) of the original dense checkpoints.
4.1 EXPERIMENTAL SETUP
All upcycled experiments begin from pretrained dense model checkpoint. Because all of our starting dense checkpoints are trained with an inverse square root learning rate schedule, training can be continued without discontinuities training, showing performance for varying amounts of continued training. As baseline, we also continue the training of the original dense model ("dense continuation").
the learning rate schedule. We upcycle models and continue
Vision experiments. MoE Vision Transfomers ("V-MoE") models are trained broadly following the protocol of Riquelme et al. (2021). Upstream pretraining done on JFT300M (Sun et al. 2017), with validation metrics computed a held-out set of 894,574 examples. Few-shot transfer follows Dosovitskiy et al. (2021). whereby a least-squares regressor predicts one-hot classes given frozen image representations. We further validate results ImageNet using 10-shot i.c. 10 training examples per class. We do this for different training sets, and report average accuracy replace the pretraining head with randomly initialized head,
them. For full finetuning, and finetune the entire network. See Appendix A.2. for further details.
Language experiments. Our language experiments follow the setup of Raffel et 2020):
pretrain using the span corruption task on the English C4 dataset (Raffel et al. (020) and finetune on proportional mix of all SuperGLUE (Wang et al., 2019) tasks simultaneously. We include specific training details in Appendix A.2 but highlight one important aspect here: For Base model sizes, for which we perform the majority of our ablations, we pretrain the dense baseline starting checkpoint ourselves. To highlight the versatility of our upcycling algorithm, for Large and XI. models, we instead begin all experiments from official T5 1.1 checkpoints (Narang al.) [021] Roberts et 2022).
4.2
EXPERIMENTAL RESULTS
4.2.
CORE RESULTS
Figure[ishows a detailed comparison of upstream metrics upcycled models and dense continuation models at various model sizes both for vision (left panel) and language (right panel). For any given model size and task, we observe that the dense and upcycled models perform close to each other when we apply very limited extra training budget indeed, close to their discontinuous horizontal line representing the original checkpoint's performance. Once we apply a non-trivial amount of extra compute, clear pattern emerges showing the strong gains delivered by the upcycled architecture.
5

---

Published as a conference paper at ICLR 2023
ILSVRC2012 Finetune Accuracy
melhod Upeycling
SuperGLUE Finetune Score
method
87%
87%
L/16
Upeycing
variant
B/32
85%
Large
B/16
Large XXL
85%-
B/16
82%
83%
81%
Base
79%
B/32
73%
10
102
10
102
Extra Pretraining Time (TPU-core-days)
Extra Pretraining Time (TPU-core-days)
Full finetuning performance achieved by the dense continuation and upeycling methods. The left plot shows the performance on ImageNet and the right one on SuperGLUE tasks. The x-axis shows the extra pretraining time (TPU-core-days), with respect the total time needed train the original dense checkpoints, for each size. The horizontal lines indicate the quality (y-axis) of the original dense checkpoints.
Figure 3:
JFT Validation Precision at 1
53%
Upcycling
C4 Validation Token Accuracy
Upcycling
McE
MoE
49%
69%
45%
68%
41%
67%
37%
10a
10a
Extra Pretraining Time (TPU-core-days) Extra Pretraining Time (TPU-core-days)
Figure 4: Pretraining performance achieved by the upeycling method and a MoE model trained from scratch, for B/16 (left plot, vision task) and Base (right plot, text task). The x-axis shows the extra pretraining time (TPU-core-days), with respect to the total time needed to train the original dense checkpoints. The MoE model trained from seratch only catches up the language upcycled model after about 120% the original dense checkpoint computation budget (second last orange and green dots from the right).
Figure 3 shows the performance after finetuning the models trained Figure For vision (left panel), the upstream performance gains generally transfer fairly cleanly downstream. For language (right panel), there is more variance the performance after finetuning. Nevertheless, the trend clearly favors the upcycled language models.
Figure compares sparse upcycling with sparse models trained from scratch. As training from scratch does not reuse the computation cost already sunk into the dense checkpoint, it takes longer, on a extra train time basis, to catch up with the upcycled models. The language MoE model trained from scratch requires about 120% of the original dense checkpoint's computation budget to catch up to the upcycled model. The relatively faster quality gains, on a per step basis, of the MoE models trained from scratch can be attributed to the relatively larger learning rate and that the experts able to independently develop and diversify from the beginning. Figure 4 suggests that, given a very large computation budget (> 100% of the initial dense model's computation budget), the MoE-from-

---

Published as a conference paper at ICLR 2023
C4 Validation Token Accuracy
C4 Validation Token Accuracy
Dense Upcyccling
Dense Upcycling
Warm start
Warm start
70%
69%
69%
68%
Base
68%
Base
101
Extra Pretraining Time (TPU-core-days)
Extra Pretraining PetaFLOPs
Figure 5: Pretraining performance achieved by sparse upcycling and dense upcycling from T5 Base checkpoint (text task). The x-axes show the extra pretraining time (TPU--cre-days) and extra pre- training (Peta)FLOPss, with respect to the the original dense checkpoints. Following recommendations in (Rae et al. (021). we only increase the number of layers when warm starting ("depth tiling") to roughly match the runtime of the sparse model.
scratch model will eventually catch the upcycled model. For such large computation regimes, may be preferable to train MoE models from scratch. For constrained or limited compute budgets (< 1.0% of the initial computation budget), sparse upcyeling a more efficient use of resources.
Finally, Figure 5 compares sparse upcycling with warm starting ("dense upcycling")). We warm start larger models from the dense Base checkpoint by replicating new layers ('depth tiling") in the same tiling patterns in (Rae et al. [2021). The densely upcycled models quickly see gains over the original dense checkpoint, but underperform the sparse model. We did not attempt to increase the model hidden dimensions ("width tiling"), which (Rae al. 2021 found to be less effective.
4.2.2 ABLATIONS
In this section we summarize important architecture and training ablations relative to the baseline model described in Section 3 Full results provided in Appendix B Unless stated otherwise, vision ablations use a B/16 sparse model with 32 experts, = and MoE layers placed in the last few blocks of the model. The dense checkpoint
trained for 14 epochs, and we train for additional epochs (up to a total of 21 epochs). For our language ablations, our default configuration Base model with 32 experts, C = 2 and 6 MoE layers interspersed throughout the model. We train for between 0.5 million and 1 million extra steps.
unchanged:
Amount of dense pretraining. The upcyeling efficiency may, in principle, depend how converged the initial dense model is. To explore this, in Figure 6) we upcycle a B/16 vision model starting from different dense checkpoints with varying amounts of pretraining. From a given dense checkpoint, we compare upcycling and dense continuation for 200k steps. Independent of when we start upcycling, the performance improvement from doing so is fairly consistent.
Router type. While our default upcycling recipe uses Expert Choice routing (in the encoder), the same recipe can be applied other routing mechanisms. For vision, Appendix B. shows that although Top-K routing, with Batch Prioritized Routing (BPR) (Riquelme et al.) 202T). matches Expert Choice routing performance on per step basis, as it is slightly slower, Top-K routing underperforms Expert Choice routing per train time basis. Note both approaches beat dense.
Expert capacity factor. The more tokens processed per expert, the greater the amount of compute input example and (generally) the higher the model quality. Our sparse models smoothly control this via the expert capacity factor C. Appendix B.2 explores how the performance-speee varies as a function of C. Although increasing the expert capacity yields quality gains per step basis, we find that C generally offers the best quality on a compute time basis, for both language and vision models.
trade-off
7

---

Published as a conference paper at ICLR 2023
0.75
ILSVRC2012 10shot Accuracy
0.50
0.74
JFT Validation Precision
0.72
0.46
0.70
0.69
0.44
0.67
Dense
Upcycling
200k 400k 600k 800k 1.2M dense pre-training steps
200k 400k 600k 800k 1M 1.2M
dense pre-training steps
Figure 6: Upeycling performance as a function of the amount of pretraining steps for the original dense checkpoint. The y-axis shows the performance after 200k steps of further training top of the original dense checkpoint, for both the dense continuation and upcycled models. The x-axis shows for how long the original dense checkpoint consistent independent of the amount of initial pretraining. Note: for this particular ablation, we use capacity factor of C 1, to ensure that the FLOPS and run times of the dense model and sparsely upcycled model are roughly comparable on a per step basis.
trained. The gains from upcycling fairly
Number of MoE layers. A key decision when upcycling model is how many sparse layers to add. Model capacity and parameter-count increases with the number of MoE layers, but at the expense of slowing down model run time. Appendix(B.44 provides an ablation over the number of MoE layers in upeycled B/16 vision model. In this case, around 6 MoE layers (out of a total of 12 layers) offers the best precision quality, although fewer sparse layers lower the compute costs.
Initialization of experts. The standard upcyeling recipe copies and replicates the dense MI. each expert. As the router directs different tokens to each expert, the experts will start to diverge from one another, and from their initial MLP weights. Appendix [B5 compares the standard recipe with randomly initializing the experts (i.e. train them from scratch). For limited computation budgets, randomly initializing experts underperforms the standard recipe; for larger compute budgets eventually matches the standard upcycling recipe performance.
We also tried copying the original MLP weights and adding independent (Gaussian) noise each expert, in an effort to promote more expert diversity. Adding too much noise when copying MLP weights into experts hurts performance, while adding small amounts of noise has little to no effect; see also Appendix B.9
Number of experts. Adding more experts increases the number of model parameters and, up to point, the quality of the model. Given that the number of tokens each expert processes is inversely proportional to the number of experts (see Section 2.Tp, adding more experts does not significantly affect the model FLOPS nor its running time. However, for very large number of experts, the upcycled model may experience a larger initial quality drop relative to the baseline dense model. Appendix B.3 explores this trade-off and shows that, at least for Base sized models, more experts usually yield better performance.
RELATED WORK
Reuse of trained parameters. Prior work has focused on speeding up training through warm start by reusing parameters an existing model. Berner et al. (2019) explore ideas of reusing the previous edition of trained model during a long training process using
dedde--dveeloment environment. Given a trained model, Net2Net (Chen et al. 2015) propose a function-preserrvnn initialization to warm start training a deeper or wider model. Recently, Gopher (Rae et al. 2021 also explored warm starting larger models from smaller models in a large compute training regime and show that the larger, warm started model trained from scratch. In Section 4.2. we show that warm starting significantly underperforms sparse upcycling. Yang al, (2021): Lin al. (20TT) show that they reduce the number of training
converge
quality comparable to that of the equivalent model
8

---

Published as a conference paper at ICLR 2023
iterations with models that inititally share parameters across layers (Lan et al. 2019 Dehghani et al. 2018) but gradually unshare (or "delink") the parameters while training.
In effort to reduce total training cost, several works explore progressively growing models during training (Gong et al. 2019, Dong et al. 2020; Li et al., 2020; Shen et al. 2022). The core idea is to decompose the training process into stages, each of which apply growth operators to increase the model size from the previous stage by copying weights or stacking new layers on top. In some cases, each training stage will only update parameters of the new layers, which saves the cost of full backward computation (Yang et al. (020). Gu et al. (2020) show that compound scaling (scaling depth, width and input length together) favorable and propose strategy with various growing operators on each dimension.
Sparse upeycling. which we introduce in this paper, follows a similar motivation. However, unlike the above works, we focus on compute regimes that are fraction of the original model's training. We also present recipe for growing trained dense model to a sparse model, instead of a larger dense model. This enables us enjoy the extra capacity due to increased parameters, while maintaining the inference cost due to the sparsity of computation.
Pruning. Pruning is typically employed as a post-training architecture search to construct smaller and faster models from larger models (LeCun et al., 1989) Gale et al.] 2019: Blalock et al., 2020). However, "dynamic pruning" (Evci et al. (020) has also been used during training to find sparser architectures from dense models. Similar to pruning, sparse upcycling also introduces sparsity to a dense model, however, unlike pruning. we grow the existing dense models into a larger sparse model.
Sparsely-activated Mixture-of-Experr (MoE). In this work, we sparsify existing dense models into MoE models. MoE models (Shazeer et al. 2017) offer the promise of increasing model scale (parameter count) with sublinear increases in computation cost (FLOPS). Recently, there has been a growing number of MoE works achieving state-of-the-art quality and remarkable efficiency gains on both language and vision tasks (eeiikhin et al. [021] Fedus et al. 20222 Riquelme et al. [202]]
Artetxe et al., |021 Du al. 2021 Zoph et al. 2022; Mustafa et al. 2022). All of these models are large and trained from scratch with randomly initialized weights and lixed architectures.
Several MoE works have also attempted to improve upon typical training algorithms by adapting or "evolving" the model architecture during training. [Nie et al. (2021) progressively sparsify MoE layers during training by slowly adjusting the gating function from a "dense setup". where all tokens are routed to all experts, to a fully "sparse setup", where tokens are only routed to subset of experts. ZZhng et al. (2022) observed that, for most inputs, only small fraction of Transformer MLP activations are nonzero. Based this, they propose sparsification procedure that splits the parameters of MLP blocks into multiple experts and add a routing mechanism. Similarly, Zuo et al. (2022) split up the MLPs in pre-trained dense model into multiple experts to form a sparse model for fine-tuning. Closely related to our work. Wu et al. (2022) present a novel algorithm to sparsify dense models in the context of finetuming
detection and segmentation tasks. Similar to Nie et al. (2021), the initial performance drop when training on the original dataset is avoided by applying a dense mixture of experts in the forward pass. However, at our target large scales, simultaneously activating all experts for each token is not feasible. Finally. Gururangan et al. (2022) adapt sparse, domain expert language models to new domains by initializing new domain expert from the most probable existing expert under the domain posterior distribution.
CONCLUSIONS
Training large neural networks on huge datasets has proven to be a remarkably successful trend in deep learning research, especially in recent years. It has also proven to be very computationally expensive. Pretrained models are now widely available, thus making it possible for many practitioners to further finetune and adapt fixed model architectures on their data of interest. However, significant progress requires providing more flexibility in adapting and improving the model architecture itself.
We proposed simple recipe to reuse pretrained dense checkpoints to initialize more powerful sparse models. Our algorithm leverages the pretrained model compute and weights, and provides a smooth transition to sparsely activated of inference. We presented experimental results both for vision and language models at various scales; these evidence large performance gains relative continuing to the dense model. Our ablations
models that offer more capacity and flexibility
9

---

Published as a conference paper at ICLR 2023
highlight the importance of careful algorithmic choices, and suggest key aspects to consider when trying to find good performance-com trade-offs for specific compute budgets.
Transfer learning and prompt tuning is becoming increasingly popular, and for good reason. It allows the reuse and tuning of models by a larger body of researchers and practitioners that may only have access to limited computational and data resources. Accordingly, we believe that techniques aimed at growing existing models, compartmentalizing or freezing submodules, replicating and then decoupling model components, and finally smoothly resuming training after model surgery, will prove essential for a dynamic ecosystem of models. We summarize such process as upcycling, and offer a first instance in the context of sparse models. We look forward new extensions and improvements on this simple idea.
REFERENCES
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan lyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona T. Diab, Zornitsa Kozareva, and Ves Stoyanov. Efficient large scale language modeling with mixtures of experts. CoRR, abs/2112. 2021. URL wwtp/r////bbb...ao https: http:////aaaa...rr//////22/22210
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449-111 1222449---6600, 12460, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint urXiv: 1912.0680,, 2019.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerberttVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Sys- tems, volume 33, pp. 11877-190111 Curran Associates, Inc., 2020. URL http/ttttppoooeediiiiiie Bwt.4400bbbaccccc 42f642- pdf
heurips. wtwp:ipss.ccpppppe/e/220. 1tt.:////EE1111170000o0ooooo
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint
2015.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204002311. 2022.
Aidan Clark, Diego Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hemnigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models. CoRR, abs/2202.01 19, 2022. _ https:://aaattt http://:///axii../////aasse// wttpeerg/aabba///aaee 01169
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Fukasz Kaiser. Universal transformers. arXiv preprint 1807v:1807.038// 2018.
Mostafa Dehghani, Anurag Amab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer. arXiv preprint arXiv:21 10. 10.01.///// /2894, 2021.
10

---

Published as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-train Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers); pp. 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/199-111 1423. URL https
Minneapolis, Minnesota, June
Chengyu Dong, Liyuan Liu, Zichao Li, and Jingbo Shang. Towards adaptive residual network training: neural-ode perspective. In International conference on machine learning, pp. 2616-2626. PMLR, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, URL URLnttps: / opwp/eweenreeeeeeeeeeeeeee www.eeeeeetoeeaeeeeeeeeeeeeeeeee
May 3-7, 2021, 2021.
Nan Du Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang. Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. CoRR, abs/21 12.06055, 2021. URL https: / http//aaivvvv..o org/ abs / 212.1111.2 06905
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952.
PMLR, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-3 2022. URL / fm1 wwt.///gpppppp..e...oo v/23//111 hhmm
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 2019.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pp. 2337-2346. PMLR, 2019.
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive bert training. arXiv preprint 2rXiv:2010.12562, 2020.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 5557-5576, Seattle, United States, 2022. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2022. www.e2222.aaaallieeeeaaa www.n-aaainaaaaaaa 407. URL Intpp!
Maa.lantholooy.erg///000
nww.naaaaaaaaaamaaa 407
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909. 1942, 2019.
11

---

Published as a conference paper at ICLR 2023
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. URL https::///peeeeee 7X
n
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6265 6274. PMLR, 2021. URL httpp://proceeeingss ml http:///www......../////////////
html
Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu. Shallow-to-decy training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), 2020.
Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Yong Li, Wei Lin, et al. M6-10t: A sharing-delinking paradigm for efficient multi-trillion parameter pretraining. arXiv preprint arXiv:21 ( 10.03888, 2021.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint arXiv:2206.0277 02770, 2022.
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel. Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Mareus, Adam Roberts, and Colin Raffel. Do transformer modifications transfer across implementations and applications? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7758-5773 Association for Computational Linguistics, November 2021. doi: 10. 10.18653/v1/20211 www.eeeennnmmmmeeeeeeeee URL 465
wwwp:llooyyyrr//000 emnle-mmlmmmm main
Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma, Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang, and Bin Cui. Evomoe: An evolutional mixture-of-experts training framework
dense-to-sparse gate, arXiv preprint arXiv:2203.02155, abs/2112. 2021.
URL latps
14397
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow 2022.
instructions with human feedback. arXiv preprint
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 1 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published as a conference paper at ICLR 2023
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.17188 2022.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Processing Systems 2021, NewrlPS 2021. December 6-14, 2021, virtual, pp. 17555-17566, 2021. URL http::tttp/oroeeeiiiioo wewriinrrppp wwwporips.ccpppppee pape//20/
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon
Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go. chess and shogi by planning with a learned model. Nature, 588(7839):604-609, 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML 2018, 2018
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-
26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
net @oorummiddBIckMgg22
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. arXiv preprint arXiv:2203.06211, 2022.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint N.Xiv:2109.01652 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-ther art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguisties. httl/tttss:/// //t::///wwww. a web. httb.brr//antholgg//0000000
emn p- demos. 6
Lemeng Wu, Mengchen Liu, Yinpeng Chen, Dongdong Chen, Xiyang Dai, and Lu Yuan. Residual mixture of experts. arXiv preprint arXiv:2204. 09666,,9, 2022.
Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for bert training speedup. arXiv preprint arXiv:2011.13635,, 2020.
Shuo Yang, Le Hou, Xiaodan Song, Qiang Liu, and Denny Zhou. Speeding up deep model training by sharing weights and then unsharing. arXiv preprint arXiv:2110.03844 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2005.01917, 2022.
13

---

Published as a conference paper at ICLR 2023
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-12113, 2022.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguisties: ACL 2022, pp. 877-890, 2022.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du Yanping Huang, Vincent Y. Zhao, Andrew M. Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing. CoRR, abs/2202.09368, 2022. URL http://///wxxi..oo/abab/.//
09368
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.0899 2022.
Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He. Tuo Zhao, and Weizhu Chen. Moebert: from bert to mixture-of-experts via importance-guict adaptation. arXiv preprint arXiv:2204.07675. 2022.
14

---

Published as a conference paper at ICLR 2023
JFT-300M Precision@)) (%)
0.5
0.4
0.3
0.2
Dense
Upcycling (C=1)
Upcycling (C=2)
Upcycling Starting Point
0.0
0.0
0.5
1.5 Training Steps
1.0
2.0
2.5
Figure 7: Effect of upcycling VIT-B/16 model after 14 epochs of dense training. We show a number of cooldowns (decreasing learning rate to zero) for each model, in case that is the maximum training budget available. The difference in slope for the dense and upcycling training curves is significant.
A TRAINING AND EVALUATION DETAILS
In this section, we describe the precise setup for our language and vision experiments. Figure illustrates the type of combined training curves we obtained before and after upcycling.
A.1 UPSTREAM TRAINING
A.1.1 LANGUAGE
We first pretrain dense Base model from scratch for million steps with batch size of 512 sequences. We use the Adafactor optimizer with an inverse square root decay and a peak learning rate of 0.01. This results in plateauing performance for the dense Base model. Upeycled models then initialized from the million step dense checkpoint, and compared, on a compute time hasis, with further training of the dense model ("dense continuation") "); see Figure 2 in the main text. To highlight the versatility of our upcycling algorithm, for Large and XL models, we instead begin all experiments from official T5 1.1 checkpoints (Narang al. 2021
We use the same hyperparameters for the upeycled model as for the corresponding dense model that we initialized from, continuing the inverse square root learning rate schedule where the dense checkpoint left off. For all sizes, every other layer was upcycled, using 32 experts, starting with the second layer. Similar to dense pretraining, we do not include any dropout (or expert dropout; see Section A.2. below). below.. Router parameters are initialized randomly with zero-mean normal distribution with standard deviation 0.02. We use a maximum routing group size of 4096 tokens. For Top-2 routing (in the decoder) we include auxiliary MoE loss, with scaling factor 0.01, to ensure
tokens are distributed more evenly across all experts the decoder (Shazeer et al. 2017 Fedus et al. 2022).
Upcycling was performed on TPU v4 accelerators using 64 chips for Base and Large and 256 chips for XI. All sizes used expert partitioning. but only XI. used model partitioning, with 4 partitions.
For experiments starting from the official checkpoints match the official, larger batch size (2048), to ensure discontinuities in continuing the dense baseline pretraining.
15

---

Published as a conference paper at ICLR 2023
A.1.2 VISION
Dense ViT models are pretrained on JFT300M Sun et al. (2017). We train with Adafactor (Shazeer
Stern 2018 and decoupled weight decay (magnitude 3 head and 0.03 on body) following Zhai
et al. 2022 We use batch size of 4096. The learning rate schedule consists of a learning warmup of 000 steps, followed by reverse square root decay with timescale 100 000 steps and ending with a linear cooldown to 0 over 50 000 steps. We use fixed peak learning rate of -0
Models
employing patch size of 32 (i.e. B/32, L/32) were trained for total of 14 epochs, while those employing a patch size of 16 (i.e. B/16, L/16) were trained for 28 epochs. By default, we begin upeyeling half-way through the total number of epochs in each case. This roughly corresponds to the number of epochs used to train the corresponding variant in ViT (Dosovitskiy et al. (2021). For B/16, for example, we assume that dense checkpoint trained for 14 epochs given, and either continue training or apply upcycling for another 14 epochs.
A.2
MODEL TRANSFER
A.2.1 LANGUAGE
For finetuning on SuperGLUE, we generally adopt the conventional setup (Raffel et al. 2020 Narang al. 2021) where finetune on all SuperGLUE tasks, in proportional mix, for 200K steps with batch size of 128. Each example has input length 512 and target decoding length of 62. In our figures, report the average SuperGI.UT accuracy score across 3 runs for each data point.
For finetuning Dense models on SuperGLUE, we use Adafactor with the default, constant learning rate of 10-3 3 and dropout rate of 0.1 (Raffel et al., 2020 Narang et al., 2021). For finetuning upcycled models, because there are many more parameters, it can be helpful to increase the dropout rate for the experts (Fedus et al., 2022), while using the default dropout rate of 0.1 for all "dense" parameters. For upcycled Base models. we obtained the strongest results for constant learning rate of 10 with an expert dropout rate of 0.1. For upcycled Large models, we found slightly stronger results with a learning rate of 10-3 and an expert dropout rate of 0.3. Decreasing (or increasing) the learning rate was not helpful for the Dense Base Large models.
A.2.2 VISION
Few-shot linear evaluation. The fewshot evaluation poses classification as a linear regression task, where inputs are frozen representations computed by pretrained model, and outputs are one-hot vectors representing the ground truth (Dosovitskiy et al. [2021) There two key changes compared to prior works which used this method (Dosovitskiy et al. 2021 Riquelme et al. 202I):
Multiple seeds. The evaluation involves randomly selecting V examples per class. To reduce dependency that choice, run 5 random seeds, and report the average test accuracy across them.
Fixed L2 regularization. Prior works considered a range of L2 regularizations. The optimal value was picked based on average test accuracy across all datasets considered for few-shot evaluation. We fix the 1.2 regularisation at 1024.
Full finetuning. We finetune models on ImageNer2012 using SGD and batch size of 512. We use cosine decay learning rate schedule with linear warmup. We sweep over two training schedules: (i) 5k steps, with warmup of 200 steps, and (ii) 10k steps, with a warmup of 400 steps. Alongside this we sweep over learning rates I0.1, 0.03, 0.01, 0.003, 0.001, 0. 0.0003]. For each pretrained model, there therefore 12 finetuning sweeps; select based on optimal validation accuracy, and report the corresponding test accuracy.
A.3
MODEL PARAMETERS
Table D gives the number of parameters for models used in the main text.
Note that this is slightly different to ViT (Dosovisskiy et al. 2021 which changing the learning rate slightly based the model variant.
16

---

Published as a conference paper at ICLR 2023
Table Model sizes. The number of parameters for sparsely upeycled and MoE-from-scrane models are the same (both are of type "Sparse"). The number of parameters is also unchanged between different routing mechanisms.
Modality
Model
Type
Fraction of MoE Layers
# Experts
# Parameters
Vision
B/32
Dense
101M
Vision
B/16
Dense
100M
Vision
L/32
Dense
324M
Vision
L/16
Dense
322M
Vision
B/32
Sparse
6/ 6/12
32
980M
Vision
B/16
Sparse
6/
32
978M
Vision
L/32
Sparse
12/24
32
3.44B
Vision
L/16
Sparse
12/24
32
3,44B
Language
Base
Dense
248M
Language
Large
Dense
783M
Language
XL
Dense
2.85B
Language
Base
Sparse
6/12
32
2.00B
Language
Large
Sparse
12/24
32
7.22B
Language
XL
Sparse
12/24
32
26.26B
A.4 PARALLELIZATIO STRATEGIES
Sparsely activated Mixture-of-Eaae -frpeetss (MoE) models combine three types of parallelization strategies to train large models across multiple accelerator chips: data, model and expert parallelism. We use data parallelism shard the training batch across devices. We use expert parallelism to partition experts across devices; for example, placing experts and 2 on device 1, experts 3 and 4 on device 2, Model parallelism is third axis along which model weights (matrices) can be sharded across devices; for example, expert is split across devices and 2, expert 2 is split across devices and 4, and so on. Model parallelism is beneficial for scaling larger model sizes. See also (Fedus) et al. 2022 for a more detailed discussion of these three parallelization strategies.
and
B ABLATIONS AND ADDITIONAL EXPERIMENTS
In this section, we present results for number of model ablations that try to identify good choices for the main upcycling algorithm decisions. As mentioned in the main text, unless stated otherwise, vision ablations
a B/16 sparse model with 32 experts, C 1 and MoE layers placed in the last few block of the model. The dense checkpoint was trained for 14 epochs, and we train for an additional 7 epochs (up to total of 21 epochs). Note that, for C 1, comparing performance per step basis is a reasonably close approximation of a comparison on a per train time basis.
For language ablations, default configuration is unchanged: experts, C and 6 MoE layers interspersed throughout the model. We train for between 0.5 million and million extra steps.
Base model with 32
B. ROUTER TYPE
While our default upcycling recipe uses Expert Choice routing Zhou et al. (2022) (in the encoder), the same recipe can be applied to other routing mechanisms. Here, we compare with Top-K rout- ing (Shazeer et al, (2017). which is very popular alternative. Table 2 shows that, for vision, sparse upcycling with Top-K routing works comparably well to Expert Choice, on a per step basis, provided we also use Batch Priority Routing (BPR) (Riquelme al. 2021). BPR sorts tokens according to model confidence proxy so that wwhen experts are full- high confidence tokens are given priority. We suspect this may be helpful right at the beginning, when applying the upcycling, to avoid discarding important tokens. Expert Choice avoids this problem by design, as experts are always balanced and select the most 'relevant" tokens.
17

---

Published as a conference paper at ICLR 2023
Table 2: Sparse Upeycling on L/32 vision models with Expert Choice and Top-K routing (also known as T)p K refers to the number of selected experts per token, while C refers to the capacity factor. Notice that with Expert Choice routing, each token chooses C' experts on average. The initial dense checkpoint was trained for 7 epochs. Note that these comparison are per-step basis, and that Expert Choice upcycled models are actually slightly faster than Top-K K models; see Figure
Model
Capacity
From
Extra Epochs
Val Prec@]
ImageNet 10shot
Dense
Dense
49.60
73.59
Expert Choice
C 1
Dense
7
51.91
74.04
Top-K
K
Dense
51.51
74.40
Expert Choice
C 2
Dense
52.80
74.83
Top-K
K =
Dense
52.88
74.91
Expert Choice
C' = 1
Scratch
7
50,42
72.95
Expert Choice
C 2
Scratch
51.28
74.01
Expert Choice
C 1
Scratch
14
54.84
75.02
Expert Choice
2
Scratch
55.46
75.75
C4 Validation Token Accuracy
Switch
Tokens Choice (with BPR)
Experts Chaice
70%
69%
10
Extra Pretraining Time (TPU-core-days:
Figure 8: Comparison of Expert Choice, Top-2 and Switch (Top-1) 1) routing mechanisms for Base upcycled language model.
For language, similar ablations (Figure shows that Expert Choice routing outperforms both Top-2 routing (with BPR) and switch (Top-1) routing, a train time basis.
B.2 ROUTER CAPACITY FACTOR
Sparsifying the dense model increases the model capacity (number of parameters). However, if the capacity factor C 1, then the FLOPS is very similar to the original, dense model (modulo the small routing costs). We increase the per-token compute by increasing C. Figure investigates this, and shows our results for vision (left and center panels) and language (right panel).
For vision,
that extreme values (C 1 and C 5) underperform intermediate values ( (C' = 2 and C = 3) that offer better trade-offs. For language, the trend is even stronger: A capacity factor of C' 2 stands out the best option per compute basis.
B.3 NUMBER EXPERTS
Adding more experts increases the number of model parameters and, up to point, the quality of the model. Given that the number tokens each expert processes inversely proportional to the number of experts (see Section 2.1 adding more experts usually only leads to very modest computational (and wall time) overheads. However, for a very large number of experts, the upcycled model may experience larger initial quality drop relative to the baseline dense model.
18

---

Published as a conference paper at ICLR 2023
ILSVRC2012 1 Accuracy
Validation Token Accuracy
1.25
JFT Valydation Precision at
2.0
77%
2.0
3.0
5.0
78%6
5.0
74%
73%
avr
63%
71%
30a
10.
Pretraining (TPU-rore-days
Pretraining (TPU-core-days) Extra Pretraining Time (TPU-core-days)
Figure Pretraining performance achieved by upcycling using different capacity factors, for 6/16 (left and center panels, vision tasks) and Base T5 model (right plot, text task). The x-axis shows the extra pretraining time (TPU-core-days), with respect to the total time needed to train original dense checkpoint, Although using bigger capacity factors can result in an absolute better performance when runtime disregarded (e.g. see the vision results), for a given fixed compute budget, it is usually better to use capacity factor of around 2.0.
64%
11S. LLS.co22e L0chet Accuracy
1LSNII220 2 10blett Accunncy
Volidatton Precidin
23N-
Extra Prst mrsiiinin Time (1P0-4)aa Extm Pretriining Time ((PU-aanesap))) Extre Powinkning Time ITPU-oonedagg Extrx Pratrsining Time (TPUlooredaa)))
Figure 10: Pretraining performance achieved by the upcycling method on the vision tasks, using different number of experts per MoE layer (two left plots, with total number of 6 MoE layers), and different number of MoE layers (two right plots, with a total number 32 experts; all MoE layers are placed at the top Transformer blocks). The x-axis shows the extra pretraining time (TPU-core-days), with respect to the total time needed to train the original dense checkpoint.
Figure (two left panels) shows the results of a vision experiment with 6 MoE layers with number of experts ranging from 2 to 128. For a fixed amount of compute (value in the x-axis), we see that more experts is generally better for this B/16 model. Figure [1 shows the final metric values both for upstream (JFT precision at I) and downstream (ImageNet 10-shot) with respect to the number of experts. We see steady improvements upstream, and -at some point- diminishing returns downstream.
B.4 NUMBER OF MOE LAYERS
Another key decision is how many layers to sparsify. More layers leads to higher model capacity, while -especially for higher C- it introduces significant extra wall time overhead. We ablate this for vision models, as shown in Figure I0 (two right panels). For a B/16 model with 12 blocks, we train upcycled versions with an increasing number MoE layers; MoE layers are consecutive and start from the last layer. For example, the model labeled as '5' corresponds to model where the last MIPP layers are sparsified, and so on. Thus, model '1' only has one MoE layer (the last one) and is the computationally cheapest in terms of wall time. We do not include model where all layers sparsified (would correspond to '12') as we found that sparsifying the very first block tends to be problematic.
We see Figure 10 (two right panels) that more MoE layers not always better even on a per step basis; see Figure 12 for both upstream and downstream metrics. Looking fixed value of the r-axis Figure 10 (right panels), we conclude that something between Last-5 and Last-6 (40-50% of layers sparsified) offers the most attractive trade-off in this case.
19

---

Published as a conference paper at ICLR 2023
54
JFT-300M Precision@) (%)
ImageNet 10-shot Acc (%)
76.0
75.5
52
75.0
74.5
Upcycling
74.0
Original dense model
50
32
32 Number of Experts
Number of Experts
Figure 1I: Final upstream and downstream performance for upcycled B/16 vision models with different number of experts per MoE layer. The number of MoE layers is fixed at 6. The upcycled model is trained for an additional epochs (from 14 to 21) relative to the original dense model. The dashed horizontal lines show the performance of the dense model when trained for additional 7 epochs.
JFT-300M Precision@1 (%)
ImageNet 10-shot Acc (%))
52.5
52.0
75.0
51.5
51.0
74.5
50.5
74.0
50.0
Dense Model
11 10
8
5 4 2
11 10 8
5
2
MoEs placed in Last-N layers
MoEs placed in Last-N layers
Figure 12: Performance as function of the number of MoE layers for upcycled B/16 models (C = 1) trained for 7 additional epochs on JFT, starting from dense checkpoint originally trained for 14 epochs. MoE layers are consecutively placed starting from the last block. We train models ranging from MoE layer (L to 11 MoE layers (Last- i.e. all but the very first. The dashed horizontal lines show the performance of the dense model when trained for an additional 7 epochs.
B.5 EXPERT INITIALIZATION
The standard upcycling recipe copies and replicates the dense MLP each expert. As the router directs different tokens to each expert, the experts will start to diverge from one another, and their initial MI.P weights. Figure 13 explores whether loading the MLPs indeed good idea, or whether the model would be better off learning the experts from scratch (random initialization). We train for 7 extra epochs (dense was trained for epochs, and we keep training up to a total of 21). Note that the computational cost of both approaches is identical.
takes a long time for the model with randomly initialized experts to recover and catch up with the algorithm that upcycles the expert weights from the dense MLP layers, regardless of the number of experts. We also tried an intermediate approach (not shown). where we only upcycle a subset of experts and initialize the rest of scratch, but that also underperformed upcycling all of the experts.
20

---

Published as a conference paper at ICLR 2023
8 experts
32 experts
128 experts
FT-300M Precision (%)
0.50
0.45
0.40
0.35
0.30
Load Experts False
0.25
Load Experts True
0.80
ImageNet 10 shot Acc (%))
0.75
0.70
0.65
0.60
0.55
Experts False
Load Experts True
0.50
100k
300k 400k
600k
100k 300k 400k 600k Extra Training Steps
100k 200k
300k 400k 500k 600k
Figure 13: Performance comparison between upcycling experts ("Load Experts = =True") and randomly initializing the experts ("Load Experts False"). We include upstream (top row) and downstream (bottom row) performance metrics, and also ablate the number of experts per MoE layer (over the columns).
B.6 RESUMING THE OPTIMIZER STATE
When upcycling model, together with the model parameters. Figure [4 shows that reusing the optimizer state gives a performance boost for vision models, independent of the number of experts We did not, however, see any improvement from reusing the dense model optimizer state in our language experiments, so we only reuse the optimizer state for vision models.
resume the optimizer state from the original dense checkpoint
B.7 COMBINE WEIGHT NORMALIZATION AFTER ROUTING
simple trick that found useful for the upcycling of vision models was normalize the combine weights after routing. The weights of each token are normalized so that the sum is 1. This follows that intuition that each token was previously only processed by single "expert" MLP in the dense model. In the event that token is not routed at all, the combine weights remain 0.
We illustrate this normalization trick with two simple examples.
Several experts selected. Suppose a token r is selected by three different experts e1, and e3 with routing weights wi 0.3, wa 0.2, and W3 0.1 respectively (adding up 0.6).
The normalized weights are:
0.3
0.2
0.1
W
0.5,
W3 0.6
00.3333..
W3
0.1666...
0.6
0.6
The final output o' is:
x' Wi e1 (x) + W2 e2(x) + 03 ex(x).
For some parameter, such as the router weights, do not have any original optimizer state that
21

---

Published as a conference paper at ICLR 2023
experts
32 experts
128 experts
JFT FT-300M Precision (%)
0.40
0.35
0.30
0.25
Load Optimizer False Load Optimizer True
0.20
ImageNet 10-shot Acc (%)
0.75
0.70
0.65
0.60
Load Optimizer False
Load Optimizer True
0.55
100k
300k 400k 500K 600k
0 100k 300k 400k 600k
100k 200k 300k 400k 500k 600k
Extra Training Steps
Figure 14: Performance comparison between reusing ("Load Optimizer True") and not reusing ("Load Optimizer = False") the optimizer state. We include upstream (top row) and downstream (bottom row) performance metrics, and also ablate the number of experts per MoE layer (over the columns).
Table Training from scratch on V-MoE-B/32 vision models with Expert Choice routing. Comparison with and without weight renormalization after routing.
Capacity
Renormalization
Val Prec@1
ImageNet 10shot
C ==1
No
48.71
69.68
C=1
Yes
48.23
70.19
C
50.02
71.26
C
Yes
49.75
71.55
Only one expert selected. In this case, regardless of the selected weight W1, the output routing weight will be W, 1.0 after normalizing it:
r' wi en (x) 1.0 (x)))
While this approach can be in principle bit problematic (those tokens only selected by one ex- pert have vanishing routing gradients). Table shows that, even if we are training vision models from scratch, applying weight normalization does not hurt performance (while indeed helps for upcycling).
However, router weight normalization was not helpful for language models. Upstream accuracy after IM steps was comparable: 70.8% (no normalization) 70.7% (normalization), but downstream average scores on SuperGLUE lagged: 79.3% (no normalization) vs 78.8% (normalization). A similar quality degradation were observed in MoE language models trained from scratch. One hypothesis for this different behavior is that the vision models use Expert Choice routing everywhere, but the language models use Expert Choice in the encoder and Top-K routing in the decoder.
22

---

Published as a conference paper at ICLR 2023
B/16 models
1/16 models
TLSVRC2012 10shat Accuracy
0.70
12SVRC2012 10shat Accuracy
0.65
0.76
0.60
0.74
0.72
0.50
0.45
0.70
0.40
Capacity ratio C
Capacity ratio C
ExpChrose (Norm)
ExpChoose
Top K=1
Top K=2
Dense
Figure 15: The effect capacity size ratio on the initial performance of B/16 (left) and L/16 (right) models after upeycling (i.e. at the very first new step); when routing weights are normalized (Section B.7) and capacity is large, the upcycled model retains the dense model's function.
B.8 CLOSER L.OO AT DESIGN CHOICES IMMEDIATELY FOLLOWING UPCYCLING
Once model is upcycled, there is drop in performance due the sudden deviation from the dense model's learned function, even with MoE experts reusing the dense MI.P weights. Routing design decisions make significant difference in ameliorating this drop.
Section B.2 ablates the long-term upcycled model performance as a function of the capacity factor C. Figure| 15 shows the immediate effect of modifying C' on the upcycled model. Increasing C reduces the Tikelihood that tokens are dropped; when routing weights are normalized to sum to (see Section B. the upcycled model is exactly equivalent the dense model for those tokens selected by at least one expert. Note the set of tokens not selected by any expert decreases in size as we increase C.
Note that although different routing mechanisms significantly impact the starting point for upcycling, the subsequent training can smooth over many differences. For example, the start Top-K routing clearly worse than Expert Choice with weight normalization, but Table shows that a model upcycled with Top-K routing eventually catches up.
Figure shows the effect of the group size parameter. Routing, and in particular the top-k operations, are performed in groups Using smaller group sizes will speed up routing, but will lead to higher variance in expert assignment across groups. For smaller groups, we may expect more tokens to be dropped (not selected by any expert).
How many MoE layers we upcycle, and where we place those layers, also affects the initial per- formance drop. Figure 17 shows the effect of this in the initial drop for Expert Choice routing, using normalized combined weights (Section B.7) Upeycling the bottom layers causes larger initial performance drop. Upcycling the last layers consecutive layers or interleaving them -as in every other layer- yields the smallest initial performance drop.
Finally, we analyze how the number of experts per MoE layer affects the initial upcycling. Figure [8 suggests that routing to more experts leads to heavier drop. Figure 12 shows that upcycled models can recover from this eventually though, and sometimes even achieve higher performance.
B.9 THINGS THAT DID NOT WORK
We list several unsuccessful attempts, beyond the preceding ablations, to improve the performance of the upcycling. Adding (truncated) Gaussian noise to router weights, expert weights or both, in an effort to diversify the router or expert weights, did not help. Modifiying the learning rate experts, routers or both, to account for the fact that the addition of sparsity may demand different learning rate from the rest of the model, generaly hurt performance; increasing the learning rates sometimes rendered the models
unstable. We attempted to add fixed temperature term to the softmax of
23

---

Published as a conference paper at ICLR 2023
B/16 models
L/16 models
ILSVRC2012 10shot Accuracy
0.70
ILSVRC2012 10shot Accuracy
0.77
0.65
0.75
0.74
0.35
0.73
0.72
0.50
0.71
0.45
0.70
1 1.000
2000
3000
1000
2000
3000
Group size
Group size
ExpChoose (Norm)
ExpCboose
Top-K=1
Top-X=2
Figure 16: Increasing group size does significantly affect performance using Expert Choice routing, but improves initial performance for Top-K routing.
B/16 models
L/16 models
1LSVVRC2012 10shat, Accuracy
0.70
11SVRC2012 10shat Accuracy
0.65
0.60
0.55
0.40
0.35
0.30
15
25
Number of MoE layers
Number of MoE layers
Last
First
Interleaved
Dense
Figure 17: Effect of position and number of MoE layers on the initial performance after upcycling (i.e. the very first new step). Note that L/16 models have more MLP layers..
B/16
1/16 models
ILSVRC202 10shot Accuracy
0.70
ILSVRC2012 10shot Accuracy
0.77
0.65
0.76
0.75
0.60
0.74
0.50
0.45
2
64
32
64
Number experts
Number of experts
ExpChoose (Norm)
ExpChoose
Top-K-1
Top-K=2
Dense
Figure 18: Effect of the number of experts per MoE layer on the initial drop of ImageNet 10-shot performance (i.e. step 1), for B/16 and L/16 upcycled models with MoE layers and 1. The initial performance is lower when using more experts.
24

---

Published as a conference paper at ICLR 2023
router to encourage more random or less random routing. We also varied the initialization scheme for the router, included the router z-loss (Zoph et al. 2022). Unfortunately, none of these attempts led to any significant performance improvement.
25

---

C SELECTED RESULTS
Published as conference paper at ICLR 2023
Table 4: Selection of results vision tasks for different methods and architecture variants. We report the JFT-300M Validation Precision at (%), the ILSVRC2012 10shot Accuracy (%), the ILSVRC2012 Accuracy after Finetuning (%), and the extra TPUva-coredays and ExaFLOPs used, both in absolute and relative (%) terms with respect to the corresponding dense checkpoint (when these they correspond to the initial dense checkpoints).
ILSVRC2012
ILSVRC2012
Extra
Relative Extra TPUv3-days
Extra ExaFLOPs
Relative Extra
Method
Variant
JFT-300M
10shot
Finetune
TPUv3-days
ExaFLOPs
Dense
B/32
37.78
61.92
78.91
0.00
0.00
0.00
0.00
Dense
L/32
44.69
70.97
83.08
0.00
0.00
0.00
0.00
Dense
B/16
46.67
72.19
83.77
0.00
0.00
0.00
0.00
Dense
L/16
53.54
78.63
86.50
0.00
0.00
0.00
0.00
MoE
B/32
20.24
39.82
66.95
1.91
14.60
4.02
14.64
Upeycling
B/32
38.73
63.01
79.58
2.53
19.31
5.32
19.36
MoE
L/32
23.76
44.63
69.11
5.70
14.11
13.79
14.36
Upeycling
L/32
45.33
71.28
83.35
5.78
14.31
13.15
13.69
26
MoE
B/32
38.77
63.24
78.80
9.57
73.01
20.12
73.19
Upcycling
B/16
47.81
73.19
84.13
12.36
13.54
28.31
12.68
Upcyeling
B/32
46.34
69.98
82.18
21.60
164.88
45.43
165.30
Upcycling
L/16
53.89
78.78
86.56
33,99
10.87
81.36
10.40
Upcycling
L/32
51.19
74.43
84.66
49.95
123.58
113.63
118.29
Dense
B/16
48.12
73.28
84.23
53.19
58.26
130.04
58.26
MoE
B/16
50.24
74.96
84.57
73.58
80.59
168.49
75.49
Upcycling
B/16
52.59
76.48
85.67
85.94
94.13
196.80
88.17
Upcycling
B/16
54.42
77.39
86.03
159.52
174.73
365.28
163.66
Dense
L/16
54.91
79.17
86.68
182.16
58.26
455.85
58.26
Upcyeling
I/16
56.71
79.70
87.10
236.27
75.57
565.58
72.29
Dense
L/16
55.65
79.46
86.48
338.49
108.26
847.05
108.26
Upcycling
L/16
57.84
80.01
87.17
438.55
140.27
1049.79
134.18
Dense
L/16
56.19
79.63
86.89
494.81
158.26
1238.24
158.26

---

Published as conference paper at ICLR 2023
Table Selection of results on text tasks for different methods and architecture variants. We report the C4 Validation Token Accuracy, the individual metrics on each and ExaFLOPs used, both in absolute and relative (%)
SuperGLUE task after finetuning, well the overall SuperGL.UE score, and the extra terms with respect the corresponding dense checkpoint (when these are they correspond the initial dense checkpoints).
SuperGLUE Score
Extra
Relative Extra TPUv4-days
Extra
Relative Extra
Method
Variant
C4
BoolQ
CB
COPA
MultiCC
ReCoRD
RTB
WiC
WSC
TPUv4-days
ExaFLOPs
ExaFLOPs
Dense
Basc
67.97
82.48
94.64 / 93.69
70.00
38.30/ / 76.52
77.26 77.26/78.24
82.31
69.59
83.65
77.17
0.00
0.00
0.00
Donse
Large
71,80
96.43
87.00
54.35 84.35
84.84/ 85.78
92.06
75.55
87.50
85.06
0.00
0.00
0.00
Dense
XI.
74.15
0.00
0.00
0.00
0.00
Donse
Base
68.20
81.77
92.86 / 91.79
70.00
37.88 / 76.04
77.25 /7225/78.00 78.15
82.67
67.55
81.73
76.34
13.33
40.00
55.15
40.00
Dense
Basc
68.33
83.00
94.64 13.61
72.00
38.93 / 76.82
77.46/78.49
84.84
67.71
78.85
77.05
19.99
60.00
82.73
60.00
27
Upcycling
98.21
80.90/
82.67
87.50
79.23
20.31
82.14
59.57
MoE
Base
69.52
79.66
91.07 89.08
63.00
30.12
76.36 77.48
79.78
69.75
84.62
74.45
20.33
61.01
82.14
59.57
Upcycling
Base
70.22
84.04
100.00 100.00
76.00
40.08 / 78.20
82.06 83.05
83,39
68.03
84.62
79.72
30.47
91.47
123.21
89.36
MoF.
Base
70.12
80.34
96.43 / 97.38
72.00
30.33 73.87
77.43
1 78.66
80.51
70.06
84.62
76.82
30.49
91.52
123.21
89.36
Dense
Base
/33.21
77.27
77.55
78.46
83.03
69.91
82.69
77.36
33.32
100.00
Upcycling
Base
70.83
84.31
98.21 198.88
79.00
39.77 1
83.02
84.
83.39
69.59
81.73
79.86
50.79
152.45
205.35
148.94
Dense
Large
72.03
88.81
96.43 /94.3
89.00
56.87 / 85.49
87.65
88.44
90.97
73.20
90.38
85.87
123.54
40.00
637.72
40.00
Dense
Large
72.12
88.81
96.43 /94.30
88.00
85.54 56.56
87.78 88
90.25
73.35
90.38
85.67
185.42
60.03
956.58
60.00
Upcycling
Lurge
73.34
88.62
96.43 95.00
90.00
55.93 85.22
87.65
88
90.61
72.26
92.31
86.04
197.38
1076.51
67.52
Upcyeling
Large
73.68
100.00 100.00
90.00
85.04
88.58
89.62
92.78
72.57
90.38
86.74
287.48
93.08
1614.73
101.28
Dense
Large
72.26
88.38
96.43 / 94.30
87.00
57.08 / 85.43
87.83
88.57
91.34
72.57
87.50
85,20
309.04
100.06
1594.30
100.00
Dense
XI.
74.34
508.73
40.00
2369.07
40.00
Dense
XL
74.46
1050.73
82,62
3401.70
57.44
Upcyeling
XL
75.03
1405.57
110.52
4985.16
84.17


Reviews:
[{"review_id": "wvTImMsRDjk", "reviewer": "Reviewer_36d4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach for initializing sparse Mixture of Expert (MoE) models from dense checkpoints. The approach called upcycling reuses an already trained dense checkpoint by copying all the parameters from the original checkpoint (except for the MoE router parameters). The experts in this new model are replicas of the original MLP layer(s) in the dense model. The experiments demonstrate that the upscaled models outperform the sparse models trained from scratch when using the same computation budget as the initial dense pretraining. ", "summary_of_the_review": "Overall, I like the idea and the fact that the paper covers a relatively comprehensive set of experiments for both vision and language domains. ", "strength_and_weaknesses": "The paper is generally clear and easy to follow. The problem of using already trained dense checkpoints for initializing sparse models is important, especially given the promising results of sparse MoE models in different domains. I found the idea of replicating the MLP layer(s) and randomly initializing the routing mechanism simple yet powerful. I have the following questions and comments:\n1. How do the distribution of token to expert assignment look like? Are we seeing few experts which dominate the probability distributions? \n2. How many experts are involved in processing each token (on average)?\n3. How do the self-similarities and intersimilarities of inputs between experts look like?\n4. The related work, is missing some works on converting a dense model to its corresponding MoE version. See for instance:\n    1. MoEfication: Transformer Feed-forward Layers are Mixtures of Experts \n    2. MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation\n    3. DEMIX Layers: Disentangling Domains for Modular Language Modeling (not directly related but uses a similar idea of replicating the MLP units). \n5. It would be nice to add a comparison to one of the aforementioned approaches to show the advantages of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pMIA3yQU4I", "reviewer": "Reviewer_zFGk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel yet simple method to reuse pretrained dense Transformer checkpoints to initialize larger sparse\nmodels. \nThe authors showcase that using this technique will result in a more performant MoE model versus continuing training the dense counterparts on vision and language domains up to 3B scale. \nExtensive ablations are provided for the design choices. \n", "summary_of_the_review": "it is no doubt that sparse upcycling tries to leverage the inference & scaling benefits from MoE and pre-trained weight from a smaller Dense model, and it achieves better results given the more introduced capacity; \n\nThe major concerns for me are the inherited instability of training MoE also appears/worse in upcycling and will that affect the initial transition and how will the initial transition affect the end performance; \n\nMore details for the hyperparameter sensitivity will also be valuable to add. ", "strength_and_weaknesses": "Summary Of Strengths\n- the paper is clearly written and presented;\n- the efficacy of the sparse upcycling is verified both across domain (vision & language), transfer (downstream) and scales (up to 3B); \n- extensive ablations are provided (router, #dense pretrain, #MoE layers, #experts, MoE locations, and initialization) to illustrate the design choices, which may pave the road for future researchers and practitioners in the efficient computing area. \n\nSummary Of Weaknesses\n- Fairer comparison: it is clearly presented in figure 2 that a sparsely upcycled model outperforms a continual dense counterpart on vision, and language pretraining. However, the sparsely upcycled model also has more capacity, I am wondering if will it be a fairer comparison for:\n\n  (i) upcylce a dense model to a large model (though it will introduce more inference cost) as in [1,2,3,4], especially [1,2] show that staged warm starting may also help the efficiency of training; \n\n  (ii) continue to train a MoE model as shown in Figure 4, and why the Appendix Table 2 seems to disagree what presented in Figure 4?\n  7+7 extra epochs of MoE models seem to outperform a sparsely upcylced model with 7+ extra epochs. \n\n- Why continual training of a dense model turns to hurt the downstream performance on SuperGLUE; \n- How sensitive are the design choices like in the following will affect (1) the initial transition; (2) the ending performance? \n\n    (i) the learning rate schedule: resuming/restarting [1], (also in Appendix A.1.1 suggest using a learning rate schedule where the dense checkpoint left off, does that suggest you will initially plan for more epochs, what if the learning rate has already decay to 0)\n\n    (ii) MoE losses: [5] uses half loading balance + half importance and strong regularization like noise jitter and larger capacity ratio to make the V-MoE stable. Will all the techniques (except for the BPR) also be used here and will all these hyperparameters also be sensitive. \n\n- the intuition in Figure 4 that why will a upcycled model will outperform continue-training MoE\n\n   In Figure 3 and Appendix that the initial transition of the sparsely upcycled model underperforms a dense model, which I assume will also underperform a MoE model. Then using the same schedule, will the sparsely upcycled model consistently underperform a MoE counterpart that trained from scratch?\n\n\n- Is it possible to upcycle MoE models?\n\n[1] Rae, Jack W., et al. \"Scaling language models: Methods, analysis & insights from training gopher.\" arXiv preprint arXiv:2112.11446 (2021).\n[2] Shen, Sheng, et al. \"Staged Training for Transformer Language Models.\" ICML 2022.\n[3] Gu, Xiaotao, et al. \"On the transformer growth for progressive bert training.\" NAACL 2021.\n[4] Gong, Linyuan, et al. \"Efficient training of bert by progressively stacking.\" ICML 2019.\n[5] Riquelme, Carlos, et al. \"Scaling vision with sparse mixture of experts.\" Advances in Neural Information Processing Systems 34 (2021): 8583-8595.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IzLVBLJaiX", "reviewer": "Reviewer_vrXq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes sparse upcycling -- copy the model into a sparse model for faster and more efficient training. They conduct experiments on JFT-300M and English C4 dataset. The empirical results show its superiority.", "summary_of_the_review": "see above", "strength_and_weaknesses": "The paper proposes a training strategy to make MoE training faster. The proposed sparse upcycling is very reasonable. The experiments also show the superiority of this training strategy. \n\nI have the following concern and doubts:\n\n1. The proposed sparse upcycling only increases training speed, but not performance. As shown in Figure.4 right, upcycling and moe from scratch would finally converge.\n\n2. The proposed method seems to take as long as the moe from scratch to finally converge. In Fig.4 and Fig.5, the training curve is not finally converged so it occurs to me that sparse upcycling are just faster in the early stage but could take as long as the moe from scratch to converge.\n\n3. The training speed still seems to be at the same level as training a normal moe. The improvement in time is not a game changer.\n\n4. In the ideal case, MoE only increases the params size but not computation cost or computation speed. (which depends on topk) So the dense model training time should be similar to the sparse model training time even though the sparse model is much larger. I am not sure if that really matters to increase the training speed. A demonstration of the relation between params size, computation cost, and the training speed using normal training or the proposed upscale training is missing from the paper. In another word, under what circumstances (params size, FLOPS) do we care about the training speed, and how much this method can improve?\n\nAbove all, the method is simple and solid. My major concern is that the speed improvement does not seems to be significant and crucial in a real application and the paper does not demonstrate why this is useful. I am also not sure if upcycling is on a different speed level of a normal MoE with decent optimization. \n\n\n\n----------Afrter Rebuttal---------------\n\nI have read the response from the authors. I still have doubts about the speed improvement: not likely to be significant and mostly comes from the slow MoE implementation. I would keep my rating.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0N36mp_NRk", "reviewer": "Reviewer_puDi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an interesting and useful idea for reusing pretrained dense checkpoints to initialize larger sparse models for better performance on different vision and language tasks. This method helps to improve performance with some additional computational budget but does not require training large sparse models from scratch. The paper conducts analysis on T5 and ViT models with various sizes and discusses different design decisions for upcycling.", "summary_of_the_review": "Overall, this is an interesting work which shows that we can get improved performance by initializing sparse models with dense pretrained checkpoints when there is additional computational budget available. Initialization of sparse models from dense checkpoints is not new to the community but this work shows some good design choices which are important and will be useful. Hence I recommend this rating, and look forward to the author discussion phase.", "strength_and_weaknesses": "### Strengths\n- Sparse upcycling clearly shows improved performance for both vision and language pretraining, with significant improvements when some non-trivial computation budget is available for further training.\n- importance of this paradigm is in general great to reuse/recycle old models\n- Performance improvement is fairly consistent irrespective of how long the dense pretraining was done, which is promising.\n- Authors discuss various design choices for upcycling clearly. Ablation experiments are thorough and clearly justifies different design choices.\n\n### Weaknesses\n- Initializing MOE models from dense checkpoints is not novel and is somewhat a known practice. \n- Figure 3 is missing the finetuning performance for XL models for language tasks. \n- The performance on downstream language tasks has a lot of variance which increases with larger models and somewhat seems that upcycling does not help in SuperGLUE tasks to a large extent. Is this dependent on downstream tasks? Is this behavior same on other downstream language tasks as well?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster