M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
Authors: Junyang Lin
An Yang
Jinze Bai
Chang Zhou
Le Jiang
Xianyan Jia
Ang Wang
Jie Zhang
Yong Li
Wei Lin
Jingren Zhou
Hongxia Yang
Venue: ICLR 2022 Submitted (2022)
Keywords: Extreme-Scale Pretraining
Language Modeling
Natural Language Processing

Abstract: Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters. However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory footprint suffers from frustratingly low efficiency in model convergence. In this paper, we propose a simple training strategy called “Pseudo-to-Real” for high-memory-footprint-required large models. Pseudo-to-Real is compatible with large models with architecture of sequential layers. We demonstrate a practice of pretraining unprecedented 10-trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the application of Pseudo-to-Real, we also provide a technique, Granular CPU offloading, to manage CPU memory for training large model and maintain high GPU utilities. Fast training of extreme-scale models on a decent amount of resources can bring much smaller carbon footprint and contribute to greener AI.

Under review as a conference paper at ICLR 2022
10T: A SHARING-DELINKING PARADIGM FOR EF- FICIENT MULTI- TRILLION
MULTI-TRILLION PARAMETER PRETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters. However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory foot- print suffers from frustratingly low efficiency in model convergence, In this paper, we propose simple training strategy called Secudo-to-Reil for high-memory- large models. Pseudo-to-Reas with architecture of sequential layers. We demonstrate practice of pretraining unprecedented Atrillion-paramett model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the ap- plication of Pscudo-to-Real Reaal,, we also provide technique, Granular CPU offloading, to manage CPU memory for training large model and maintain high GPU utilities. Fast training of extreme-scale models on decent amount of resources can bring much smaller carbon footprint and contribute greener AI.
compatible with large models
INTRODUCTION
Foundation models with self-supervised learning on big data have become an emerging paradigm of artificial intelligence systems [2]. as they mostly possess high transferability to wide range of downstream tasks and even multiple modalities. The scale of foundation models across domains, including natural language processing, computer vision, and cross-modality representation learning, have been growing tremendously from millions to trillions of parame- 33 37
ters [3 24
26
29 14, 16, 47 46
42
44
(38 thanks to the concurrent advancement
30 and hardware design, and these studies However, the training of these transformer- based models incurs high financial costs and even environmental damage due to the massive carbon footprint and thus training extreme-scale models under decent amount of resources but with high efficiency should be fundamental goal for both the research and industrial communities to achieve, which promotes the progress of greener Al 22
in distributed training framework 20 31
28
I
12
have made demonstration of the neural scaling law [3]
Generally there are two tracks of research in large-scale pretraining, dense models and sparse expert models respectively. A typical case of large-scale dense models is GPT-3 (3). 175-billion- parameter transformer model trained with 10, 000 GPUs for months, incurring striking financial and environmental costs. Researchers have been searching for methods training large-scale models with decent amount of costs. Solutions include effective management of memory with gradient and optimizer state partitioning 22 or more efficient model parallelism and pipeline
parallelism [37 20 [2]. A series of following studies apply those techniques realize fast training of 10-billion parammerr transformers with hundreds of GPUs
months. [6
Sparse expert models with large model capacity capable of fast training owing to the combination
of data parallelism and expert parallclism [35] 14
11 32), and is even accessible to train a
1-trillion- Saalliennparammtte transformer with more than 500 GPUs
42
Be there as it may, a question emerges in our mind: it possible to train an extreme- scale model with only a decent amount of resources. e.g., training GPUs? Tackling the problem requires the utilization of external memory except for GPU memory, for instance, CPU memory even NVMe storage 31 28 These methods resolve the problem of
model with around 500

---

Under review as a conference paper at ICLR 2022
high memory footprint, but instead, their extra cost is low training efficiency caused by the frequent swap on-and-out between memories.
In this paper, we provide solution to training large models that require high memory footprint, and we demonstrate a successful practice of pretraining an unprecedented extreme-scale model with over 10 trillion parameters, an order of magnitude larger than the previous ofte-of-the--rts [1] 2 The whole pretraining was conducted on solely 512 NVIDIA-VII V100 GPUs and lasted around 10 days. simple and effective training strategy called Prseudo-to-Real enables the sharing and delinking of parameters. This training strategy is compatible with architectures built by stacking layers of an identical structure, including dense models like GPT [23 24 3], BERT 7. or sparse expert models like M6 [6 42) It is essentially a two-stage training. It first trains a relatively small model but with a computation graph of a large one with the utilization of cross-layer parameter sharing, and we name "Pseudo Giant". Then it builds a correspondingly large model and delinks the parameters of the shared layer for second-stage model initialization. In this way, we achieve fast convergence in the first stage as the training costs much less memory and speeds up with large batches. Parameter sharing that addresses the communication overhead improves training speed as well. The second-stage training is responsible for the final convergence for better performance.
We unlock the secret of pretraining an unprecedented sxtreme-scale model with over 10 trillion parameters limited resources of 512 GPUs. Compared with the previous M6-T on around 500 GPUs, we do not have a significant increase in computation resources but level up the model scale by order of magnitude. Besides the application of the Seeuudoto--Ral" training strategy, provide a faster offloading mechanism for both management of CPU memory for parameter storage and utility of GPUs. We successfully train the M6-10T within 10 days to reach strong performance in log perplexity evaluation and outperform the baseline M6-T.
Contributions at a glance are below:
We illustrate the training difficulty of extreme-scale models on limited resources and pro- vide a simple but effective solution called "Pseudo-to-Real Upstream and downstream evaluation demonstrates the effectiveness of the strategy.
We further demonstrate a successful practice of pretraining a 10-trillion- parammeer model on 512 GPUs and reach an outstanding performance within 10 days.
RELATED WORK
Large-Scale Pretraining In recent years, pretrained language models with growing magnitudes of parameters have been proposed, keeping to raise the validated upper limit of scaling law for model capacity w.r.t the number of parameters 13. Earlier milestones of extreme-large models come from GPT-2 24 and Megatron-LM (37). which demonstrates that scaling the transformer model up to billions of parameters can result in improvement on language modeling benchmarks [19 21 Turing-
NLG [3), as a successor, implements 17-billion-parama parameter transformer and achieves further lower perplexity. Similar phenomena are also observed on classification language tasks by T5 model [26]. The GPT-3 3 pushes the boundary of model scale to 1,750 billion parameters and demonstrates its striking effectiveness on downstream tasks in even zero-shot settings. Furthermore, large-scale pretraining has recently demonstrated success in other fields, including pretraining on other languages or cross- looss-lingua pretraining [41: 46 44 6). gross-modal pretraining [25) 29; 16 g 48) and code
generation [4] Along with the benefit from increasing the model scale, the concern of unaffordable
pretraining cost in time, computation resource and energy keeps emerging 22 2, resulting in the
strong demand for more efficient and greener large-scale pretraining
Methods Train Larger Models Faster Our work is about designing algorithm to train extreme- scale models efficiently. Researchers have been demonstrating different types of methods to reach this objective. To reduce the computational cost during training, some studies introduce sparsity to the model. Mfxture-of-Expert (MoE) [35] was proposed and was reintroduced in Mesh Tensorflow (36). shows that MoE with sparsity can significantly improve the model scale efficiently without increasing computation, and the models achieved state-of-the -hetthe-rt performance in language modeling and machine translation. GShard [4] extended it to a tremendously large scale 600B parameters with the sophisticated collaborated design of model architecture and demonstrated its effectiveness
2

---

Under review as a conference paper at ICLR 2022
Add LayerNorm
Feed-Forward
Add LayerNorm
Self attention
Share Weights for Pseudo Stage
Delink Weights Stage
Add LaverNorm
Feed-Forward
Add & LayerNorm
Self attention
Figure I: A demonstration of the Pseudo-to-Red training paradigm. It first shares parameters across layers at the Pseudo stage, and then delinks the parameters at the Real stage.
across over 100 languages. Similarly, Switch Transformer reached 1.6 trillion parameters and showed its strong performance on NLU tasks. Those models with high sparsity are computationally efficient, and therefore though they possess large model capacity they still can be trained with high efficiency. Most other studies still focus on training dense models to validate the scaling law, and thus the emerged problem is the distributed training of large models. The most influential distributed framework should be DeepSpeed that proposed ZERO 27 that partitions optimizer states and gradients to multiple 3T as well as ZERO-Infinity
GPU devices, and ZERO-offload
(28) can even offload parameters to CPU memory and NVMe storage. The memory footprint management makes training extremely large models on limited resources possible. However, such offloading mechanisms still have some defects that they may fail to fully utilize the fast hardware. For example, when offloading all parameters to the CPU, the GPU memory can be idle. In this work, we tackle this issue by proposing a granular offloading mechanism that can determine which parameters be offloaded.
In addition to reducing the amount of computation in single iteration, another route to speedup training reduce the needed iterations for model convergence. Child et al. (5) and Xiong al. 40 propose to put forward the layernorm operations in transformer blocks for more stabilized and faster convergence. You al. 43 employs layer-wise adaptive optimizer to enable super-large pretraining batches. Zhang & He 455 progressively increases the layer dropping rate in a stochastic manner which significantly speedups pretraining.
APPROACH
3.1
MODEL ARCHITECTURE
Choice in model architecture depends on several factors. First, the architecture should contain sequence of stacking layers, as the sequential structure enables cross-layer parameter sharing, We prefer a simple encoder or decoder architecture, instead of an encoder-decoder framework where there are cross attentions that bring extra parameters and incur difficulties in activation checkpointing. Second, a model of such architecture should be compatible with different types of downstream tasks, including understanding and generation, and it is more preferable that it can be compatible with multiple modalities. Third, as we mention that dense models and sparse expert models are two main tracks of scaee pretraining, sparse expert models. Therefore, we select M6 [6 as an option, and we evaluate the effects of our method on M6 of different scales and types.
prefer the model that flexibly become whether dense or
3

---

Under review as a conference paper at ICLR 2022
M6 is built with stacking transformer layers, which includes self attention and ferwaro neural nets (FFN), For the transformation from dense models to sparse expert models, we should only replace FFN layers with the Mixture-of-Expee which are usually FFNs distributed on different devices. gating network decides the dispatching and combining behaviors of each token and thus tokens can be processed in diverse devices. Such mechanism is combination of data parallelism and expert parallelism, and thus it is highly efficient though with large model capacity. For the training, realize the learning of both understanding and generation, the model is trained with text denoising and language modeling on plain text data and with image-basee based text denoising and image captioning on multimodal data. The model is compatible with different types of downstream tasks and can process information of multiple modalities.
(MoE) layers. MoE consists of multiple experts,
3.2 PSEUDO-TO-REM
This section demonstrates the details of "Pseudo-to-Rer two-stage training strategy that enables fast training of memory-footprint-reee
transformer models. The strategy consists of two stages. The first stage trains a model with many fewer parameters but with a large computation graph ("Pseudo Giant"), and the second stage trains corresponding large model ("Real Giant") initialized with the delinked weights the shared layer. Thus we name the strategy "Pseudo-to-Real and the general idea is demonstrated in Figure
3.2. "PSEUDO" STAGE: LAYER-WISS WISE PARAMETER SHARING
The core of "Pseudo" stage is to train a Pseudo-Gian that shares parameters across layers. Cross-layer parameter sharing has proved successful in maintaining satisfactory performance while keeping much smaller amount of parameters. We introduce it to training an extreme-scale model, and we hypothesize that the first-stage training can gain benefits from cross-parameter sharing it can address communication overhead and it consumes much less memory footprint. Also, as Pseudo Giant with much fewer parameters is not bounded by memory. it can be trained with large batches for acceleration.
Suppose we build M6 model with layers that share parameters across all layers. The Pseudo Giant though consists of computation graph of L-layer transformer, its number of weight parameters and optimizer states should be 1/L. of those of the original one. As the gradients, we can accumulate the gradients of each layer in the backward computation process, and therefore the amount of gradients becomes 2/ L of the original one. Such saving memory enables much faster training with larger batches. Also, is capable to use fewer resources even due to lower memory consumption.
This can also be applied to sparse expert models, as their architecture is stacking transformer layers. However, different from dense models, the MoE partition their weights to all devices, and each token is distributed to a selected expert by a routing function. Thus the models can trained with data parallelism across all devices and expert parallelism where parameters at each device activated. This mechanism though enlarges model capacity significantly without much efficiency loss, yet limits the fiexible usage of GPU resources in different stages, say training and inference. Due to the application of data parallelism, we can implement dense models on different numbers of GPU devices at the training and inference stage. However, as mentioned above, the combination of data parallelism and expert parallelism requires the identical amount of devices at different stages.
To tackle this problem, we specifically design methods for expert merging and partitioning. Due to cross-layer parameter sharing, at the Pseudo stage it is capable of implementing more experts on each device, and available to use fewer GPUs while keeping the total number of experts. We name expert merging for simplicity. At the Real stage, the delinking of parameters with the same amount devices will cause out-of-memory errors. We delink the parameters and partition experts to more devices, e.g., from 256 to 512, and thus there are fewer experts each device where memory is sufficient. Thus we name it expert partitioning. Therefore, now it is available to use different number of GPU devices at different stages for training sparse expert models.
3.2.2 "REAL" STAGE: DELINKING THE SHARED PARAMETERS
We name a large model without cross-layer parameter sharing "Real Giant", in comparison with Pseudo Giant. Both models share identical computation graph, but possess different numbers
4

---

Under review as a conference paper at ICLR 2022
Table 1: Experimental results downstream evaluation of natural language understanding. We evaluate the performance of models on 8 tasks of GLUE dev set except for WNLI following Devlin
al. (7].
Model
#Params
SST-2
CoLA
MNLI
QNLI
QQP
MRPC
RTE
STS-B
Avg
BERT
345M
60.6
86.6
92.3
91.3
88.0
70.4
90.0
84.1
M6
350M
94.2
61.3
86.3
92.3
91.4
91.4
83.4
89.0
86.2
M6 (P)
65M
90.5
46.2
80.8
89.4
89.9
90.1
70.7
85.6
80.4
M6 (P2R)
65M/350M
94.4
58.6
86.6
93.3
91.5
91.5
84.5
89.1
86.2
of parameters. Given a Pseudo Giant fully trained until convergence, we apply the delinking of cross-layer shared parameters to accelerate Real Giant training. There is no need to train a large model from scratch. The model can start its convergence from low perplexity.
Embedding initialization can be directly restored, but the layer weights should be treated specially. In practice, there is only one layer of weights Shared in Pseudo Giant, and there are I. layers of weights {01, ........... ... initialized with ashared. Without further training, this model is equivalent to fully-trained Pseudo Giant.
,0L } in Real Giant. Thanks to their identical structure, each layer of Real Giant can be
This extremely simple training strategy is highly beneficial for the high- gigh-memor--iie ceannory-footprit-rruuuue large models, especially extreme-scale models like the 10- 10-trillion-parre parameter M6. While the first stage of training saves much time for faster convergence, we can use decent amount computational resources in this stage lower efficiency in this stage hecomes acceptable. Therefore, in the practice of training an extreme-scale M6, we apply CPU offloading to utilize CPU memory. Therefore, we can use limited amount of resources, e.g., 512 GPUs, train an unprecedented Ee-triliio-pprammee model efficiently, which is an order of magnitude larger than the state-of-the-ars
3.2.3 TIMING FOR SWITCHING
A question naturally emerges: when should switch from the Pseudo stage to the Real stage? As mentioned above, the greatest advantage of Pseudo stage for training is the significantly faster convergence speed. Yet the performance of Pseudo Giant is bounded by its limitation in the number of parameters. Training Pseudo Giant until convergence apparently incurs much waste of time.
In practice, present simple strategy determine the training step switch from Pseudo Real based on the convergence speed. During the training of the Pseudo stage, we evaluate a training step fixed interval by attempting transfer it into the Real stage and training for a small while (e.g.. 30 minutes). After that, we will revert the model parameters to the evaluated step and continue the training of the Pseudo stage for the same training time as the Real stage. If the decreasing speed of loss in the Real stage surpasses that of the Pseudo stage, we determine the evaluated training step as the best switching point for the next-stage training.
3.3 EXPERIMENTS
In this section, we demonstrate experiments to evaluate the effectiveness the training strategy by evaluating the model quality and observing the performance improvement.
3.3.1 PERFORMANCE EVALUATION
We aim to validate two hypotheses: 1. Pseudo-to- training paradigm can help the model reach competitive performance with the one trained from scratch without parameter sharing; 2. Pseudo-to- Real can effectively accelerate convergence training of large-scale models.
We aim discovering the model quality brought by different pretraining strategies. "Pseudo (P)" refers to training with parameter sharing across layers, and "Pseudo-tor Morrdo-tt-Rees Real (P2R)" refers to the proposed two-stage training of a sharing-delinking paradigm. We pretrain the model BookCor- pus [49] and English Wikipedia (7]. which are corpora with around 16GB of plain texts. Following Radford al. 23 and Lewis al. [15] we use vocabulary of around 50, 000 subwords. Each
5

---

Under review as a conference paper at ICLR 2022
Table 2: Experimental results downstream task evaluation. "#Params" refers the number of parameters. We report the PPL evaluation on WikiText-103 and the ROUGEI, ROUGE-2, and ROUGE-L evaluation on Gigaword.
Model
#Params
WikiText-103
Gigaword
Megatron-LM
350M
16.69
UniLM
340M
38.5/19.5/35.4
M6
350M
16.59
38.8/20. 1/36.0
M6 (P)
65M
28.60
36,9/18. 1/34.3
M6 (P2R)
65M/350M
16.60
38.3/19.3/35.7
Table 3: Model refers to the types of model. dodedl and d refer the hidden size and intermediate size. refers to the number of layers in the computation graph, and refers to the number of transformer layers with parameters. #Heads refers to the number of heads in self attention. #Params refers to the total number of model parameters. We also report their training speed on 48 GPU devices by the number of consumed samples per second.
Model
dmodet
dss
I/L
#Heads
#Params
Speed
Base
1024
4096
24/24
16
350M
650
Pseudo
1024
16384
1/36
16
1.4B
248
Real
1024
16384
36/36
16
1,4B
48
sample consists of sentences from an identical passage, and we use a sequence length of 512 and correspondingly truncate pad the sequence. Following the common practice in pretraining 17. we apply AdamW optimizer
[18] for optimization. To determine the most suitable learning rate of the stages for fast convergence, we have made some preliminary tests and finally used the peak learning rate of 2e-4 for "Pseudo" stage and 8e-5 for "Real" stage, respectively. We use cosine decaying mechanism for learning rate scheduling and a warm-up ratio of 0.001. We pretrain the models until convergence and transfer them to downstream tasks.
For comprehensive analysis, the experiments include natural language understanding and generation tasks. Notably, for NLU tasks, we follow Devlin et al. 77 and validate transfer effects on GLUE (39). For natural language generation, we specify zero-shot language modeling and text summarization to evaluate both models' upstream and downstream quality on generation. Specifically, conduct experiments on WikiText- 103 and Gigaword for both tasks.
Experimental results NLU tasks demonstrated in Table For better comparison, also present the experimental results of BERT with a similar amount of parameters, and it shows that M6 can achieve a better performance the baseline. From Table brings a significant performance downgrade in the tasks. It is fair to believe that the limited amount of parameters hinders downstream performance, though with a large computation graph. However, in comparison, we find that P2R can help the model perform similarly to the one trained from scratch without parameter sharing. Specifically, M6 with P2R can even outperform the baseline 6 tasks, including SST-2, MNLI, QQP, MRPC, RTE, and STS-B. We further validate their performance on zero-shot language modeling and text sumarization tasks. Table Table on both tasks, and we additionally add Megatron-LM 337 and [0] for better comparison. Similarly, Pseudo-only training leads far worse performance, and P2R is also able to help the model achieve similar performance with Real training.
find that Pseudo-only training
demonstratte the model performance
3.3.2 EFFICIENCY EVALUATION
Next, we evaluate the efficiency of training strategies by observing model performance under the condition of identical computation budget.
Experimental Setup To satisfy the requirements of high memory footprint where the two-staged training can make difference in training efficiency, we conduct experiments on 1.4B 1.4B-.-4raaa -arammmett model. We present the details of model configuration in Table 3

---

Under review as a conference paper at ICLR 2022
Table 4: Experimental results of large models trained with limited budget downstream task
evaluation. We report the PPL on WikiText-103 and the ROUGE scores on Gigaword.
Model
#Params
WikiText-103
Gigaword
limited budget
M6-1B (P)
90M
56.79
36.8/17.9/34.1
M6-1B (R)
1.4B
26.80
36.9/18.2/34.2
M6-1B (P2R)
90M/1.4B
23.60
37.3/18.3/34.5
For both Pscudo and Real Giant, we train them with a total batch size of 6, 144. In practice, we micro-batch size of 32 and gradient accumulation step of 4, and we train models 48 NVIDIA- V 100 GPU devices. We compare the model performance with a limited budget, where models have been trained for the same duration of time. Correspondingly, "Pseudo" has been trained for around 25, 000 steps, Real has been trained for around 4, 700 steps, and "P2R" has been trained for 12, 000 steps.
Model Performance Table 3 demonstrates the training speed of the models. We report
their training speed on 48 NVIDIA-VI00 GPU
38
devices with their consumed samples per sec- ond. Training Real Giant model from scratch is highly time-consummngg Pseudo Giant training has advantage of around times of conver- gence speed over Real Giant training. We also observe the loss convergence of both P2R and Real Giant trained from scratch. In Figure we present the development of pretraining language modeling loss, which is the log perplexity, on the time basis. The log perplexity of P2R de- creases much faster than that of the Real Giant, with an advantage larger than 0.3.
37
3.6
3.4
ga3
3.2
3.1
Powido Stnge
2.9
Suage
2k
3k
Training Time (min)
Figure 2: Comparison of pretraining language mod-
cling loss of M6-IB P2R and Real on time-basis.
Experimental results on downstream tasks pre- sented Table are consistent with our hypoth- esis that Secdo-to-Real training speed up training effectively, In the setup of limited budget, the M6 model trained with Pseudo-to-Red Real can outperform the Pseudo Giant and Real Giant both language modeling and text generation.
4 TOWARDS I0-TRILLION-PARANNE
MODEL
Previously, training large-scale models brings tons of challenges to the collaboration algorithm design, distributed training, as well as hardware design, etc. Training a GPT-3 of 175B parameters with combination of data parallelism
over 500GB of data should cost around 300 GPU-years. Later with the emergence of partitioning on optimizer states, gradients, and even weights, GPU memory be fully utilized without performance degradation. Now can
the CPU memory or even NVMe storage to store the parameters, but we have to bear the costs of efficiency. Therefore, attempt tackle the difficulty of extreme-scale model training from the perspective of algorithm design and thus we apply the aforementioned Pseudo-to-Red training strategy to train an extreme- scale model.
4.1 MODEL SETUP
We design a 10-tilllion-prramete M6 model with the combination of existing methods and proposed strategies to demonstrate case of how to train an extreme-scale model efficiently. In comparison with the previous studies of trillion-paramen parameter models [1 42]. this one is almost 10 times larger. To efficiently utilize the memory. we adopt Mixture-oft offtuure-o--Epperts and we replace every FFN layer with the memory-efficient MoE layers. Notably, remove the auxiliary loss that consumes memory and
7

---

Under review as a conference paper at ICLR 2022
FO
Fn-1
Bo
B1
B2
Bn-1
Bn
GPU
An-1
An
Wn-1
Wn
CPU
AO
A1
A2
W2
Figure 3: A demonstration of Granular CPU Offloading mechanism.
demonstrates little effects on model quality, and we follow Yang al. 42] apply expert prototyping for improved model quality and training stability. To be more specific, the hidden size dmmode is 1024 and the intermediate size dys is 9984. The number of model layers is 48. For each MoE layer, there are 10240 experts distributed on multiple devices. We use 80 prototypes of experts based on our experience preliminary experiments. The training batch size per GPU device models on EFI .OPs, a distributed training platform with an advanced server architecture and new network topology [9]. Specifically our models are trained on a cluster of 8-GPU workers connected by RDMA networks with a bandwidth of 100Gb. The CPU memory of each worker is around 750GB. The expert distribution and Granular Offloading strategies are supported by Whale framework 12]
We implement
4.2 GRANULAR CPU OFFLOADING
To utilize CPU memory for larger models with fewer resources, we apply the Granular CPU offloading which has a higher efficiency compared with the conventional CPU offloading. Previously we note that conventional offloading mechanisms offload all parameters, which may fail to effectively utilize GPU. To improve the efficiency of CPU offloading, we propose new CPU offloading mechanism called Granular CPU offloading. The training process is composed of phases including "Forward (Fn)", "Backward(Bn)" and "Apply (An)". Offloading all model parameters to CPU in Fn and Bn requires loading parameters from CPU to GPU memory twice. Activation checkpointing that brings recomputation needs the parameters loaded in Bn. An requires the gradients offloaded from GPU memory CPU memory. Assume the model parameter size is W, the above processes bring parameter movement of size * W.
In offlodding, PCIE is the bottleneck of the whole training process. We observe that when offloading all parameters with recomputation, the GPU memory is idle. We can fill up the GPU memory by selective offloading. leaving part of the model in GPU memory. In this way, the model can be accelerated by reducing across-device memory copy. In our preliminary experiment, with the setting of training& a 48-layer 78B-parameter M6 model on 8 NVIDIA-VIOD V00 GPU devices, the step-time costs 89 seconds when fully offloading the parameters into CPU memory. In comparison, granularly offloading the first 24 layers into CPU memory and leaving the remaining 24 layers on GPU reduces the step-time to only 45 seconds. The significant difference in training step-time indicates that the time-cost of parameter movement between CPU and GPU will dominate the training step-time when offloading is employed, thus the granularity of offloading and the utilization of GPU should be seriously considered extreme-scall pretraining. In addition, offloading the whole model can result in OOM error in the CPU when the model is extremely large.
With Granular CPU offloading, we successfully implement a 10-trillion- (O-tillion-parrmmtte M6 model on solely 512 NVIDIA-V --11--11000 100 GPUs. Furthermore, at the Pseudo stage, a computation graph of 10 trillion parameters only with 256 GPU devices without the utilization of CPU memory for offloading. Thus in practice, we train Pseudo Giant with only 256 GPUs, and
train Pseudo Giant with
8

---

Under review as a conference paper at ICLR 2022
M8-10T
65
Lo0 Perploity
Perplexty
5.6
4.5
3.5
25
Training Time (min)
Training Samples
(a) M6-10T P2R vs. Real on time-basis
(b) M6-10T P2R vs. M6-T on sample-basis
Figure 4: The log perplexity of M6-10T Pesudo-to-Real (P2R) compared with baselines on time- basis and sasple-basis, respectively. (a) Compared with M6-10T Real on time-basis, M6-10T P2R converges much faster due to the sigmificant reduction of time-cost on offloading in the Pscudo stage. (b) Compared with IT-parameter M6-T model on sample-basis though undergoing the Pseudo stage which limits the model capacity, M6-10T still has an advantage in training sample efficiency.
then partition the experts and redistribute them to 512 GPUs. This saves the usage of GPU resources, which is more resource-effiiiet and also more environmentally friendly.
4.3 ANALYSIS
Training Efficiency We pretrain the M6-10T with Pseudo-to-Real training strategy for around 10 days, and we additionally train an M6-10T from scratch without the strategy for around 3 days for comparison. The Real stage training can be facilitated without Out-of- help of Granular CPU offloading, but its step-time is only around 180s. In contrast, the step-time of Pseudo stage is only 14s without the cost of offloading, which greatly boosts the training efficiency of M6-10T P2R. The M6- 1OT with P2R has been trained for 15k steps, but the one from scratch has been trained for solely 1.3k steps. We record the log perplexity of both models trained on the M6-Corpus on the time basis in Figure 4(a) Results show that within the same length of time M6-10T with P2R can outperform the one trained from scratch by large margin.
errors with the
Convergence Analysis We have trained M6- 10T with Pseudo-to- Reall strategy for around 10 days, and the model converges to a low level of log perplexity based on the upstream evaluation on the M6-Corpus. For better comparison, we also show the convergence performance of the 1T- ITT-paramtter M6-T model proposed in the previous work. As shown in Figure [bb] the observation consistent with our intuition that the model with a larger capacity can converge faster on the sample basis, and it should achieve the best performance on language modeling, What leaves open is whether it can positively lead better downstream performance concerning different types of downstream tasks. Finetuning extreme-scale models should be difficult and there is still much room for us to discover the potential of extreme-scall models.
CONCLUSION AND FUTURE WORK
Pseudo-to-Re training strategy is simple and effective way to train large-scale models that are highly memory consuming, and it is also essential to training extremely large models with limited resources with significantly higher training efficiency. We unlock pretraining unprecedented extreme- scale models with 10 trillion parameters with limited resources of 512 GPUs in 10 days. Besides the application of Pseudo-to-Real training strategy, we further provide Granular CPU offloading to enhance GPU utility while breaking the GPU memory wall with cost in efficiency. The advances take a leap towards extreme-scale model training beyond implementation on limited resources. With only few GPU cards, training large models with tens hundreds of parameters has become accessible to many researchers. We believe that this can motivate low carbon dioxide production and encourages the progress of green AI.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This work is highly concerned with large-scale language models and multimodal pretrained models. These models have been pretrained on broad data of plain texts and image-text pairs, which might contain harmful information, such hate speech, terrorism, pornography, etc. We have put much efforts to remove these kinds of data in our datasets by quality evaluation on texts and images. However, this problem cannot be eliminated and ignored, and it is common in the pretraining community. For those models that are not trained on commonly-user public datasets, we will carefully release the model checkpoints before careful evaluation, and also limit the access avoid misconduct.
REPRODUCIBILIIY STATEMENT
This work is generally reprodueible. Following the description in Section 3 researchers can easily im- plement the training strategy on the codebases for pretraining, including Huggingface Transformer Fairseq etc.
REFERENCES
[1] Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, Anjali Sridhar, and Min Xu. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https: /
2021.
[2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108 072558, 2021.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan- -ien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020.
[4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.033:
2021.
[5] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXivv1904. 10509, 2019.
[6] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL 2020, pp. 1440-8451, 2020.
[7] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1171-4186. Minneapo- lis, Minnesota, 2019. Association for Computational Linguistics.
10

---

Under review as a conference paper at ICLR 2022
[8] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. CoRR, abs/2105.13290. 2021.
[9] Jianbo Dong, Zheng Cao, Tao Zhang. Jianxi Ye, Shaochuang Wang, Fei Feng, Li Zhao, Xiaoyong Liu, Liuyihan Song, Liwei Peng, Yiqun Guo, Xiaowei Jiang, Lingbo Tang. Yin Du, Yingya Zhang, Pan Pan, and Yuan Xie. EFLOPS: algorithm and system co-design for a high performance distributed training platform. IEEE International Symposium on High 2020.
Performance Computer Architecture. HPCA 2020, pp.
[10]
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language un- derstanding and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' A'Aché-Buu Bue, Emily B. Fox, and Roman Garnett (eds.), Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1 13042-13054, 2019.
[11] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint wrxiv2201100900 2021.
[12] Xianyan Jia, Le Jiang, Ang Wang, Jie Zhang, Xinyuan Li, Wencong Xiao, Yong Li, Zhen Zheng, Xiaoyong Liu, and Wei Lin. Whale: Scaling deep learning model training the trillions. arXiv preprint arXiv:201 1.09008, 2020.
[13] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
[14] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[15] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, 2020. Association for Computational Linguistics.
[16] Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang. Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. arXiv preprint 2Xiv:2103.00823, 2021.
[17] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint urXiv:1907.
2019.
[18] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[19] Stephen Merity. Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[20] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catan- zaro, et al. Efficient large-scale language model training on gpu clusters. arXiv preprint arXiv:2104.04473, 2021.
[21] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernari,, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset:
Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics.

---

Under review as a conference paper at ICLR 2022
[22] David A. Patterson, Joseph Gonzalez, Quoe V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint AXiv:2104411050 2021.
[23] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Improving language understanding by generative pre-
Sutskever. training. URL https://s3-us tttsss//s-uswwestte w.w.aaooaws...omoma....... hommlppeeni-asee///ssscchoorrrn aanguaguunsuprrvieed/anggggg understanding paper. pdf, 2018.
[24] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference Machine Learning, ICML 2021, pp. 8748- 8763, 2021.
[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
[27] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 1-16. IEEE, 2020.
[28] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning. arXiv preprint arXiv:2104.07857 2021.
[29] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.1200 2021.
[30] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
[31] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang. Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scall model training. arXiv preprint arXiv:2101.06652 2021.
[32] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. arXiv preprint 2xXiv:2106.0444 2021.
[33] Corby Rosset. Turing-nlg: A 17-billion parameter language model by microsoft. Microsoft Research Blog, 2020.
[34] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AL. Commun. ACM, 63 (12):54-633 0015546-322200 2020.
[35] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[36] Noam Shazeer, Youlong Cheng. Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanan- takool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, pp. 10435-10444, 2018.
tesshteensoffoo:: Deep learning for supereomputers. In Advances
12

---

Under review as a conference paper at ICLR 2022
[37] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint 19Xiv:1109.088
2019.
[38] Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint urXiv:210.00213)) 2021.
[39] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA,
USA, May 6-9, 2019, OpenReview.net, 2019.
[40] Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng. Chen Xing. Huishuai Zhang, Yanyan I Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architec- ture. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10524-10533, PMLR, 2020.
[41] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m'T5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, On- line, June 2021. Association for Computational Linguistics.
[42] An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang. Yong Li, et al. M6-t: Exploring sparse expert models and beyond. arXiv preprint arXiv:2105. 15082, 2021.
[43] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[44] Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, al. Pangu-o: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint 1141::2004.13366 2021.
[45] Minjia Zhang and Yuxiong He. Accelerating training of transformer-basedd language models with progressive layer dropping. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Taenn-Tienna Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[46] Zhengyan Zhang, Yuxian Gu, Xu Han, Shenggi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi. Jian Guan, Pei Ke, et al. Cpm-2: Large-scal cost-effective pre-trained language models, arXiv preprint arXiv:2106.1011
2021.
[47] Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. Cpm: A large-scale generative chinese pre-trained language model. Al Open, 2:93-99. 99, 2021.
[48] Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. M6-ufe: Unifying multi-modal controls for conditional image synthesis. arXiv preprint 1.0.v::2005.122.11 2021.
[49] Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 19-27. IEEE Computer Society, 2015.
13


Reviews:
[{"review_id": "TzmfrPl9P6", "reviewer": "Reviewer_BFav", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The proposal training strategy is novel and interesting, and has the potential to become the standard practice for training transformers.  I personally will certainly try it for my next large language model training.\n\n2. The proposed strategy is simple and straightforward to implement.\n\n3. Experiments are conducted on real systems and on real datasets. \n\nWeakness or areas of improvements.\nIn general, this paper will be greatly improved with a better experiment design. \n\n1. It's unclear whether the M6-1B model with longer training time in Table 2 was started from P2R strategy or from the Real Run. Actually it's better to list the results from both to study how P2R affects the long term convergence of the model. I'm interested in see what happens to Figure 2 after 7k mins of training time until convergence. Will M6-1B (P2R) perform worse than M6-1B (Real) at convergence?\n\n2. It' unclear to me what speed-accuracy was achieved by the proposed P2R strategy from the experiments section.  There are a lot of unanswered questions such as: what's the budget set in table 2?  How are the downstream task performance (few-shot) changed with the compute budget?  How much quality loss is introduced by the 5x saving in the compute cost during the pseudo phase? I would suggest the authors conduct more thorough studies to help readers understand the pros and cons of P2R.  These additional experiments can be done with little additional training cost as P2R also applied to smaller models as long as the stacking layers are identical in structure.  For example,  I would expect that  a larger model (like Transformer-Big) trained with P2R will have less training time but worse performance than training from scratch (like Transformer-Big Real) , but still outperforms the base model (like Transformer-Base) in model quality with compatible training cost. Something like the table below:\n\nModel &  Training time &.  pplx & one-shot downstream performance.\n\nTransformer-Base (Real) &. 3  & 2.5 &  40\\\\\n\nTransformer-Big (P2R) &. 3 &  2.2 & 45 \\\\\n\nTransformer-Big (Real) &. 10  & 2.1 & 50 \\\\\n\n3. In Table 2, M6-1B's Gigaword performance is worse than M6 with smaller scale, which is unexpected. I disagreed with the author's conclusion that the worse performance is not related to the training strategy.\n\n4. I think the significance of the proposed strategy to large language models is actually limited.  First, it's not going to save the serving cost. Second, the amount of  time saving from the P2R approach might be small compared to the overall convergence time, especially for the 10T scale.  It will be more interesting to see how this approach works with regular size transformer in various applications.\n\n", "summary_of_the_paper": "The paper proposed an interesting strategy to reduce the training time for large scale language models consisting of stacking layers with identical structures. Users first trained the models with shared parameters across the layers, then relax the tie constraints so that parameters at different layers are updated differently.  The paper showed some empirical evidence that the proposed strategy converged faster given a limited training time budget and demonstrated the feasibility of training a 10T model.", "summary_of_the_review": "The paper proposed an interesting idea to reduce training time for models with identically repeated layers. The empirical evidence provided in the experiments are not convincing enough to make a standard practice.  However, it's still worth a try for practitioners in the model pretraining community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BM-CWmNh_Ox", "reviewer": "Reviewer_YjPt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach makes sense. However, I have the following major concerns:\n\n1. The 10-trillion parameter model (M6-10T P2R) might be under-trained. It is hard to tell if training a relatively smaller model (e.g, 100 billion parameter) on the same number of GPU hours can give better numbers. I believe a larger model can outperform smaller models only if it's trained enough. The authors advocate for green AI, and train a 10-trillion parameter model. Thus, the main point should be the 10-trillion parameter model uses fewer GPU hours to get the same quality. I do not see such evidences.\n    * In the \"Convergence Analysis\" paragraph and Figure 4, the authors showed that M6-T converges faster than M6-T in terms of the same number of seen examples. But I think the more important question is, does M6-T achieve better numbers than M6-T given the same GPU hours yet.\n    * In Table 2, how long does \"training for long\" mean? Additionally, if there is only \"limited budget\", the important question is should we train M6-1B (P2R), or a smaller model for longer.\n\n2. The pretraining dataset is very small compared to the scale of the network. Practically speaking, training a multi-billion parameter model requires more data than BookCorpus and English Wikipedia, let alone a multi-trillion parameter model. The experimental results would be more convincing if the authors used larger datasets.\n\n3. As the authors mentioned, we do not have results of finetuning the 10-trillion parameter model on downstream tasks. We don't know how impactful the proposed method can be.\n\n4. The sharing-delinking paradigm is very similar to the sharing and unsharing method proposed in a previous ICLR submission: https://openreview.net/forum?id=jz7tDvX6XYR . The authors should discuss this work, even if it's a resubmission by the same authors, since the claims and experiments are different.", "summary_of_the_paper": "The authors propose to train very large neural language models via a \"Sharing-Delinking\" paradigm. The proposed method first trains a model with weights shared across layers. In this way, the model appears to be smaller and it can fit into fewer GPUs. At some point, the authors delink the weights, and continue training the model in the conventional way.\n\nThe authors also propose a granular CPU offloading mechanism to save CPU memory.", "summary_of_the_review": "It is very impressive to be able to train a 10-trillion parameter model. However, the benefit of training such a large model is unknown, since critical evaluations are missing. Overall, it appears to me that the experiments on very large models are not ready, given the lack of a reasonably sized dataset or downstream evaluations. Thus, it is not clear how impactful the proposed method can be.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "AKVMp0-eZ6j", "reviewer": "Reviewer_gmHp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Overall, the paper is well-written and the contributions of authors are clearly explained.\n* To my knowledge, this is the first work that leverages both expert sparsity and parameter offloading to train a model of unprecedented scale, which is an important engineering achievement if it was indeed executed succesfully.\n\nWeaknesses:\n* The overall claim of pretaining a 10-trillion parameter model in 10 days is a bit misleading: first, the stopping criterion for pretraining is unclear, that is, it's not entirely clear whether the model has converged after just 15k steps and other models of this scale took significantly more steps for pretraining with even larger batches. Second, there is no downstream evaluation for M6-10T or comparison with other models of comparable scale: it might have been possible to reimplement some of the prior work as baselines.\n* Although pretraining with parameter sharing may indeed be more memory-efficient, the (seemingly) inherent disadvantage of P2R is that in the second stage of training, that efficiency is lost due to the unlinking. As a result, the users of this methodology will still be forced to train the model of full size at some point, and the gains of P2R are not fully quantified. For instance, why not just pretrain the model with parameter sharing for the entire duration of the experiment, as in the original ALBERT paper? Moreover, even if the wall-time convergence of P2R is much faster, how does Pseudo pretraining compare with Real in terms of iteration convergence?\n* The pretraining objectives, such as text denoising and image-based text denoising, are not described in detail. Authors should provide the code of the objectives, describe it in the paper or at least give the exact reference to prior work, because there exist several different objectives which fall into this category. For instance, it is unclear how the resulting model is applied to the language modeling and summarization tasks, because in some definitions of denoising pretraining, the model actually \"sees\" the entire input sequence and thus does not learn to generate it in an autoregressive manner.\n* In Section 3.2.3, authors note that in order to determine the moment of pseudo-to-real switching, they need to repeatedly attempt to swiitch at different training steps and measure the difference in loss between continuing to train with shared parameters and training with unlinked ones. This strategy seems to highly increase the computation load of the entire procedure, unless in practice the intervals between attempts and the evaluation period are very conservative: the exact values for the main experiment as well as the number of \"unsuccesful\" attempts are not given in the paper, and I believe that this issue needs to be discussed in more detail.\n* Looking at the results in Table 1, it was quite surprising to observe a difference in throughputs of Pseudo and Real which is this significant. Although the parameters are shared across layers, the computational requirements of both forward and backward passes should be approximately the same. While there might be some additional inefficiencies connected with the use of pipeline parallelism, the work does not discuss or analyze this difference in performance; in the original ALBERT paper, authors have found training with shared parameters to be only 1.7 times faster than without sharing. I believe that the comparison should be without offloading, as it is an orthogonal factor that needs to be evaluated separately.\n* The granular offloading mechanism also needs to be compared both with ZeRO-Offload [1] and L2L [2] in terms of its efficiency. As of now, it is not quite clear whether the bottleneck outlined by authors is indeed a problem: in [1], the authors attempt to address the communication bottleneck by overlapping it with computation and asynchronous optimization.\n* Lastly, using idle GPU memory for storing extra layers means less training examples processed in a microbatch for pipelining or gradient accumulation, which may also reduce the efficiency of distributed training. Currently, there is no discussion of this tradeoff, and this question might be important in practice, since currently the most popular approach is to train models of this scale on extremely large batches.\n\nQuestions and typos:\n* In Section 3.2.1, you say that \"the amount of gradient [when sharing parameters] becomes $2/L$ of the original one\"; if there are L times fewer parameters, shouldn't that be just $1/L$, the same factor which you arrived at for weights and optimizer states?\n* For the M6-10T model, what datasets and pretraining objectives did you use? Is it the same data and tasks as in Section 3.3.1?\n* Small nitpick: the methods offered as a part of the DeepSpeed framework are grouped under the name of ZeRO, not ZERO.\n\n[1] ZeRO-Offload: Democratizing Billion-Scale Model Training. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He\n[2] Training Large Neural Networks with Constant Memory using a New Execution Algorithm. Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, Sujeeth Bharadwaj", "summary_of_the_paper": "This work studies the problem of efficient pretraining of large-scale models for language and vision representations, namely the issue of significant memory requirements for models with billions to trillions of parameters. Authors propose two modifications: first, to reduce the memory load and improve convergence at the initial stage of training, they suggest to train a multilayer model with shared parameters and then unshare them. Second, to maximize GPU utilization with offloading, authors develop a method for granular CPU offloading, which keeps larger chunks of the model in GPU memory. When combined, the proposed methods allow the authors to train a 10 trillion parameter model on 512 GPUs.", "summary_of_the_review": "This work proposes two methods for improving the efficiency of large-scale deep learning. While the methods are conceptually very simple and thus compelling, the paper might benefit from a more thorough validation of claims about these methods, both in terms of compaing with prior work and more detailed empirical evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8owfgJA9naR", "reviewer": "Reviewer_3GYN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n1. The paper focuses on training extremely large models with limited computational resources, which is of great empirical importance.\n\n2. The proposed P2R method is easy to implement and can be adapted to different models (as long as there is a decent number of repeated layers, whose weights can be shared).\n\n3. The efficacy of P2R is empirically verified.\n\n**Cons:**\n1. *Insufficient related work discussion on fast training methods*. There is a large volume of works on fast training of deep learning models, and the notion of \"weight sharing\" was also explored previously. For instance, \"Trellis networks for sequence modeling\", \"Speeding up Deep Model Training by Sharing Weights and Then Unsharing\" show that sharing weights leads to faster convergence. Other works that train a smaller model first and then grow the model into a large one can also sometimes be recast as sharing and delinking (for example, \"Net2Net: Accelerating Learning via Knowledge Transfer\"; \"Multi-level Residual Networks from Dynamical Systems View\"; \"Efficient Training of BERT by Progressively Stacking\").\n\n2. *Need a more comprehensive empirical evaluation*. Given that the main contribution of this paper is to propose a novel empirical training method, it needs more thorough evaluations. In particular, I think it would be good to have experiments covering: a) various models. Only M6 models are trained in this paper. The scale of M6-10T is promising, but it would be good to include other models which also have repeated layers. b) various downstream tasks. Evaluations only on WikiText-103 and Gigaword are somewhat limited. c) various domains. There are models commonly used in other domains (e.g., image classification) that have repeated layers. Does the proposed method work for them?\n\n3. *Clarifications needed for Section 4.3*. In the training efficiency, the M6-10T with P2R is trained for around 10 days and another M6-10T is trained without P2R for around 3 days for comparison. Why is P2R training time longer than without P2R?\n\nFigure 4(a) plots the Log Perplexity v.s. clock time and the conclusion is \"P2R can outperform the one trained from scratch by a large margin\". I feel this claim is not fully supported as the M6-10T does not converge yet. Give both more time, will they converge to a similar performance?\n\nFigure 4(b) compares M6-10T P2R v.s. M6-T on sample-basis. Why the baseline is changed here and how does M6-10T P2R compare to M6-10 Real on a sample basis? Further, both do not seem to converge given 10M samples. How will the curve look like when giving more training samples (e.g., 20M)? \n\nIt also confuses me a little as the M6-10T P2R curves in 4(a) and 4(b) do not look the same - I'm confused because I think they should only differ by some scaling in the x-axis (changing from training time to samples trained)?\n\n**Questions:**\n\n1. For Table 1, the speed difference between Pseudo and Real is huge. What leads to the acceleration? Is that primarily coming from the GPUs' communication overhead? It seems that both Pseudo and Real are trained on 48 GPU devices according to the description, then they should have similar communication overhead?\n\n2. For Figure 4, when does the \"delink\" happen? How frequently does P2R check for switching, and is that time included in Figure 4(a)?\n\n3. After delink, the \"Real Giant\" will require the same amount of computational resources as training the model in the standard way? Is the computational saving then only appears in the \"Pseudo Giant\" stage? If so, what is the typical training time spent in the \"Pseudo\" and \"Real\" stages, respectively?\n\n**Minor Comments:**\n\n1. In the second paragraph of Section 3.2.1, it says \"the amount of gradients becomes 2/L of the original one\". Why not 1/L?\n\n2. In the description of Table 1, $l$ should be $L$.\n\n3. It's probably better to use a larger fontsize for the numbers and legends in Figure 2 and 4.", "summary_of_the_paper": "This paper illustrates the training difficulty of extreme-scale models (with a special focus on memory requirement) and provides a simple but effective solution called \"Pseudo-to-Real\", where in the \"pseudo\" stage the weights of the repeated layers are shared, and in the \"real\" stage the weights are delinked. Experiments are conducted to demonstrate the efficacy of the proposed method.", "summary_of_the_review": "This paper proposes an empirical method for fast training of models with repeated layers. The M6-10T experiments warrant scalability. However, I believe a more comprehensive set of experiments is needed to demonstrate the efficacy of the proposed training paradigm. Please see detailed comments in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Xwo6-HxA__", "reviewer": "Reviewer_pBTW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an important problem of improving the efficiency (time and resource) of training massive models, as these models rapidly scale in size to achieve greater performance. P2R seems to be a promising approach that aligns with the intuition that the structure of the model (and data) for effective learning could be dynamic throughout the training process. I believe this intuition is the basis of successful schemes such as pre-training+finetuning, lottery tickets hypothesis, etc.  \n\nTwo major weakness of the paper are:\n1. Does not demonstrate P2R for M6-10T, the 10T parameter model, since no downstream task results are presented for this model in the evaluation section. Similarly, the claim of 10 days pre-training remains unsupported by the draft. \n2. The parameter sharing approach seems arbitrary and un-insightful. Why have all the layers share 1 set of parameters, which not 2 or 4, or something fewer than number of layers? ", "summary_of_the_paper": "The paper proposes a technique called Pseudo-to-Real (P2R) for reducing the computational and time requirements of training massive (or Giant) models with trillions of parameters. The key idea of P2R is a two phase training approach for Giant models. The first phase involves training a smaller version of the model (a.k.a., Pseudo-Giant) which is obtained by making all layers share parameters. The second phase involves training the Giant model after initializing with Pseudo-Giant weights. The paper further proposes Granular CPU offloading which is to offload some but not all model parameters to CPU memory to reduce GPU memory consumption. Finally, the paper provides some evaluation results to demonstrate P2G.", "summary_of_the_review": "Although the problem is important and the proposed P2R approach is promising, the submission does not demonstrate the effectiveness of the proposal on the problem. Specifically, the paper lacks downstream task results of a 10T model trained with P2R. It is difficult to fairly judge this paper without these qualitative results, especially given the claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a weight sharing method to speed up the pretraining of large language models. Basically, during the training, it first share weights across all the layers with the same architecture, and then untie the shared weights at some point later. The main advantage of weight sharing is that it can reduce the memory load. Our reviewers have many concerns on this work. The method is not well motivated or explained, and many experimental details are missing. In particular, there is no downstream task result presented for the so-called 10T parameter model. The claim highlighted in the title remains unsupported. In addition, one of our reviewers pointed out that the proposed method is fairly similar to the method in a previous ICLR submission: https://openreview.net/forum?id=jz7tDvX6XYR.
Decision: Reject