The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
Authors: Conglong Li
Minjia Zhang
Yuxiong He
Venue: NeurIPS 2022 Accept (2022)
Keywords: natural language processing
language model pre-training
autoregressive language model
training instability

Abstract: Recent works have demonstrated great success in pre-training large-scale autoregressive language models (e.g., GPT-3) on massive GPUs. To reduce the wall-clock training time, a common practice is to increase the batch size and learning rate. However, such practice is often brittle and leads to a so-called stability-efficiency dilemma: increasing the batch sizes and learning rates leads to better training efficiency but can also result in training instability, leading to poor generalization accuracy or failed runs. To better understand this phenomenon, we conduct an in-depth analysis on large-scale pre-training experiments replicating the GPT-2 model with public dataset. We find that there is a strong correlation between training instability and extreme values of gradient variance. We further identify that samples with long sequence lengths contribute to these extreme gradient variance values, especially at the beginning of the training, indicating that long sequence length can be a main source of training instability.

Based on the analysis, we present a simple yet effective Sequence Length Warmup method that aims to solve the training stability-efficiency dilemma by avoiding extreme gradient variance values. Moreover, we present a lightweight tuning strategy that allows us to tune our method with just a small portion of the expensive full training. Experiments replicating GPT-2 models (117M and 1.5B) show that our approach enables stable training with 8x larger batch size and 4x larger learning rate, whereas the baseline approach struggles with training instability. To achieve the same or better zero-shot evaluation results, our method reduces the required number of training tokens and wall clock time by up to 2.2x and 3.7x, respectively. Experiments replicating GPT-3 model (125M) show that our approach enables stable training with 8x larger batch size and 40x larger learning rate, and retains 99\% of the zero-shot accuracy on 11 tasks using 10x less data and 17x less time compared to the original GPT-3 training recipe, while the baseline diverges under the same settings and only retain 95\% of accuracy under lower learning rate.

The Stability-fffcieneyy Dilemma: Investigating Sequence Length Warmup for Training GPT Models
Conglong Li Microsoft conglong, 11 www.micoosoott..mmmmmm
Minjia Zhang
Yuxiong He Microsoft yuxhe@microsoffe com
Microsoft
minjiazemiicrosoft.ccm com
Abstract
Recent works have demonstrated great success in pre-training large-scale autore- gressive language models (e.g., GPT-3) massive GPUs. To reduce the wall-clock training time, common practice is to increase the batch size and learning rate. However, such practice is often brittle and leads to so-called stability -effeiit--ffciiency dilemma: increasing the batch sizes and learning rates leads to better training efficiency but also result in training instability, leading to poor generalization accuracy or failed runs. To better understand this phenomenon, we conduct an in-depth analysis on large-scale pre-training experiments replicating the GPT-2 model with public dataset. We find that there is a strong correlation between training instability and extreme values of gradient variance. We further identify that samples with long sequence lengths contribute to these extreme gradient variance values, especially at the beginning of the training, indicating that long sequence length can be a main source of training instability.
Based on the analysis, we present simple yet effective Sequence Length Warmup method that aims to solve the training stability-effciinnge dilemma by avoiding extreme gradient variance values. Moreover, we present lightweight tuning strategy that allows us to tune our method with just small portion of the expensive full training. Experiments replicating GPT-2 models (1 17M and 1.5B) show that our approach enables stable training with 8x larger batch size and 4x larger learning rate, whereas the baseline approach struggles with training instability. To achieve the same or better zero-shot evaluation results, our method reduces the required number of training tokens and wall clock time by up 2.2x and 3.7x, respectively. Experiments replicating GPT-3 model (125M) show that our approach enables stable training with 8x larger batch size and 40x larger learning rate, and retains 99% of the zero-shot accuracy on 11 tasks using 10x less data and 17x less time compared to the original GPT-3 training recipe, while the baseline diverges under the same settings and only retain 95% of accuracy under lower learning rate.
Introduction
Large-scale Transformer-based language models have achieved great success in many natural lan- guage processing tasks [46] [T] Among them, large-scale autoregressive models, such as GPT-3 (6). have attracted lots of attention due to their superior performance zero-shot generalization, i.c., they can perform wide range of tasks that they are not explicitly trained on. However, pre-training GPT models raises huge challenges on training efficiency and less-discussed training instability issues. On the efficiency side, as the model size continues to grow from few hundreds of millions (e.g.. GPT (3I). to billion-scale parameters (1.5B GPT-2 (B3)). and to more recent hundreds of billions of parameters (175B GPT-3 (6), the training cost also increases exponentially: requires 9.2 days
This paper was previously titled "Curriculum Learning: A Regularization Method for Efficient and Stable Billion-Seaa GPT Model Pre- Tr--Trainnn" in an early arxiv preprint version 22]
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

on 512 V100 GPUs to train 8.3B GPT-2 40), and 47.8 days on 2240 A100 GPUs to train a 530B GPT-3-style model 26). Such long training time makes it almost infeasible for most research labs and practitioners to reproduce these models. Various solutions have been proposed to reduce the training wall clock time of these large models 40 25 35 However, many solutions require using more GPUs or sophisticated system techniques.
In this work, we investigate speeding up the pre-training of GPT-style models via exploiting data efficiency, not at the cost of excessive hardware resources. In particular, in distributed training environment, increasing the batch sizes and/or using more aggressive learning rates can make the model converge faster [41]. However, it has been observed that larger batch sizes and learning rates can make large-scale GPT model training more difficult, e.g., causing training instability that leads To investigate this training instability issue, we conduct
to divergence or slow convergence 49 9
a thorough study of the GPT-2 pre-training task [33 40) with different models sizes under various batch sizes, learning rates, and sequence lengths. We find a stability-efficiency dilemma:
larger batch size (and larger learning rate) increases the per-iteration computational efficiency but with increasing likelihood of training instability and even divergence. smaller batch size makes the training more stable but decreases the per-step computation efficiency significantly.
We find it difficult to overcome this dilemma by existing techniques such as extra gradient clipping. More recent proposed techniques such as batch size warmup proposed in [6] does not provide stability benefit in our evaluations. Recently, Shortformer(30) shows that by adding an additional first training stage with a shorter sequence length, language models can achieve the same dev. set perplexity with shorter total training time. However, (1) its main focus is to purely improve training efficiency instead of solving the ffficiecc--stablli dilemma, and our evaluations show that Shortformer's ' 2-stage approach is insufficient for overcoming the training instability issue for large models (Section 5.1), (2) it is primarily evaluated on small scale transformer models (247M) on WikiText datasets (103M tokens) without considering large-scale generative models like GPT with hundreds or even billions of parameters, and (3) it does not discuss how to choose some of the important hyperparameters, which is very expensive to figure out for large-scale model training.
Inspired by Shortformer, we investigate the importance of sequence length in training GPT models and find that plays an important role in both training stability and efficiency. Based on our investigation, we propose a simple yet effective method called Sequence Length Warmup (SLW), which starts training with short sequences and gradually increases the length. We observe that our approach enables stable and efficient training with much larger batch sizes and learning rates than baseline approaches. Specifically, we make the following contributions: (I) We conduct an extensive study of the GPT-2 pre-training task, which provides detailed insights about the training stability-efficiene dilemma, the correlation between instability and gradient variance outliers, and how sequence length plays a critical role (Section B). (2) Based on the study, we present simple yet effective sequence length warmup method for GPT-style model (and autoregressive model in general) that enables stable training with improved training efficiency. We also identify a lightweight hyperparameter tuning strategy for the approach, which identifies promising hyperparameters by only incurring a small fraction of the expensive total pre-training cost (Section 4). The implementation of our approach as well as the necessary changes to the GPT-2/3 pre-training framework has been open sourced in a deep learning optimization library called DeepSpeed" (3) We conduct large-scale experiments to demonstrate the proposed work's s ability to provide superior training stability and efficiency at the same time (Section 5). Our empirical results show that:
SLW enables stable and efficient training with 8x larger batch size and 4x larger learning rate on GPT-2 (117M and 1.5B) models with public datasets, while the baseline and related works struggle with instability under the same settings. To achieve the same or better zero-shot WikiText- 103/LAMBADA evaluation results at the end of training, SLW reduces the required number of training tokens and wall clock time by up to 2.2x and 3.7x, respectively.
On GPT-3 model (125M) pre-training we study an even more aggressive training scenario where only 10% of data can be used. Our method, with 8x larger batch size and 40x larger learning rate than the original GPT-3 training recipe, is able to maintain the training stability, retaining 99% of the zero-shot accuracy on 11 evaluation tasks, and use 10x less data and 17x less time. Without our
htnps//gghhbbboommmmiissstttttep http://ww.......ooo/oooooeeeeeee http:///wwwwwweeeppped..lll
2

---

method, the baseline has unrecoverable divergence under the same settings, and can only retain 95% of the zero-shot accuracy after lowering learning rate to 30x.
2 Background and Related Work
Language Model Pre-training: The accuracy of transformer-base language models grows substan- tially with its model size [3 [3 []. Today, large language model such as GPT-3 [6] contains up to 175B parameters, and recent studies show that model accuracy can continue to improve with even larger model sizes [16.. However, training these large models often incurs excessively long training time and training difficulties ( Therefore, there are lot of demands of performing efficient and stable training for these models. To have the pre-training finished in a reasonable amount of time, the most common way is to leverage data parallelism to train models on multiple GPUs. However, the speedup gains often saturate beyond a few tens of GPUs, because communication becomes the major bottleneck, i.e., the workers will spend more time communicating gradients than computing them, as the number of GPUs increases. To mitigate this bottleneck, recent works such as 1-bit Adam + have studied gradient compression and demonstrate their effectiveness against auto-encoding models such as BERT [TT] An alternative approach to alleviate these overheads is to use large batch sizes. For example, LAMB [52] and 1-bit LAMB (21 enable stable and efficient distributed BERT pre-training with batch size up to 64K/32K (for sequence length 128/512, i.e., 8M/16M 6M tokens per batch) while maintaining the cample-wiss convergence speed. For encoder-decoder models, T5 size up to 2K (for sequence length 512, i.e., 1M tokens per batch). For autoregressive models such as the GPT family B1 33 6 existing works use batch size up 1.6K (for sequence length 2K, i.e, 3.2M tokens per batch). Despite the benefit of reduced communication overhead, large-batch training is sensitive hyperparameteers and often leads to issues such as slow convergence, training instabilities, and model divergence. For example, recently a research project shared that they are dealing with challenging training instability issues when pre-training a 104B GPT-style model with batch size 2K [49], and another work on a 540B model with batch size 2K observed spikes in the loss roughly 20 times during training, despite the fact that gradient clipping was enabled 9
[4 uses batch
Our method can be viewed as a kind of curriculum learning (CL) which presents easier/simpler examples earlier during training and gradually increases the sample difficultie:] Comparing with traditional CL works which focus on solely improving the convergence speed under the same batch size, learning rate and other hyperparameters, our work is motivated by the stability-efficinnyy dilemma and we aim to achieve both efficient convergence and better stability by enabling stable training with more aggressive hyperparameters. To our knowledge, we are the first to investigate and confirm that certain curriculum learning method can provide a dual stability-efficinncy benefit.
Curriculum Learning:
39 2]
In the NLP area, most of the curriculum learning works focus on small-scale one-stage tasks and downstream fine-tuning tasks, such neural machine translation (NMT) 11
and There are also a few works explore curriculum These works show that curriculum learning can improve convergence speed, reduce training time, and improve accuracy under the same training hyperparameters baseline. In these works, the curriculum difficulty metrics for each training sample are usually defined as the sentence length, vocabulary frequency, the inference loss on smaller/cheaper models, or based on self celf-paced learning [19]. For the pacing function (i.e., to decide the curriculum difficulty range when sampling next training data batch), these works usually use fixed predefined functions (c.g.. gradually increase difficulty upper bound by linear, root, and exponential functions), bucketing heuristics (group data with similar difficulties, and sample from a subset of buckets every time), or based on self-paced learning.
56
29
57
natural language understanding (NLU) 36
37
44
50
learning for language model pre- pre-training 30
55
GPT-2 Pre-training Stability
Analysis
In this section we perform an in-depth analysis of pre-training tasks (without our method) replicating the GPT-2 models with public data. We follow the training pipeline from the NVIDIA Megatron-LM work 40 3 All of the experiments are performed on 128 NVIDIA V100 GPUs (32GB memory). There are 16 nodes and GPUs per node. GPUs inside the same node are connected by NVLink 2.0. and nodes are connected by 100 Gigabit InfiniBand EDR noder-node network. We evaluate
"The shorter sequences are not necessarily easier but can be viewed as simpler examples since there are less context to embed.
https://githuu.comm// www../////..D/MMMM/aM.o.....
3

---

Train lo55
9. V
20000
17500
Buz 512,
seqien
E
17500
20000
s42Mn 1X
15000
Buz soginn
E 15000
sealen 3K
Training
in
2sc 4K.
12300
segien
12500
Bez 4K
7500
7500
5400
5000
2500
2500
500
100B
1500
50B
1006
1500
506
1006
1506
0
50B
1008
1508
Tokens
Tokens
Tokens
T Tokens
(a) 117M training loss
(b) 1.5B training loss
(c) 1 117M variance norm
(d) 1.5B variance norm
0.06
0.05
0.07
Biz 1K, segien
Bie 322 acaler
BR 532 seslet
0.005
0.06
0.07
sogi01
tax
tsz 512,
0.6
0.8
1.0
3.58 Biz
Varfanic norm
loso rail
Normaliced
1.00
0.8
0.6
1.58 55.822 Loss o VVarin anriannee max
0.4
0.4
0.01
0.2
0.2
0.00
100B Takens
1508
0.00
1008
1508
0.0
500
1000
1508
0.0
50B
100B Tokens
1500
Take ns
Tokons
(e) 117M variance max
(f) 1.5B variance max
(g) Var norm correlation
(h) Var max correlation
Figure I: Training loss, Adam variance norm/max element, and correlations between loss spikes and variance norm/max during GPT-2 pre-training (without the proposed method) under different model sizes, batch sizes (and LR), and sequence lengths. In Appendix A. we plot the same figure but zoom
in the first 30B tokens.
two GPT-2 model sizes from the original GPT-2 work (33]: 117M parameters (12 layers, 768 hidden size, 12 attention heads) and 1.5B parameters (48 lay- ers, 1600 hidden size, 25 attention heads). For training data, we collect and use the same dataset blend as the Megatron-LM work: Wikipedia [11), CC-Stories [45), RealNews [54] and Webtext (32].
Measuring training instability by the ratio between the current step training loss and the minimum loss among all previ- ous steps. Larger ratios (esp. those greatly larger than 1. 1.0) indicate larger training insta- bility/divergence. Proposed work (SLW) and related works (last two rows) are discussed
Table 1:
in Section 5
We evaluate two sets of training parameters. The first set follows the Megatron-L.M work: batch size 512, 300K total training steps (157B tokens), and learning rate 1.5 x 10-4 with a linear warmup of 3K steps and a single cycle cosine decay over the remaining 297K steps (1 x 10-5 min. learning rate). The second parameter set tests more aggressive training strategy: batch size 4K (8x larger). 37.5K total training steps (157B tokens and learning rate 6 6 10 (4x larger) with a linear warmup of 3K steps and a single cycle cosine decay over the remaining 34.5K steps (same min. learning rate). For sequence length/context size, mainly 1K which is the default for GPT-2. But we also test 2K (on the 1 117M model with batch size 512 and 157B tokens) which is the default for GPT-3. All experiments are performed with mixed
Aistep with max loss ratio > 1.2 loss (% of total stops) ratio
Pre-trnining parametees
Case
117M:
I: Baseline
Soe5l1-sellennK K
0 (0.0%) 1.05
60K
(0.0%)
3: Baseline
bsz44--sqqlen IK
22 (0.06%)
1.42
SLW 20K
Kszzl--sqlen1K
0 (0.0%)
1.02
Baseline
bsz512 bsz512-seylen2K
0 (0.0%)
1.04
6: SLW H10K
bs2512 1---eeleee2K
0 (0.0%)
1.04
I,5B:
7: Bascline
bsz512-segles 1-seollle
114 (0.04%)
2.10
8: SLW 270K
bs2512 102512-sl
0 (0.0%)
1.06
Baseline
bsz4K seqlen1K
1381 (3.68%)
5.65
10: SIW 45K
hez4K-seqleen
0 (0.0%)
1.02
II: Shortformer
bez4K-seqlen
219 (0.4%)
2.86
12: Baz Warmup
bsz4K-seqlen1K
1179 (2.01%)
4.32
mreisioon/PP16 training, Adam optimizer (3, = 0.9, B2 = 0.999, e e x 10-5) [17). 0.01 weight decay, same random seed, and gradient clipping at 1.0. For both batch sizes we use the same number of gpus (128). It is true that under fewer nodes, smaller batch sizes can also achieve good computation efficiency. However, in practice, the goal of a training task is usually "given a fixed number of hardwares, how to train the model in the fastest wall clock time". And given the increasing model sizes, pre-training on hundreds of GPUs is not uncommon. Thus we believe that using the same hardware resources is a fair comparison.
The -fffiliiyyefficieneyy dilemma: Figure I(a) and ((b) present the training loss curves of 5 baseline cases under different model sizes, batch sizes (and LR), and sequence lengths. At 1 117M, the baseline has few training loss spikes batch size 4K. At 1.5B, the baseline has many loss spikes when training with either batch size 512 or 4K. As an indicative measurement to quantitatively study training instability, we define "loss ratio" which measures the ratio between the current step training loss and the minimum loss among all previous steps. A ratio larger than 1.0 means that current step's ' loss is larger than the previous minimum loss, thus larger ratio indicates a larger loss spike and training instability. Table I summarizes the number of steps with loss ratio larger than 1.2, and the
For pre pre-training it is common to keep the number of training tokens the same for fair comparison.

---

Table Zero-shot evaluation of the trained models on the WikiText- and LAMBADA datasets. following the evaluation methodology from [4]]. Case 2 to 9 are compared with case 1, and case are compared with case 10. Proposed work (SLW) and related works (16, 17) are discussed in Section
Pre-training
Training
Training
Traiiing
Wekilext
LAMBADA
Case
parameters
sleps
lokens
lime
PPL
accuracy
117M:
1: Baseline
bsz512 1sz512-seglen seglen
300K
157B
37Hrr
27.78
33.19%
2: SLW 60K
bsz5 12-sollenn
200K
(1.8x)
20Hr (1.9x)
27.74
34.78%
3: SLW 60K
bsz5 1-
330K
157B (1x)
33Hr (L.x
27.01
34,41 %
4: Baseline
bsx4K-seqlen K-Seqlen1K
37.5K
(1x)
16Hr (2.3%)
28.09
32.54%
5: SLW 30K
bsz4 Ksz4K-scll
37K
92B (1.7%)
10Hr (3.7%)
27.77
33.40% %
6: SUW 30K
hez4 segleen
52.5K
157B
16Hr (2.3x)
27.15
34.16%
7: Baseline
wegglen
150K
(17)
3211r (1.2K)
28.19
32.99%
8: SLW 110K
bes512-seglen2K
122.5K
71B (2.2%)
15Hr (2.5%)
27,06
33.24%
9: SLW 1OK
bsz5 b-5egl-se
205K
157B (Ix)
31Hr (1.2x)
26.03
34.58 %
Baseline
hsz512 segleel
300K
157B
341lfr
I1: SLW 270K
1e2512-seqlen18
360K
122B (1.3x)
286Hr (1 (1.2x)
13,89
$7.38 %
12: SLW 270K
hs 2 seglenlKK
428K
157B (1x)
36411r (0.9k)
13.88
57.89
13: Bascline
0524K-icqlenll
37,3K
(1571((1))
115HT (2.5%)
14.76
53.06%
14: SLW 45K
bx44K seglen1K
SOK
121B (1.3x)
1211Ir (2.8%)
13.88
58.20 %
S1.W
K-sgglen
(1x)
158Hr (2.2x)
%
16: Shorffrmme
bsz4 -Segllenn
55K
157B (1x)
162Hr (2.1x)
14.14
57.23%
17: Bsz Warmup
58.8K
157B (1X)
165Hr (2.1x)
14.21
56.36%
Reference
Original GPT-2 117M diflerent
works:
19: Original GPT:2 1,5B [33] different data
17.48
63.24%
Megatron-I.M GPT-2 355M
[3 same
21: Megatron-LLM GPT-2 2.5B
[3), same data
12.76
61.73%
maximum loss ratio during the training. At 117M model size only the baseline with batch size 4K has high loss ratios up 1.421. At 1.5B model size the baseline with both batch size 512 and 4K has much more steps with large loss ratios, and with the maximum loss ratio as high as 5.65. Baseline with batch size 4K is less stable than baseline with batch size 512, indicating that larger batch sizes (combined with larger learning rates) could lead to more training instability risks. In Appendix [A..I] show that larger learning rates under the same batch size could also increase training instability.
Training instability are undesirable because (1) it could lead to divergence that never recover as in [49] and our GPT-3 experiments (Section 5.2 (2) in our GPT-2 case it leads to worse convergence, validation loss, and zero-shot downstream task accuracy. Table summarizes the zero-shot WikiText- 103/LAMBADA evaluation results. For both 117M and 1.5B models, increasing baseline's batch size (and LR) or sequence length leads to training instabilities and loss spikes, and it requires a nontrivial number of training steps/tokens to recover the training loss back to normal level (e.g.. Figure [(b). These training "detours" slow down the learning and eventually lead to worse evaluation results (c.g., case 13 vs case 10 in Table 2 On the other hand, increasing batch size (and LR) or sequence length improves training efficiency, reducing the training time by up to 2.3x under the same number of training tokens (case 1, 4, 10, 13).
Overall, the above observations demonstrate the the training is more stable and can achieve better final generalization, but presumably with poorer training efficiency under smaller batch size/learning rate/sequence length; increasing them leads to better training efficiency, but with lower stability and worse generalization.
dilemma for baseline pre-training:
The correlation between instability and gradient variance outliers: For stochastic gradient optimization, when the gradient variance is large, the algorithm might spend much time bouncing around, leading to slower convergence and potential divergence (88.. Previous studies show that variance reduction methods improve training stability in areas such as reinforcement learning [24,88
I()) plot the I norm and max element of Adam's variance state (v (g)) ) When baseline's batch size increases, the max variance norm decreases but the max element increases. Comparing GPT-2 117M and 1.5B cases, larger model size leads to larger variance norm and max element. When sequence length increases for the GPT-2 11MM case, the variance norm stays the same but the max element increases.
Figure I(c) I(d) [(e) and
where U: B201-
To further study the link between instability and gradient variance, Figure (gg
plot the loss ratio (defined earlier in this section) and gradient variance norm/max element (all normalized by max value) for the most unstable 1.5B baseline with 4K batch size. Results show that when training loss spike happens and loss ratio increases, the gradient variance norm/max also increase (especially the max outliers). Table presents the Pearson correlation coefficient calculations, which demonstrate
1(h
We use 4 norm avoid outlier amplification.
5

---

Training loss.
14
SedjanKK, beseine
:22
Sadan325
Table 3: Pearson correlation coefficient (with range (-1, 1)) between loss ratio and gradient variance norm/max. Low p-value indicates that the correlation is statistically significant.
10
Sedan12 10000000000000
2K
4K
7K
10K
Steps
Pearson
Figure 2: Step-wise training loss during GPT-2 1,5 pre-training (first 10K steps only) with batch size 4K, comparing seqlen 1K (baseline), seqlen 128, and mixed seqlen of 128+1K (IK seqlen used at the cyan areas).
comelation
cueflicient p-value
Loss Gradicnt variance norm Loss ratio va Gradient variancee max
0.23
0,26
0.0
statistically significant positive correlation between loss ratio and gradient variance norm/max. Overall, our analysis shows that training instability has a strong correlation with gradient variance norm and (especially) max element outliers.
Length of early data sequences is critical to training stability: Aiming to solve the stability- efficiency dilemma we first tried traditional methods such as increasing gradient clipping, but it does not fully resolve the instability issue (Appendix [A.32)). Seeing that in Figure | the training instability mostly happens at the first half of training, we then explored whether we can solve the issue by gradually increasing any of the batch size/learning rate/sequence length during training. We already employed the same learning rate warmup mechanism used by existing GPT-2 and GPT-3 works [33 40 6). We tried the batch size warmup method proposed in GPT-3 work [6], but the instability issue still appears when increasing the batch size (Section 5.1). Our investigation on the sequence length leads to interesting findings, where we find that sequence lengths play a critical role in training instability. Figure 2 presents the training loss curve of the most unstable GPT-2 1.5B pre- pre-training training with batch size 4K and seqlen IK, together with another two artificial settings: one with seqlen 128, the other with mixed seqlen where we feed 900 steps of seqlen 128 then 100 steps of seqlen IK in every 1K steps. The seqlen 128 case has no instability issue, even with large model size/batch size/learning rate. The mixed seqlen case has instability issues, and (1) they mostly happen when we switch to seqlen 1K (e.g., at step 900, 1900, 2900...); (2) they mostly happen during the first 5K steps, and after that it becomes more stable than the seqlen 1K case. These observations indicate that training instability is strongly correlated with early long sequence lengths, which motivates us to explore the sequence length warmup method described the next section, and evaluations in Section 5 will demonstrate how this method provides a gradient variance reduction effect and solves the stability-efficiencyy dilemma in our experiments.
The Sequence Length Warmup Method
The analysis in last section about training instability and sequence lengths motivates us to explore sequence length warmup methods: the model needs to start learning with short sequence length for more stable training, then gradually increase the length when training becomes more stable so that the model can still learn from longer contextual information to achieve better final model accuracy.
The sequence length warmup strategy depends on two factors: how to support variable sequence length during training and how to adaptively decide the sequence length for each iteration (the pacing function). For the first component, we develop an efficient cruncationbbaee implementation: For the baseline GPT-2 pre- training, the raw text inputs are indexed into sequences with the same length before training, so that the model can efficiently retrieve a batch of fixed-length sequences regardless of the actual sentence boundaries. It's possible to index the text inputs based on all possible sequence lengths, but that adds significant amount of overhead due to the massive pre-training data. To avoid the large indexing overhead, we take a lightweight approach: our implementation still lets the dataloader index the raw text into only the full sequence length. At each training step, our method uses pacing function to determine the sequence length and then truncates the full-length sequences to obtain modified version of the mini-batch for training. It is true that this truncation- based implementation will drop some data in the current step. However, with some implementation changes, it's possible to record the index of dropped data and use them in future steps.
We define the pacing function as a step-wise linear function with the following properties: Given a starting sequence length seglen,, an ending sequence length seglen, (full sequence length), and a total duration T (number of steps), the sequence length used for the training batch at step i is seqlen, = seqlen, + (seglen, - seglen.) x min( A 1). Besides step-wise linear, we also explored 3 other pacing functions: i) We tried discrete 2-stage pacing function from [30], but it leads

---

to unstable training and worse convergence (Section 5.1p ii) We tried a step-wise root function (seglen; = seglen, + (seqlen, - seqlen,) x min( +J, 1), where is the root degree), which performs similar to linear but requires one extra hyperparameter. iii) We tried an adaptive pacing function based on training/validationn losses, which also performs similar and requires extra tuning.
Pacing function analysis and tun- ing strategy: To study the impact of
2000
21.0
8 1750
our approach's 's pacing function, we set
1500
20.8
the starting sequence length (seqlen,) fixed at 8 and perform grid search for
6 1250
0
20.6
1000
750
20.4
62K
the pacing function duration (T) on the GPT-2 117M case full training (de- tails in Appendix A.4). After full train- ings we perform evaluation on pretrain data test set and WikiText/LAMMAAA zero-shot tasks to confirm which du- ration T provides the best accuracy performance. All the cases have quite comparable evaluation results, indicat- ing that the performance is not very sensitive to the duration T within a reasonable range. This grid search sheds light on a low-cost tuning strategy: we find that (for GPT-2 117M training with batch size 512 and 4K) the best duration T is the longest duration that does not have significant validation perplexity fluctuation during the first 10K steps (i.e., a few multiples of the LR warmup steps). In on our study, this "significant fluctuation" can be defined "whether the perplexity value becomes larger than 1.3x of the previous best perplexity". In Figure B(a the SLW 60K is the longest duration we tested that does not have significant validation fluctuation during the first 10K steps. In Figure 3(b and Appendix SLW 60K does provide the best final validation perplexity, best final test perplexity, and second best eval results. Since it does not require training the model until full convergence, this heuristic greatly reduces the hyperparameter tuning cost of our approach. Another grid search on the starting sequence length seglen, shows that it's generally better to set it as small as possible, to maximize the stability and convergence speedup benefit. However, if the validation perplexity has significant fluctuation near the starting sequence length, increasing seqlen, would lead to better convergence.
500
250
20.2
120K
0 2K
4K
6K Steps
8K
10K
20.0
0
508
100B
1508
Tokens
(a) Step-wise validation per-
(b) Boken-wise validation per- plexity (beginning of training) plexity (end of training)
Figure 3: Validation perplexity during GPT-2 117M seglen IK pre-training with batch size 512 and different duration T. ("SLW 20K" means proposed approach with T=20K steps).
Overall, the low-cost tuning strategy can be summarized as: (1) Start with seqlen, 8 and T = few multiples of LR warmup steps. (2) Increase seqlen, until the validation perplexity no longer has significant fluctuation the very beginning. (3) Perform a binary search find the largest T that does not have significant validation perplexity fluctuation during the first few multiples of LR warmup steps. This tuning strategy relies only on the validation set and does not require test set or downstream task evaluation. For the GPT-2 1.5B and GPT-3 125M models, we used this strategy to tune T and seglen, for the pacing function, and results show that this low-cost tuning strategy could provide similar subility-efficieng benefit as grid search on full training runs (GPT-2 117M case).
5 Evaluation
5.1 GPT-2 experiments
For GPT-2 model, dataset, and hardware, we follow the same methodology in Section 3 For proposed work's pacing function configurations (Section 4 we use seglen, = 8/64 (for 117M/1.5 model based on tuning) and seglen, = 1K/2K (full sequence length). To fully utilize the NVIDIA Tensor Core acceleration, we add seqlen, = seglen; sequence length is always multiple of 8. For the total duration T. we tune this parameter (grid search for 117M and low-cost tuning for 1.5B) for each case. For the training parameters, for our approach we use the same shared parameters as the baseline except two parameters: 1) Because during sequence length warmup the number of tokens in a data batch is smaller, we modify the training termination condition so that all cases stop when reaching the same 157B training tokens. 2) Because of 1), proposed approach now has more training steps, which make it necessary to modify the learning rate decay schedule to have a fair comparison with the baseline. We change the learning rate decay to coken-wise over the 157B tokens (still cosine decay) instead of step-wise over the total number of steps. We describe the underlying rationale in Appendix A.2 A.2
(seqlen, mod 8) postprocessing to make sure the
7

---

Validato 0 13.5
g. 14.5
15.0
15.0
20 12500
117500
15000
E0.05
0.07
Bar 532.
522
12:5
13.0
12.5
13.0
bacak
8 10000
7500
0
Bar 4k
4K
12.0
12.0
12.5
568
1008
1508
60 120 300
16C 24K
0.00
8K 14k 24K 32K
Tokens
Walk-llock time (hr)
Steps
Steps
(a) Token-wise valida-
(b) Time-wise validation (c) Step-wise Adam vari-
(d) Step-wise Adam vari- ance max element
tion perplexity
perplexity
ance norm
15.0
15.0
E 1:4000 2
0.07
14.5
14.5
06
14.0
14.0
tu ee
122000
8
8 12.5
13.0
11.5
12.0
1008
19 13.0
0 12:5
12.0
11.5
120
No
Duc
360
10000
6000
4000
2000
Euc
40K
50.04
0.02
0.00
0.003
time (hr)
Steps
Steps
(e) Token-wise valida-
(f) Time-wise validation (g) Step-wise Adam vari-
(h) Step-wise Adam vari- ance max element
tion perplexity
perplexity
ance norm
Figure 4: Validation perplexity and Adam variance norm/max element during GPT-2 1.5B seqlen IK pre-training, comparing the baseline and proposed work (SLW) under different batch sizes/L.R Also compare with related works ("Shortformer" [3) and "Bsz Warmup" (6) at 2nd row. Each row of subfigures share the same legend ("SLW 45K" means our work with T=45K steps).
Based on the following observations, we demonstrate that our approach resolves the dilemma and simultaneously improves the stability and efficiency. We will mainly present the GPT-2 1.5B results and leave some GPT-2 17M results in Appendix.
Significant stability gain: In Section 3 [able we discussed how we measure the training instability based on the "loss ratio" metric, which shows that the baseline becomes less stable under larger model size/batch size/learning rate/sequence length. Comparing with baseline and proposed work in this table shows that our work reduces this instability measurement to zero in all cases, together with max ratio close to 1.0 (no spike). This demonstrates the significant stability gain by our method.
Faster token-wise and time-wise convergence: Figure and 4(b) present the validation perplexity curves during GPT-2 1 1.5B seqlen IK pre-training, comparing baseline and our approach. When the batch size increases from 512 to 4K for baseline, the time-wie wise convergence becomes faster but the loken-wise convergence becomes slower and poorer. On the other hand, our approach at batch size 4K provides faster and better convergence both token-wist and time-wise comparing with the best baseline curve in each case. The shape of SLW's curves is different (worse than baseline in early stage) because SLW cases initially only learn from shorter sequences, which limit the validation perplexity it can reach (since validation data is always full-length). On the other hand, when SLW cases start to learn from longer sequences, the validation perplexity drops faster than baseline (and related works) and eventually surpasses them.
Our approach with batch size 512 provides smaller convergence speedup because (1) Baseline with batch size 512 has less instability issue, limiting the gain from the proposed approach; (2) At batch size 512 the communication overhead very high, and our approach takes more steps (i.e.. communication rounds) than baseline to reach the same 157B training tokens. This extra communication cost "cancelled" part of the time saving from our approach. For GPT-2 117M. our approach provides similar token-wise and time-wiee wise convergence speedup (Appendix A.4).
Advancing cost-quality Pareto curve: In Section 3 Table 2 we discussed about baseline's zero-shot evaluation results. For proposed work eval results in this table, we present them in two ways: one evaluated the earliest checkpoint that provides better eval results than baseline (batch size 512 and seqlen IK); the other one evaluated at the end of full training. Results show that our approach is able to advance the cost-quality Pareto curve in two ways: (1) To reach the same eval result quality as baseline, our approach reduces the required number of pre- pre-training tokens and wall clock time by up to 2.2x and 3.7x, respectively; (2) Under the same 157B training tokens, our approach can further improve the eval result quality. In (1) the time-wise saving is higher than the foken-wiss because (a) For each Transformer block, the self-attention and intermediate layers have time complexity of O(B x I,2 x H) and O(B x L x H2), respectively 6 The proposed method uses shorter sequences at
B, L, H represent batch size, sequence length, hidden size.
8

---

the beginning, reducing the time complexity quadratically for the self- self-attention sub-layer and linearly for the intermediate sub-layer of Transformer blocks; (b) By enabling stable training at larger batch size, our approach achieves additional time-wise saving by reducing the communication overhead.
Variance reduction helps stabilize training: In Section 3 we discussed the strong correlation between training instability and gradient variance norm/max. Figure and 4(d) demonstrate that proposed approach stabilizes training and reduces both the Adam variance norm and the variance max element. Importantly, it avoids all the spikes of the variance max element, which all happen to be where the baseline has training loss spikes. One may wonder why gradient clipping cannot help avoid these extreme gradient variance outliers. Although gradient clipping can avoid large gradient at every single step, it cannot avoid the gradient variance getting accumulated from multiple steps A....d
Comparing with related works: We now compare the proposed work with two related works on the most challenging "1.5B model + batch size 4K" case. The first work is the Shortformer where the first stage uses shorter sequences and the second stage uses full-length sequences [30). Following the grid search in the paper, we use seqlen 128 for the first stage and set its duration at about half of the baseline duration (20K steps). The second work is the batch size warmup technique used by GPT-3 [6), where we set the starting batch size at 128 and then gradually increase it to 4K, and set the warmup duration same as the proposed work. Other training hyperparameters are unchanged. Figure He]too 4 present the results. Both related works provide convergence speedup but it is less than our work. More importantly, they still have training instability issues. The Shortformer has an obvious training divergence at step 20K when the sequence length switches from 128 to 1K (the spike at 20K in Figure(4(h) This shows when staying at the same shorter sequence length for too long. the model becomes heavily overfitted for that length which leads to divergence risk when/after switching to full length. Although both batch size warmup and our method reduce the number of tokens per batch in a similar fashion, batch size warmup does not provide any training stability benefit compared to the baseline. This indicates that providing the same number of shorter (simpler) sequences leads to better training stability than providing fewer number of same length (same difficulty) sequences. In addition, batch size warmup has a limitation that the batch size must be multiple of data- auta-parallel size. On the other hand, for method the sequence length only needs to be multiple of 8 to enable Tensor Core accelcration. These two limitations are different: for the proposed SLW method, the "multiple of 8" limitation is fixed and unrelated to data-pat paralel size. For batch size warmup it's dynamic "multiple of nata-parallel size" limitation: since nowadays pre-training tasks are performed on up to thousands of GPUs, the data-parale nata-parallel size can easily go beyond 100, prohibiting flexible configuration of the method (or requires reducing the number of GPUs when using smaller batches, increasing the training clock time). Last but not least, both related works provide non-zero "loss ratio" in Table I and worse zero-shot evaluation results in Table 2
5.2 GPT-3 experiments
For experiments replicating the GPT-3 125M model (6) using the Pile public dataset [13]. first we reproduce the original GPT-3 training recipe: 300B training tokens, seqlen 2K, batch size 256 with batch size warmup (start with 16 then gradually increase to 256 in first 4B tokens), learning rate 6 x 10-4 with a linear warmup of 375M tokens and a single cycle cosine decay over 260B tokens (6 x 10 min. learning rate"" Then we explore an aggressive training scenario where only 30B tokens (10%) are allowed. This is because (1) GPT-3 paper admits that has poor training sample efficiency and it sees much more text during pre-training than human sees in the their lifetime [6] 23 (2) There could exist cases where the total amount of data/computation resource is limited. We adjust several hyperparameters in this 30B-token training: 8x batch size (2K) for better training efficiency, learning rate decay reduced to 30B tokens (based on study that LR schedule should match total training tokens [4], warmup stays at 375M), min. learning rate reduced to 0 (based recent study on GPT-3 [5T). For baseline we keep using 4B-token batch size warmup, but when our method is used (seglen, = 72, T = 11.5K based on tuning) we disable it since both methods reduce tokens per batch. And for both cases we tune the learning rate and use the highest one that provides stable training, which is 30x (1.8 x 10 2) for baseline and 40x (2.4 x 10 2) for our method. All experiments are performed on 128 V100 GPUs
Figure 5 and 6 present the training loss and gradient variance max for the GPT-3 pre-training experiments. When applying 40x learning rate to the baseline (batch size warmup), it quickly
Diifferent from GPT-2, GPT-3 uses token-based learning rate schedule and we follow it.
9

---

10
0.25
Training loss
repro
0.20
Dandin Sandrn--22
Table 4: Zero-shot evaluation of the trained GPT-3 125M models on 11 tasks used by the original GPT-3 work [6. Per- task eval results in Appendix A.5
Jariance
0.15
S/W BO
0.10
0.05
250M 500M 750M 18
0.00
0
250M 500M 750M 18
Batch
Traiiin
Tamig
Average
Tokens
Tokens
Case
size
tokens
accuracc
Figure 5: Training loss
Figure 6: Gradient vari- ance max during GPT-3 125M pre-training (first IB tokens).
Original [6]
256
300B
33.6
during GPT-3 125M
2: Baseline repro
256
300B (Ix)
61Hr
31.4
Baseline 30x LR
30B (10x)
7Hr
(95%)
pre-training (first IB to-
4:
SLW 40x LR 2K
30B (10k) 3.5H: (17s) 31.1 (99%)
kens).
diverges and cannot continue to train due to NaN losses. The corresponding gradient variance max element becomes a flat line after divergence because the gradients all dimensions are large that all gradients get clipped including the max element. After lowering the learning rate to 30x, the baseline is able to finish the whole training, but it can only retain 95% of average sero-shot accuracy on
11 tasks (HellaSwag 153 LAMBADA (28), TriviaQA 15]. WebQs 3 Winogrande [38). PIQA [4],
(27) compared with the case that reproduces the original GPT-3 training recipe as shown in Table In contrast, our approach enables stable training with 40x learning rate, demonstrates lower gradient variance max outliers than baseline with 30x learning rate, retains 99% of the original training recipe's average zero-shot accuracy, and achieves 10x data saving and 17x time saving This demonstrates that the proposed method not only solves the stability-efficienn dilemma, but also opens a promising direction of significantly reducing total training cost in a different data efficiency dimension.
ARC Challenge/Fasy [TO] ANLI R1/R2/R3
Finally, in Appendix A.6 we evaluate the proposed method on a larger GPT-3 1.3B model, including not only zero-shot but also few-shot evaluation. Results show that under the same 300B training tokens the proposed SLW method provides better average accuracy than the baseline for both zero- shot (from 41.6 to 41.9) and few-shot (from 44.8 45.3) tasks, demonstrating that the proposed method (in addition to the stability Similar to the original GPT-3, under few-shot prompts the average accuracy is better than zero-shot results for both models trained with baseline batch size warmup and proposed SLW method.
benefit) can provide better accuracy performance.
6 Limitation and Future Work
Although our paper discovers the correlation between long sequences and training instability (and proposes a method to alleviate the issue), the root cause of this causal relationship is not fully deciphered and would be interesting future work direction. One assumption we have is that the shorter sequences are not necessarily easier but can be viewed as simpler examples since there are less context to embed. We think encoding shorter sequences (especially at the early training stage when weight is still relatively random) will generate less "noisy" gradients thus leading to higher stability.
This work presents correlation analysis on an empirical connection between training instability and gradient variance norm/max element, but it is not sufficient to prove a causal relationship, and training instability could be caused by other factors. Overall, our work is one preliminary attempt to study the topic of training instability in large-scale model training, and we hope it could inspire future works to further study this important issue in both theory and practice.
7 Conclusion
This paper presents the Sequence Length Warmup method, which solves a stability-effiiiemt dilemma inside GPT-style model traitraining, an critical issue that hinders efficient language model pre-training as explained in our in-depth analysis. By enabling stable training on more aggressive training recipe, this method also motivates a new dimension of training cost reduction by improving the data efficiency, as demonstrated by the 10x data and 17x time saving in our GPT-3 experiments. We believe that the effectiveness, simplicity, and easy-to-use/tune make the proposed method a must-try for deep learning practitioners, and we hope this work could motivate more studies on improving training data efficiency.
*Our reproduced GPT-3 has 2.2 point lower average accuracy than the original GPT-3, which is because of the different training data and OpenAl employed many data processing techniques (6] 'We want to emphasize here that the SLW method "only retains 99% of accuracy" because this experiment is extreme case: only 10% of original training data is used during training.
10

---

References
[1] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International Conference Machine Learning. pages 176-185. PMLR, 2017.
[2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[3] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-ansee answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544, 2013.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[5] Ondicj Bojar, Jindrich Helel, Tom Koemi, Jindřich Libovický, and Tomáš Musil. Results of the wmt17 neural mt training task. Proceedings of the second conference on machine translation, pages 525-535 2017.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balean, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[7] Daniel Campos. Curriculum learning for language modeling. arXiv preprint ar arX1v:2108.02.70. 2021.
[8] Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. International Conference on Machine Learning, pages 1141-1150. PMLR, 2019.
[9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmano, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1xXi::8803.05457 2018
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[12] Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99 1993.
[13] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
[14] Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[15] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:170.003551, 2017.
11

---

[16] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.0823 008311, 2020.
[17] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.69800 2015.
[18] Tom Koemi and Ondřej Bojar. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 379-386, 2017.
[19] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, volume 1, page 2, 2010.
[20] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704 04683, 2017.
[21] Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, and Yuxiong Hc. 1-bit LAMB: Communication Efficient Large-Scate Large-Batch Training with LAMB's Convergence Speed. arXiv preprint arXiv: 200i::22040000 100.06669 2021.
[22] Conglong Li, Minjia Zhang, and Yuxiong He. Curriculum learning: A regularization method for efficient and stable billion-scale gpt model pre-training arXiv preprint arXiv:2108.06000 060844 2021.
[23] Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? arXiv preprint urXiv:2005.0095: 2020.
[24] Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Mohammad Alizadeh. Variance reduction for reinforcement learning in input-driven environments. In International Conference on Learning Representations, 2018.
[25] Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia. Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint 17Xivv
2017.
[26] Microsoft and Nvidia. Using DeepSpeed and Megatron Train Megatron-Turing NLG 530B, the World's Largest and Most Powerful Generative Language Model.
http:://deeellloerrrvvvaaaaaaaoo http://///ooooeeeeoneeeeeeeeeeee and- mord-meagttno-ttoo traia-.mggatroon-turnn-------
generatire-language--ooo whoe-----o-o-----ooo-o--------ee teo--tth-worrrr worlds- largest-g-----r- and wwa---osttpppp mooost--overul-
2021.
[27] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint 1rXiv:1910.1455 14599, 2019.
[28] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 06666, 2016.
[29] Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M Mitchell. Competence-based curriculum learning for neural machine translation. In NAACL- HIT, 2019.
[30] Ofir Press, Noah A Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. arXiv preprint arXiv:2012, 15832, 2020.
[31] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[32] Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. Better language models and their implications. OpenAl Blog, 2019.
[33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2018.
12

---

[34] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
[35] Samyam Rajbhandari, Jeff Rasley. Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[36] Mrinmaya Sachan and Eric Xing. Easy questions first? a case study on curriculum learning for question answering. In Proceedings of the 54th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pages 453-463, 2016.
[37] Mrinmaya Sachan and Eric Xing. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 629-640, 2018.
[38] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 8732-8740, 2020.
[39] Terence D Sanger. Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE transactions on Robotics and Automation, 10(3):323-332 1994.
[40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi billion parameter language models using model parallelism. arXiv preprint arXiv:1909.088
2019.
[41] Samuel L. Smith, Pjeter-Jan Kindermans, Chris Ying, and Quoe V. Le. Don't decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[42] Shaden Smith, Mostofa Patwary. Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint urXiv:2201.1199 2022.
[43] Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit Adam: Communication Efficient Scale Training with Adam's Convergence Speed. In Proceedings of the 38th International Conference on Machine Learning, pages 10118-101210 2021.
[44] Yi Tay, Shuohang Wang, Anh Tuan Luu, Jie Fu, Minh C Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. Simple and effective curriculum pointer-gemerator networks for reading comprehension over long narratives. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4922 4931, 2019.
[45] Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint arXiv: 02247, 2018.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
[47] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00530 2019.
[48] Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. In NIPS, 2013.
[49] Thomas Wolf. The engineering group in @bigsciencew fighting training instabilities over + +100b parameters.
2021.
13

---

[50] Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Curriculum learning for natural language understanding. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104, 2020.
[51] Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203. 03466, 2022.
[52] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020.
[53] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can 2019.
machine really finish your sentence? arXiv preprint
[54] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi, Defending against neural fake news. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 1054-9065, 2019.
[55] Wei Zhang, Wei Wei, Wen Wang, Lingling Jin, and Zheng Cao. Reducing bert computation by padding removal and curriculum learning. In 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 90-92. IEEE, 2021.
[56] Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint 1.000033, 2018.
[57] Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. Curriculum learning for domain adaptation in neural machine translation. In NAACL-HLLT, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We have a section describing limitation and future work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] We open sourced the code in deep learning optimization library called DeepSpeed 10) and
Section 3 and Fadncldde all instructions needed to reproduce the experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] andtionnandSSjneeeee
all the training details.
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [N/A] Our results are based single seed given the pre-training expensive.
14

---

(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Section and include the information.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We open sourced the code in deep learning optimization library called DeepSpeed.
(d) Did you discuss whether and how consent was obtained from people whose data you'te rurre using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "lWU6G-an0pr", "reviewer": "Reviewer_c8jb", "summary": "This work introduces the Sequence Length Warmup (SLW) method: a simple but demonstrably effective technique for improving training stability of GPT models. This paper contributes some characterization of the instability problem, possible connections to gradient variance, and empirical evidence that SLW can sidestep this issue. The main body of the paper elaborates on implementation details and includes various comparisons to quantify the specific cost/quality improvements offered by SLW. The paper concludes with experiments with GPT-3, which further demonstrate the technique.", "questions": "Is it fair to use the same distributed set-up for comparing the 512 vs. 4k batch size settings? Presumably one easy way to reduce communication cost would be to use fewer nodes with the smaller batch size. From my understanding, all jobs were run with the same hardware, which might mean that the 512 baseline is slower than it needs to be. I'm not sure about this potential criticism, so I would appreciate it if the authors could explain why this is a fair comparison in terms of efficiency.\n\nHave you done any fine-tuning experiments to confirm that the benefits to zero-shot learning would also apply to other fine-tuning settings? If there is suitable prior work to establish this, it would be helpful to mention somewhere in the paper.", "limitations": "The paper doesn't have a \"societal impact\" section but I think that is OK here.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n- Within the experimental scope of the paper, the results show clear improvements, particularly in gaining access to more \"aggressive\" (and more efficient) training settings.\n- The authors make good use of a simple loss spike and loss ratio metric for quantifying instability. Through this lens, SLW has a dramatic reduction on instability as it generally reduces this quantity to 0. (But see weaknesses below)\n- SLW is simple to implement and does not seem to require prohibitive hyperparameter tuning. (Although, the tuning heuristic suggested by the authors is pretty vague -- e.g., \"significant validation perplexity fluctuation\" is unclear.)\n- The comparison between SLW and batch size warmup is interesting.\n\n# Weaknesses:\n- The empirical connection between training instability and gradient variance is not particularly convincing (Fig 1, Table 3). This connection is limited to a correlation analysis, but it is easy to doubt any causal implications. I expect that there are many arbitrary measurements of the training history that statistically significantly correlate with gradient variance.\n- The best metric for showing that SLW reduces instability is loss spikes. However, there isn't any clear evidence that loss spikes interfere with training quality. I'm willing to give the benefit of the doubt here, but it would at least make things cleaner if some analysis were provided that would better motivate the goal of preventing loss spikes.\n- The narrative of this paper is hard to follow. Ultimately, this weakens the characterization of the \"stability-efficiency dilemma.\" In sum, it seems safe to say that it reflects some poorly understood interplay between batch size, learning rate, gradient variance, and sequence length, but all that this paper really teaches us is that we can make it less of a dilemma by doing SLW. In many ways, that's a perfectly useful conclusion, but it does limit the *understanding* we gain of the phenomenon. As a more concrete example, the motivation for SLW seems to follow the observation that training instability requires sequences longer than 128. The authors leverage that observation into a clearly useful technique, but, in all honesty, all the surrounding analysis and conjecture is very *ad hoc* and not very convincing. It would have been far more valuable to use some of this paper space to dig into sequence length as a driver of instability. It's too bad that we're left with no new knowledge other than that it is involved and SLW helps.\n\n# Comments:\n- In Section 2, it would be helpful to unpack some of the prior works referenced in the *Curriculum Learning* paragraph, especially those where \"CL was explored and verified for NLP one-stage and fine-tuning tasks.\" The current phrasing is too vague and fails to provide proper context for this paper/work.\n- You write: \"Baseline with batch size 4K is less stable than baseline with batch size 512, indicating that larger batch sizes could lead to more training instability risks.\" This seems like an unwarranted claim since batch size is confounded with learning rate here. The authors should clarify this.\n- In Figure 2, it would be helpful to annotate the 10% of time where the yellow curve has 1k sequence length, especially since the main text says that these portions of training tend to trigger the instability. Also, please double check that the steps listed in line 213 (\"900, 1800, 2700...\") are accurate.\n- The figures are often hard to read because so much visual space is being used for labels under subfigures, axis numbers, etc. This will not affect my evaluation but it would be good to improve figure quality before finalizing the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZKep_KVWg-", "reviewer": "Reviewer_iCLQ", "summary": "This paper proposed a simple and effective method, \"sequence length warmup\" (SLW), to improve the training stability and efficiency, in GPT model pre-training.\n\nIn different experiment configurations, this paper reveals the \"stability-efficiency dilemma for GPT model pre-training\", and a strong correlation between training instability and extreme values of gradient variance.\n\nThe experiments are conducted on the pre-training of GPT-2 (117 M and 1.5 B parameters) and GPT-3 (125M parameters): SLW reduces the required number of training tokens and wall clock time by up to 2.2x and 3.7x. For GPT-3 (125M parameters), when limited to 10% of data available, 8X larger batch size and 40x larger learning rate, SLW can retraining 99% of the zero-shot accuracy, in 12x less time. In the same setting, baseline can retrain 95% of the zero-shot accuracy after lowering learning rate to 30x.", "questions": "* How about the few-shot performance of the pre-trained model using SLW? Adding the few-shot results (no matter higher or lower), would give us a more clear picture of how well SLW is. Because just like in the title of GPT-3 paper, \"Language models are few-shot learners\", we need to know how well SLW is in the few-shot settings, if SLW is applied on \"few-shot learners\".\n\n* On line 351-352, there is \"In addition, batch size warmup has a limitation that the batch size must be multiple of data-parallel size, which will be large for distributed training.\" May I know, if the proposed SLW has similar \"limitation\"? It seems like the \"sequence length\" also \"must be multiple of data-parallel size which will be large for distributed training\" if want to use the Tensor Core acceleration, like \n\n* A similar phenomenon in the original BERT paper, \"To speed up pre-traing in our experiments, we pre-train the model with sequence length of 128 for 90% of the steps. Then, we train the rest 10% of the steps of sequence of 512 to learn the positional embeddings.\" The increasing from sequence length 128 to 512 makes me also believe that the proposed SLW works to some degree. However, the official BERT believes that the longer sequence is to \"learn the positional embeddings\". I am not totally using this to contradict this submission,  just combining with the point one i mentioned in the weakness part above: good stability does not sufficiently leads to better accuracy performance. Hence, may I know why a smoother learning curve (called stability in this submission) is important? Because the main goal is to have a pre-trained model with higher accuracy and better generalization ability (even if it might have a little spike in the loss curve of pretraining).", "limitations": "* Given that the stability (means the smoothness of training loss curve in this submission) does not sufficiently leads to better accuracy and generalization, the importance of a stability (smoothness) needs better illustrated.\n\n* For the GPT models, like the paper title of GPT-3 \"language models are few-shot learners\", reporting the accuracy performance of pre-trained language model using SLW, in the few-shot setting for the downstream tasks, would make the submission more convincing. Or at least give us a better picture of how well the proposed SLW is.\n\n* If adding experiment results on other large language models, the submission would be more convincing about the universality part of the SLW.\n\n* The relation between the proposed SLW and Curriculum learning are not that strong, as shorter sequence are not easier for sure.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* simple and effective method for addressing the \"stability-efficiency dilemma for GPT model pre-training\"\n\nWeaknesses:\n* The stability (means the smoothness of training loss curve in this submission) typically does not sufficiently leads to better accuracy performance. Like in this paper, the SLW can \"retrain 99% of the zero-shot accuracy\", with smoother training loss curve.\n* How about the few-shot performance of the pre-trained model using SLW? Adding the few-shot results (no matter higher or lower), would give us a more clear picture of how well SLW is. Because just like in the title of GPT-3 paper, \"Language models are few-shot learners\", we need to know how well SLW is in the few-shot settings, if SLW is applied on \"few-shot learners\".\n* I agree that this work is based on good reproduction of the pre-training of GPT-2 and GPT-3, with thinking and new findings (SLW and stability-efficiency dilemma). How general is SLW might be an issue, for a claim, the name of which seems to be an optimizer contribution), some additional experiments on other large language models would make this paper more convincing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NdEOPgLFpvv", "reviewer": "Reviewer_S5Qt", "summary": "In this work, the authors address the difficulty of training large language models with a novel strategy: sequence length warmup. The authors note that transformer training requires limited batch size and learning rate, and that methods which allow both to be increased can improve training efficiency. Their proposal is to use a training strategy which uses short sequence length examples to start, and gradually brings in longer examples.\n\nThe authors argue that though increasing batch size decreases the overall norm of the gradient variance estimation vector used by ADAM, but increases the largest element - which may cause training instability. They show that sequences with longer lengths also display the same phenomenology. They correlate large values of this largest element with issues with the stability of training.\n\nThe authors then propose a strategy to combat these instabilities: sequence length warmup. During the warmup phase, the maximum sequence length of seen examples is slowly increased. This leads to better training stability at larger batch size, and overall quicker convergence (in terms of number of steps) as well as more efficient training (in terms of number of tokens seen). In particular the sequence length warmup with a linear rampup of sequence length outperforms the shortformer, which has a step-function increase in sequence length.", "questions": "One issue with the analysis in Section 3 is that both the batch size AND the learning rate have been increased simultaneously. What happens when large batch size is used with a different learning rate? It seems like you may be able to better trade efficiency and stability with simultaneous optimization of both parameters.\n\nWith regards to the analysis of the norm/max element of the variance vector: the fact that the max element increases with batch size runs counter to what one would expect from distributions with finite moments - but is a known property of long-tailed distributions (e.g. those with power law tails). Is there a way to see this directly? E.g. by computing moments/distributions of the gradients near initialization/during training?\n\nRecent work has shown the importance of the curvature of the loss landscape on optimization, particular in the early stages of learning in the transformer architecture:\n\nhttps://openreview.net/forum?id=OcKMT-36vUs\n\nDoes sequence length warmup improve the conditioning of the Hessian?\n\nIs there an explanation for the significantly different shapes of the learning curves in Figure 4 (particualarly (e) and (f)) of the SLW method as compared to the other methods? It's interesting that the curves are qualitatively different.", "limitations": "Adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The analysis of the failure modes of the transformer training with respect to batch size and sequence length was quite original and reasonably clearly presented. The results with regards to sequence length are particularly compelling. One criticism is that figures like Figure 1 are hard to parse - both because of the total number of plots, as well as their size. In addition, details of the pre-warmup quantities is of theoretical and practical interest and is unclear from the plot.\n\nHowever one weakness of the analysis is that batch size and learning rate were increased simultaneously. It leaves open the question of whether or not a better learning rate tuning would allow for the use of a larger batch size, while still maintaining some of the efficiency gains. This is also where a zoomed in look at the dynamics of the quantities early in training would be important; larger batch size should have smaller variability with equal parameters, so something in the early learning dynamics pushes the model to have more variance.\n\nIt is not clear to me that spikes in the max variance is itself is an issue; with an ADAM optimizer and/or gradient clipping, updates in these large gradient directions will be suppressed. In addition, a larger overall norm is not intrinsically a problem if paired with the right learning rate schedule/an adaptive optimizer like ADAM. It does seem though that those quantities may be indicative of larger issues with  the optimization/loss landscape progression during learning.\n\nRegardless of the issues with the analysis of the stability-efficiency tradeoff, the sequence length warmup algorithm seems to be of great significance - leading to improved performance in terms of compute and final attainable perplexity. The overall sequence length warmup algorithm is very similar to the cited Shortformer work, though the small modifications seem to have provided significant improvements. Figure 3 was particularly interesting - showing the tradeoff between learning curves in terms of number of total steps versus number of total tokens seen.\n\nThe discussion of changing the cost-quality Pareto frontier was also interesting, but felt incomplete; the paper could be improved with more explicit exploration of this frontier.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5vQjrElpVaP", "reviewer": "Reviewer_e6nP", "summary": "This paper names and addresses the stability/efficiency dilemma in large language model training: larger batch sizes and learning rates enable more efficient and faster distributed training, but also increase the prevalence of instabilities which tend to hurt performance and can even cause divergence (Figures 5 and 6).  This problem appears to get worse for larger models.\n\nThe work proposes increasing the sequence length throughout the initial stage of training (“Sequence Length Warmup”).  This was proposed already in ShortFormer, but here they use a more sophisticated schedule, describe how to choose the schedule, and provide an explanation and analysis of this method in terms of instability.\n\nTheir experiments involve training open-source versions of GPT-2 and (125M) GPT-3, and produce large efficiency gains.", "questions": "Questions:\n* Do you in fact tune the duration hyperparameter on the test set? (251-252)  I assume it is the validation set?  Else this is a major problem for the soundness of the work.\n* Would the approach mentioned on lines 230-234 scale to data limited regimes?  If I understand correctly, you are throwing away data when you truncate sequences?\n* RE 263-265: If I understand correctly, it was necessary to train to convergence in order to discover this heuristic? Did you verify that this heuristic works in other cases in order to validate it?\n* Why do you say the method acts as a regularizer? (line 330)  This claim seems unsupported and is never discussed elsewhere that I noticed.\n\nSuggestions:\n* “gradient variance” should be defined/explained more thoroughly.\n* The document should be checked for grammar and typos (e.g. 32, 64, 84).\n* A few cite’s should be citet’s (e.g. [37] in Table 2).\n* Saying the methods “solves” the dilemma (217) is overstated; e.g. it should be qualified with “in our experiments” or something like that.", "limitations": "Large language models are liable to have a large impact on society; this impact may not be positive.  Increasing the efficiency of training LLMs is likely to increase deployment, increase the size of models deployed, and facilitate more study of the impact and potential harms of these models.  This could be worth discussing briefly.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed method appears to work very well and produce major gains in efficiency.\n* There are strong supporting results demonstrating that the method removes instabilities.\n* There is a good and clear discussion of the experiments and method.  Overall clarity was good.\n\nWeaknesses:\n* The reason why using longer sequences creates instability was not discussed much.\n* The reason why instabilities lead to worse performance when they do not cause divergence was not discussed much.\n* The existence of ShortFormer detracts a bit from the novelty.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept